Software intelligent interaction channel expansion method based on bypass architecture

Abstract

The invention discloses a software intelligent interaction channel expansion method based on bypass architecture, which is applied to an intelligent agent side and comprises the following steps: receiving a task instruction issued by a user side, processing task data corresponding to the task instruction, and determining a task demand under the condition that the task data information is complete, the task requirement carries a task type and a task requirement; generating an initial software state label based on a current operating system interface screenshot, determining a current software state based on the initial software state label, and generating an execution scheme according with a target format according to the current software state, the task demand and priori knowledge; and simulating keyboard and mouse input according to the execution scheme to carry out software operation so as to realize a user request. By applying the method, the intelligent interaction capability can be newly increased for the software through a non-intrusive bypass architecture.

Description

Technical Field

[0001] The present application relates to the field of intelligent interaction technology, and in particular to a software intelligent interaction channel expansion method based on a bypass architecture. Background Art

[0002] Human-computer interaction is the process of converting interaction intentions into a series of interactive operations based on one's own experience and knowledge and executing them step by step. Compared with traditional human-computer interaction, intelligent human-computer interaction mainly uses voice as the interaction channel. Its characteristics are that humans can interact directly with machines through natural language, and the machines are responsible for understanding human intentions and automatically performing operations. Taking air-conditioning appliances as an example, in the intelligent interaction scenario, when the user intends to turn on the air conditioner for cooling, he only needs to say "I'm a little hot", and the air conditioner can recognize the voice, understand the user's needs, automatically turn on the air conditioner and lower the temperature.

[0003] At present, the way for products to expand intelligent interaction channels is reconstruction and upgrading: enterprises lead the research and development, and upgrade the capabilities by reconstructing the original products. In the process, the original products need to be modified on a large scale. Taking air-conditioning appliances as an example, to achieve voice control, the hardware and software of the air conditioner need to be redesigned: adding microphones to the hardware, upgrading the processing chip and the main control circuit board, redeveloping the software, and adding intelligent algorithms such as voice recognition, voice understanding, and command matching.

[0004] Obviously, for software companies, the existing reconstruction and upgrade methods have a long R&D cycle, high cost, and small incremental benefits. In addition, reconstruction may bring about software instability and cause the original functions to fail. For users, software upgrades and intelligent interactions are led by the company. If the company lacks the willingness or ability to upgrade, users will not be able to enjoy the intelligent interaction function. Even if each company upgrades its own software, since software of different brands are not compatible and cannot be called with each other, users cannot control different software through a unified intelligent interaction channel, which increases user usage costs and reduces the convenience and versatility of intelligent interaction.

[0005] In view of this, how to provide a software intelligent interaction channel expansion method based on a bypass architecture to realize the ability of adding intelligent interaction to the software through a non-intrusive bypass architecture has become a technical problem that urgently needs to be solved. Summary of the invention

[0006] The embodiments of the present application provide a software intelligent interaction channel expansion method based on a bypass architecture, a software intelligent interaction channel expansion device based on a bypass architecture, an electronic setting and a computer storage medium, which are used to solve the problem of inconvenience in adding new intelligent interactions to current software.

[0007] In a first aspect of an embodiment of the present application, a method for expanding a software intelligent interaction channel based on a bypass architecture is provided, which is applied to an intelligent agent terminal and includes:

[0008] Receive a task instruction issued by a user terminal, process the task data corresponding to the task instruction, and determine the task requirement if the task data information is complete, wherein the task requirement carries the task type and task requirements;

[0009] Based on the current operating system interface screenshot, an initial software state label is generated, and based on the initial software state label, the current software state is determined, and according to the current software state, the task requirements and prior knowledge, an execution plan that conforms to the target format is generated;

[0010] According to the execution scheme, keyboard and mouse input are simulated to perform software operations and realize user requests.

[0011] In a second aspect of an embodiment of the present application, a software intelligent interaction channel expansion device based on a bypass architecture is provided, which is applied to an intelligent entity terminal and includes:

[0012] The determination module is configured to receive a task instruction issued by a user terminal, process the task data corresponding to the task instruction, and determine the task requirement when the task data information is complete, wherein the task requirement carries the task type and task requirement;

[0013] A generation module is configured to generate an initial software state label based on the current operating system interface screenshot, determine the current software state based on the initial software state label, and generate an execution plan that conforms to the target format according to the current software state, the task requirements and prior knowledge;

[0014] The execution module is configured to simulate keyboard and mouse input to perform software operations according to the execution scheme to realize user requests.

[0015] According to a third aspect of an embodiment of the present application, a computing device is provided, including:

[0016] Memory and processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the software intelligent interactive channel expansion method based on the bypass architecture are implemented.

[0018] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the software intelligent interactive channel expansion method based on the bypass architecture are implemented.

[0019] The present application provides a method for expanding software intelligent interaction channels based on a bypass architecture, which is applied to an intelligent body end, and includes: first, receiving a task instruction issued by a user end, processing task data corresponding to the task instruction, and determining task requirements when the task data information is complete, wherein the task requirements carry the task type and task requirements; then, based on a screenshot of the current operating system interface, generating an initial software status label, and based on the initial software status label, determining the current software status, and generating an execution plan that conforms to the target format according to the current software status, the task requirements and prior knowledge; finally, simulating keyboard and mouse input to perform software operations according to the execution plan to implement user requests.

[0020] By applying the software intelligent interaction channel expansion method based on bypass architecture provided in the embodiment of the present application, the ability of intelligent interaction can be added to the software through a non-intrusive bypass architecture. The bypass architecture can safely and low-costly expand the intelligent interaction function of the software, without intrusion or modification to the original software, without the need for software development companies to lead and without the need for software developers to support; the intelligent interaction function of the software can be quickly, flexibly, efficiently and continuously upgraded through the incremental learning and fine-tuning training technology of intelligent agents based on large models; the intelligent control of different types of software and different brands of software can be achieved through a unified interactive entrance, reducing the user's learning cost and difficulty of use, and improving the convenience and versatility of intelligent interaction; the development difficulty and workload of the intelligent agent are reduced through the autonomous learning mode, and the efficiency of the intelligent agent's expansion of new software is improved.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0023] Figure 1 A schematic diagram of the structure of a software intelligent interactive channel expansion system based on a bypass architecture provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a method for expanding software intelligent interactive channels based on a bypass architecture provided in an embodiment of the present application;

[0025] Figure 3A schematic diagram of the structure of a software intelligent interactive channel expansion device based on a bypass architecture provided in an embodiment of the present application;

[0026] Figure 4 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] See also Figure 1 , Figure 1 The following is a schematic diagram of a software intelligent interactive channel expansion system based on a bypass architecture provided in an embodiment of the present application. Figure 1 As shown, the software intelligent interactive channel expansion system includes a three-end interactive implementation of a user terminal 101, an interactive auxiliary intelligent agent 102 and a software 103.

[0029] The embodiment of the present application constructs an interactive auxiliary intelligent agent 102 for assisting interaction between the user terminal 101 and the software 103. The interactive auxiliary intelligent agent 102 is responsible for interacting with the user terminal 101 through natural language, understanding the user's intention, and automatically operating the software 103 to complete the task.

[0030] For a certain software to be optimized, the interactive assistant agent 102 is firstly made to master the functions and corresponding operation procedures provided by the software through autonomous learning and manual teaching. In practical applications, the interactive assistant agent 102 and the software 103 run in parallel, and the user terminal 101 can communicate and express needs with the interactive assistant agent 102 through voice. The interactive assistant agent 102 is responsible for understanding the user's intention and controlling the software 103 to achieve the user's needs by simulating keyboard and mouse operations. The specific implementation steps are as follows.

[0031] See also Figure 2 , Figure 2 The flowchart of a method for expanding a software intelligent interactive channel based on a bypass architecture provided in an embodiment of the present application is applied to an intelligent agent (i.e., an interactive auxiliary intelligent agent). Figure 2 As shown, the specific steps include the following steps.

[0032] Step S202: receiving a task instruction issued by the user terminal, processing the task data corresponding to the task instruction, and determining the task requirement when the task data information is complete, wherein the task requirement carries the task type and task requirement.

[0033] Step S204: Generate an initial software status tag based on the current operating system interface screenshot, and determine the current software status based on the initial software status tag, and generate an execution plan that conforms to the target format according to the current software status, the task requirements and prior knowledge.

[0034] Step S206: simulating keyboard and mouse input to perform software operations according to the execution scheme to implement user requests.

[0035] In the embodiment of the present application, the learning stage of the intelligent agent is first described. It should be noted that the learning stage here is a stage in which the intelligent agent learns the functions of the software and the corresponding operations through autonomous exploration and manual teaching.

[0036] The specific implementation steps include: running the software in a sandbox environment, running the intelligent agent in parallel, and starting the corresponding learning strategy based on environmental factors, wherein the learning strategy includes an autonomous exploration strategy and a manual teaching strategy; based on the autonomous exploration strategy, randomly triggering software functions by simulating user inputs, and recording corresponding software state changes; or, based on the manual teaching strategy, learning the software operation method at the task level during the user operation process.

[0037] In one aspect of an embodiment of the present application, based on the autonomous exploration strategy, software functions are randomly triggered by simulating user input, and corresponding software state changes are recorded, including: taking a screenshot of the software interface, splitting the screenshot into minimized UI units through an image segmentation algorithm, and naming and saving the minimized UI units through an image description algorithm, wherein the minimized UI units include menu options, buttons, icons and texts; based on the simulated input of the minimized UI unit, the triggering operation of the minimized UI unit and the software state changes caused by the triggering operation are determined, and operation data in a triple format is generated, wherein the simulated input includes mouse input and keyboard input; the triple format carries the initial state of the software, UI units and operations, and the new state of the software; the operation data in the triple format is converted into a fine-tuning training data set, and the agent is fine-tuned based on the fine-tuning training data set to learn the relationship between software state transitions and the corresponding operation methods.

[0038] It should be noted that the mouse input here includes left-click, left-double-click, right-click, and right-double-click.

[0039] In actual applications, the intelligent agent simulates keyboard and mouse operations to explore the triggering operation of the smallest UI unit and the method of causing software state changes, including: first, the intelligent agent takes a screenshot of the software interface, and generates a description of the initial state of the software through an image description algorithm; then, the intelligent agent simulates various keyboard and mouse operation inputs; secondly, the intelligent agent takes a screenshot of the software interface again, and generates a description of the new state of the software after the simulated operation through an image description algorithm; thirdly, the intelligent agent parses the software background update message and the sandbox system update message, and supplements them into the description of the new state of the software after the simulated operation; thirdly, the process data is recorded and saved in the database in the format of a triplet of "initial state of software>UI unit+operation>new state of software"; finally, the above process is repeated until all UI units are traversed.

[0040] In another aspect of an embodiment of the present application, the method for learning task-level software operation by learning the user operation process based on the manual teaching strategy includes: receiving task execution data sent by the user end, wherein the task execution data carries task information, a start node, and an end node; recording input information corresponding to the user end operation process, and recording the corresponding state information of the software each time the user performs an operation, wherein the input information includes voice input, keyboard input, and mouse input; combining the task execution data and the input information to obtain operation data, and converting the data format corresponding to the operation data into a triple format, and saving it to a database; repeating the above steps until the user end completes all task teaching; using the teaching data corresponding to the task teaching as a fine-tuning training data set, and fine-tuning the agent based on the fine-tuning training data set to learn the relationship between software state transitions and the corresponding operation methods.

[0041] Then, the embodiment of the present application describes the application stage of the intelligent agent. It should be noted that the application stage here is the stage in which the intelligent agent obtains user needs, understands user intentions, and automatically operates the software to realize user needs.

[0042] Among them, the application node includes four parts, namely, the demand communication part, the execution plan generation part, the automatic operation software part and the execution result evaluation part.

[0043] It should be noted that in the demand communication part, users can give instructions or describe their needs to the intelligent agent through voice or text, and the intelligent agent is responsible for parsing, understanding and identifying user needs.

[0044] In an embodiment of the present application, the task instruction issued by the user terminal is received, the task data corresponding to the task instruction is processed, and the task requirement is determined when the task data information is complete, including: receiving the task instruction issued by the user terminal, selecting a corresponding algorithm according to the interaction mode of the user terminal issuing the instruction, and parsing the task data corresponding to the task instruction into text information, wherein the interaction mode includes voice interaction and text interaction; judging the integrity of the text information, and in response to the existence of missing information in the text information, sending a guidance information completion text to the user terminal through a target guidance method, wherein the target guidance method includes automatically searching a knowledge base, determining user-related information, and sending a guidance information completion text to the user terminal; or, sending a guidance information completion text to the user terminal through multiple rounds of voice interaction; in response to the task data information being complete, matching the software to the corresponding task requirement according to the user input data and the search results, wherein the task requirement carries the task type and task requirement.

[0045] In actual applications, for voice input, the intelligent agent parses the voice into text through algorithms such as speech recognition; the intelligent agent determines the completeness and clarity of the user's input information. If there is missing information, the intelligent agent guides the user to supplement the missing information by searching the knowledge base or through multiple rounds of dialogue; the intelligent agent identifies the user's intention based on the user input and search results, and matches the corresponding software task type and requirements.

[0046] It should be noted that the execution plan generation part generates an execution plan for the agent based on task requirements.

[0047] In actual applications, first, the agent takes a screenshot of the current operating system interface and generates the current software status label through an image description algorithm; secondly, the agent generates an execution plan based on the current software status, task requirements and prior knowledge. The plan format is "software initial state> UI unit + operation> software state 01>...> target software state".

[0048] It should be noted that the automatic operation software part is for the intelligent agent to simulate keyboard and mouse input to perform software operations according to the execution plan.

[0049] In an embodiment of the present application, the software operation is performed by simulating keyboard and mouse input according to the execution scheme to implement the user request, including: taking a screenshot of the software interface, generating a first software status tag through an image description algorithm, and in response to the first software status tag being consistent with the initial software status tag in the execution scheme, querying the first image information carried in the first software status tag according to the first UI unit tag, and processing the first image information based on the image positioning algorithm to determine the first real-time position corresponding to the UI unit, wherein the first software status tag is the current software status tag; based on the first real-time position corresponding to the first UI unit, simulating keyboard and mouse input to trigger the first operation of the corresponding software; taking a screenshot of the software interface after the first operation, generating a second software status tag through an image description algorithm, and in response to the second software status tag being consistent with the second software status tag in the execution scheme, querying the second image information carried in the second software status tag according to the UI unit tag, and processing the second image information based on the image positioning algorithm to determine the second real-time position corresponding to the second UI unit; based on the second real-time position corresponding to the second UI unit, simulating input to trigger the second operation of the corresponding software; repeating the above steps until the target software state is reached.

[0050] Specifically, in response to the first software status tag being consistent with the initial software status tag, the first image information carried in the first software status tag is queried according to the first UI unit tag, and also includes: in response to the first software status tag being inconsistent with the initial software status tag, the execution plan is regenerated according to the first software status tag and the task requirements.

[0051] More specifically, in response to the second software status label being consistent with the first software status label, the second image information carried in the second software status label is queried according to the UI unit label, and also includes: in response to the second software status label being inconsistent with the first software status label, a screenshot of the software interface is taken again, and a third software status label is generated through an image description algorithm, wherein the third software status label is a regenerated software current status label.

[0052] In actual applications, the intelligent agent takes a screenshot of the current software interface, generates a label for the current software status through an image description algorithm, and compares it with the initial software status in the execution plan. If they are consistent, it proceeds to the next step; otherwise, it returns to the generation of the execution plan part to regenerate the execution plan; the intelligent agent queries its image information according to the UI unit label, and obtains the real-time position of the UI unit under the current software state based on the image positioning algorithm, and then simulates the keyboard and mouse input to trigger the corresponding software operation; the intelligent agent takes a screenshot of the software interface after the operation, generates a new software status label through an image description algorithm, and compares it with the software status 01 in the execution plan. If they are consistent, it proceeds to the next step; otherwise, it returns to step a); if information to be supplemented is encountered during the execution process, such as SMS verification codes, the intelligent agent is responsible for asking the user and filling them in; the operations are performed in sequence until the target software state is reached.

[0053] It should be noted that in the evaluation of execution results, the agent generates feedback information based on the execution process data to reply to the user.

[0054] In an embodiment of the present application, execution process data is acquired, feedback information is generated based on the process data, and the feedback information is sent to a user end.

[0055] Specifically, the acquisition of execution process data, generation of feedback information based on the process data, and sending the feedback information to the user end include: recording each state comparison result carried in the process data; in response to the number of state comparison inconsistencies in the state comparison results exceeding a preset threshold, stopping task execution and feeding back to the user end that the task requirements cannot be achieved; in response to the number of state comparison inconsistencies in the state comparison results being less than a preset threshold, and the final software state label is consistent with the target software state label corresponding to the target software state, feeding back to the user end that the task is completed.

[0056] In actual applications, the intelligent agent is responsible for recording the results of each state comparison. If the execution plan generated for three consecutive times cannot be executed smoothly, the execution will be stopped and the user will be fed back that the task requirements cannot be achieved. If the plan is executed smoothly and is finally consistent with the software target state, the task completion will be fed back.

[0057] By applying the method provided in the embodiments of the present application, the intelligent interaction function of the software can be expanded safely and at low cost through a bypass architecture, without intrusion or modification to the original software, and without the need for a software development company to lead or software developers to support it; through the large model-based intelligent agent incremental learning and fine-tuning training technology, the intelligent interaction function of the software can be upgraded quickly, flexibly, efficiently and continuously; through a unified interaction entrance, intelligent control of different types of software and different brands of software can be achieved, reducing the user's learning cost and difficulty of use, and improving the convenience and versatility of intelligent interaction; through the autonomous learning mode, the development difficulty and workload of the intelligent agent are reduced, and the efficiency of the intelligent agent in expanding new software is improved.

[0058] Corresponding to the above method embodiment, this specification also provides an embodiment of a software intelligent interactive channel expansion device based on a bypass architecture, Figure 3 The following is a schematic diagram of the structure of a software intelligent interactive channel expansion device based on a bypass architecture provided in an embodiment of the present application. Figure 3 As shown, the device comprises:

[0059] The determination module 302 is configured to receive a task instruction issued by the user terminal, process the task data corresponding to the task instruction, and determine the task requirement if the task data information is complete, wherein the task requirement carries the task type and task requirement;

[0060] The generating module 304 is configured to generate an initial software status tag based on the current operating system interface screenshot, determine the current software status based on the initial software status tag, and generate an execution plan that conforms to the target format according to the current software status, the task requirements and prior knowledge;

[0061] An execution module 306 is configured to simulate keyboard and mouse input to perform software operations according to the execution scheme to implement user requests;

[0062] The feedback module 308 is configured to obtain execution process data, generate feedback information based on the process data, and send the feedback information to the user end.

[0063] In an optional embodiment, the determining module 302 is further configured to:

[0064] Receive a task instruction issued by a user terminal, select a corresponding algorithm according to the interaction mode of the instruction issued by the user terminal, and parse the task data corresponding to the task instruction into text information, wherein the interaction mode includes voice interaction and text interaction;

[0065] Determining the integrity of the text information, in response to information missing from the text information, sending a guidance information completion text to the user terminal through a target guidance method, wherein the target guidance method includes automatically searching a knowledge base, determining user-related information, and sending the guidance information completion text to the user terminal; or sending the guidance information completion text to the user terminal through multiple rounds of voice interaction;

[0066] In response to the task data information being complete, the software is matched to the task requirements according to the user input data and the search results, wherein the task requirements carry the task type and task requirements.

[0067] In an optional embodiment, the execution module 306 is further configured to:

[0068] Taking a screenshot of the software interface, generating a first software status tag through an image description algorithm, and in response to the first software status tag being consistent with the initial software status tag in the execution scheme, querying first image information carried in the first software status tag according to the first UI unit tag, and processing the first image information based on an image positioning algorithm to determine a first real-time position corresponding to the UI unit, wherein the first software status tag is a current software status tag;

[0069] Based on a first real-time position corresponding to the first UI unit, simulating keyboard and mouse input to trigger a first operation of the corresponding software;

[0070] Taking a screenshot of the software interface after the first operation, generating a second software status tag through an image description algorithm, and in response to the second software status tag being consistent with the second software status tag in the execution scheme, querying the second image information carried in the second software status tag according to the UI unit tag, and processing the second image information based on the image positioning algorithm to determine a second real-time position corresponding to the second UI unit;

[0071] Based on a second real-time position corresponding to the second UI unit, simulating input triggering a second operation of the corresponding software;

[0072] Repeat the above steps until the target software state is reached.

[0073] In an optional embodiment, the execution module 306 is further configured to:

[0074] In response to the first software state tag being inconsistent with the initial software state tag, an execution plan is regenerated according to the first software state tag and task requirements.

[0075] In an optional embodiment, the execution module 306 is further configured to:

[0076] In response to the inconsistency between the second software status label and the first software status label, a new screenshot of the software interface is taken, and a third software status label is generated through an image description algorithm, wherein the third software status label is a regenerated software current status label.

[0077] In an optional embodiment, the feedback module 308 is further configured to:

[0078] Record the comparison results of each state carried in the process data;

[0079] In response to the number of inconsistent status comparisons in the status comparison result exceeding a preset number threshold, the task execution is stopped, and feedback is given to the user end that the task requirement cannot be achieved;

[0080] In response to the number of state comparison inconsistencies in the state comparison result being less than a preset threshold, and the final software state tag being consistent with the target software state tag corresponding to the target software state, task completion is fed back to the user end.

[0081] In an optional first embodiment, before the determination module 302, it is configured as follows:

[0082] The operation module 310 is configured to run the software in the sandbox environment, run the agent in parallel, and start the corresponding learning strategy based on environmental factors, wherein the learning strategy includes an autonomous exploration strategy and a manual teaching strategy;

[0083] The first learning module 312 is configured to randomly trigger software functions by simulating user input based on the autonomous exploration strategy and record corresponding software state changes; or

[0084] The second learning module 314 is configured to learn the software operation method at the learning task level during the user operation process based on the manual teaching strategy.

[0085] In an optional embodiment, the first learning module 312 is further configured to:

[0086] Taking a screenshot of the software interface, splitting the screenshot into minimized UI units by using an image segmentation algorithm, and naming and saving the minimized UI units by using an image description algorithm, wherein the minimized UI units include menu options, buttons, icons, and texts;

[0087] Based on the minimized UI unit simulated input, determine the triggering operation of the minimized UI unit and the software state change caused by the triggering operation, and generate operation data in a triplet format, wherein the simulated input includes mouse input and keyboard input; the triplet format carries the software initial state, UI unit and operation, and software new state;

[0088] The operation data in the triple format is converted into a fine-tuning training data set, and the agent is fine-tuned based on the fine-tuning training data set to learn the relationship between software state transitions and corresponding operation methods.

[0089] In an optional embodiment, the second learning module 314 is further configured to:

[0090] Receiving task execution data sent by a user terminal, wherein the task execution data carries task information, a start node, and an end node;

[0091] Record the input information corresponding to the user's operation process, and record the software status information corresponding to each operation performed by the user, wherein the input information includes voice input, keyboard input and mouse input;

[0092] Combining the task execution data and the input information to obtain operation data, converting the data format corresponding to the operation data into a triple format, and saving the data into a database;

[0093] Repeat the above steps until the user completes all task teaching;

[0094] The teaching data corresponding to the task teaching is used as a fine-tuning training data set, and the agent is fine-tuned based on the fine-tuning training data set to learn the relationship between software state transitions and corresponding operation methods.

[0095] By using the device provided in the embodiments of the present application, the intelligent interaction function of the software can be expanded safely and at low cost through a bypass architecture, without intrusion or modification to the original software, and without the need for a software development company to lead or software developers to support it; through the large model-based intelligent agent incremental learning and fine-tuning training technology, the intelligent interaction function of the software can be upgraded quickly, flexibly, efficiently and continuously; through a unified interaction entrance, intelligent control of different types of software and different brands of software can be achieved, reducing the user's learning cost and difficulty of use, and improving the convenience and versatility of intelligent interaction; through the autonomous learning mode, the development difficulty and workload of the intelligent agent are reduced, and the efficiency of the intelligent agent in expanding new software is improved.

[0096] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the software intelligent interactive channel expansion device based on the bypass architecture, since it is basically similar to the software intelligent interactive channel expansion method embodiment based on the bypass architecture, the description is relatively simple, and the relevant parts can be referred to the partial description of the software intelligent interactive channel expansion method embodiment based on the bypass architecture.

[0097] Figure 4 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0098] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0099] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0100] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 may also be a mobile or stationary server.

[0101] Among them, the processor 420 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the software intelligent interactive channel expansion method based on the bypass architecture.

[0102] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the software intelligent interactive channel expansion method embodiment based on the bypass architecture, the description is relatively simple, and the relevant parts can be referred to the partial description of the software intelligent interactive channel expansion method embodiment based on the bypass architecture.

[0103] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the software intelligent interactive channel expansion method based on the bypass architecture.

[0104] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the software intelligent interactive channel expansion method embodiment based on the bypass architecture, the description is relatively simple, and the relevant parts can be referred to the partial description of the software intelligent interactive channel expansion method embodiment based on the bypass architecture.

[0105] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the software intelligent interactive channel expansion method based on the bypass architecture.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the software intelligent interactive channel expansion method embodiment based on the bypass architecture, the description is relatively simple, and the relevant parts can be referred to the partial description of the software intelligent interactive channel expansion method embodiment based on the bypass architecture.

[0107] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0109] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0110] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A software intelligent interaction channel expansion method based on bypass architecture, applied to an intelligent agent, characterized in that: include: Receive a task instruction issued by a user terminal, process the task data corresponding to the task instruction, and determine the task requirement if the task data information is complete, wherein the task requirement carries the task type and task requirements; Based on the current operating system interface screenshot, an initial software state label is generated, and based on the initial software state label, the current software state is determined, and according to the current software state, the task requirements and prior knowledge, an execution plan that conforms to the target format is generated; According to the execution scheme, keyboard and mouse input are simulated to perform software operations and realize user requests.

2. The method according to claim 1, characterized in that The receiving of the task instruction issued by the user terminal, processing the task data corresponding to the task instruction, and determining the task requirement when the task data information is complete, includes: Receive a task instruction issued by a user terminal, select a corresponding algorithm according to the interaction mode of the instruction issued by the user terminal, and parse the task data corresponding to the task instruction into text information, wherein the interaction mode includes voice interaction and text interaction; Determining the integrity of the text information, in response to information missing from the text information, sending a guidance information completion text to the user terminal through a target guidance method, wherein the target guidance method includes automatically searching a knowledge base, determining user-related information, and sending the guidance information completion text to the user terminal; or sending the guidance information completion text to the user terminal through multiple rounds of voice interaction; In response to the task data information being complete, the software is matched to the task requirements according to the user input data and the search results, wherein the task requirements carry the task type and task requirements.

3. The method according to claim 1, characterized in that The step of simulating keyboard and mouse input to perform software operation according to the execution scheme to implement user requests includes: Taking a screenshot of the software interface, generating a first software status tag through an image description algorithm, and in response to the first software status tag being consistent with the initial software status tag in the execution scheme, querying first image information carried in the first software status tag according to the first UI unit tag, and processing the first image information based on an image positioning algorithm to determine a first real-time position corresponding to the UI unit, wherein the first software status tag is a current software status tag; Based on a first real-time position corresponding to the first UI unit, simulating keyboard and mouse input to trigger a first operation of the corresponding software; Taking a screenshot of the software interface after the first operation, generating a second software status tag through an image description algorithm, and in response to the second software status tag being consistent with the second software status tag in the execution scheme, querying the second image information carried in the second software status tag according to the UI unit tag, and processing the second image information based on the image positioning algorithm to determine a second real-time position corresponding to the second UI unit; Based on a second real-time position corresponding to the second UI unit, simulating input triggering a second operation of the corresponding software; Repeat the above steps until the target software state is reached.

4. The method according to claim 3, characterized in that In response to the first software status tag being consistent with the initial software status tag in the execution scheme, querying the first image information carried in the first software status tag according to the first UI unit tag, further comprising: In response to the first software state tag being inconsistent with the initial software state tag, an execution plan is regenerated according to the first software state tag and task requirements.

5. The method according to claim 3, characterized in that: In response to the second software status tag being consistent with the first software status tag, querying the second image information carried in the second software status tag according to the UI unit tag, further comprising: In response to the inconsistency between the second software status label and the first software status label, a screenshot of the software interface is taken again, and a third software status label is generated through an image description algorithm, wherein the third software status label is a regenerated software current status label.

6. The method according to claim 1, characterized in that The method further comprises: Acquire execution process data, generate feedback information based on the process data, and send the feedback information to a user end.

7. The method according to claim 6, characterized in that The acquiring execution process data, generating feedback information based on the process data, and sending the feedback information to the user end includes: Record the comparison results of each state carried in the process data; In response to the number of inconsistent status comparisons in the status comparison result exceeding a preset number threshold, the task execution is stopped, and feedback is given to the user end that the task requirement cannot be achieved; In response to the number of state comparison inconsistencies in the state comparison result being less than a preset threshold, and the final software state tag being consistent with the target software state tag corresponding to the target software state, task completion is fed back to the user end.

8. The method according to claim 1, characterized in that Before receiving the task instruction issued by the user terminal, the method includes: Running the software in a sandbox environment, running the agent in parallel, and starting a corresponding learning strategy based on environmental factors, wherein the learning strategy includes an autonomous exploration strategy and a manual teaching strategy; Based on the autonomous exploration strategy, randomly trigger software functions by simulating user input and record corresponding software status changes; or, Based on the artificial teaching strategy, the user operation process is learned to learn the software operation method at the task level.

9. The method according to claim 8, characterized in that Based on the autonomous exploration strategy, software functions are randomly triggered by simulating user inputs, and corresponding software state changes are recorded, including: Taking a screenshot of the software interface, splitting the screenshot into minimized UI units by using an image segmentation algorithm, and naming and saving the minimized UI units by using an image description algorithm, wherein the minimized UI units include menu options, buttons, icons, and texts; Based on the minimized UI unit simulated input, determine the triggering operation of the minimized UI unit and the software state change caused by the triggering operation, and generate operation data in a triplet format, wherein the simulated input includes mouse input and keyboard input; the triplet format carries the software initial state, UI unit and operation, and software new state; The operation data in the triple format is converted into a fine-tuning training data set, and the agent is fine-tuned based on the fine-tuning training data set to learn the relationship between software state transitions and corresponding operation methods.

10. The method according to claim 8, characterized in that The software operation method of learning the task level of the user operation process based on the artificial teaching strategy includes: Receiving task execution data sent by a user terminal, wherein the task execution data carries task information, a start node, and an end node; Record the input information corresponding to the user's operation process, and record the software status information corresponding to each operation performed by the user, wherein the input information includes voice input, keyboard input and mouse input; Combining the task execution data and the input information, obtaining operation data, converting the data format corresponding to the operation data into a triple format, and saving the data into a database; Repeat the above steps until the user completes all task teaching; The teaching data corresponding to the task teaching is used as a fine-tuning training data set, and the agent is fine-tuned based on the fine-tuning training data set to learn the relationship between software state transitions and corresponding operation methods.

Images