Internet equipment control method and device, computer equipment and storage medium

Abstract

The invention relates to an internet equipment control method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of calling a pre-trained reasoning model in response to a triggered interaction instruction; processing the interaction instruction through the reasoning model to obtain a reasoning result corresponding to the interaction instruction; wherein the reasoning result comprises operation information for executing a related operation action flow based on the interaction instruction; executing a corresponding operation process based on the operation information; or, the operation information is sent to the target internet equipment, so that the target internet equipment executes the corresponding operation process based on the operation information. By adopting the method, the convenience of controlling the internet equipment can be effectively improved, intelligent control is realized, the control experience of the user on the internet equipment is further effectively improved, and convenience is brought to the user.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a control method, apparatus, computer equipment, storage medium, and computer program product for an Internet device. Background Art

[0002] With the development of computer and Internet technologies, most smart devices are equipped with networking modules to access the Internet. Various manufacturers have developed cloud-based systems for managing and controlling smart devices.

[0003] However, in current internet device control methods, each manufacturer's cloud is typically only used to manage and control its own smart devices, and cannot be managed and controlled through third-party clouds. If you need to manage smart devices from different manufacturers, you must log in to each manufacturer's cloud to manage and control the devices separately, which reduces user experience. Therefore, how to improve the user control experience of internet devices has become an urgent problem that needs to be solved. Summary of the Invention

[0004] Based on this, it is necessary to provide a control method, device, computer device, computer-readable storage medium and computer program product for Internet devices to address the above technical problems, which can effectively improve the convenience of controlling Internet devices and realize intelligent control, thereby effectively improving the user's control experience of Internet devices and bringing convenience to users.

[0005] In a first aspect, the present application provides a method for controlling an internet device. The method comprises: in response to a triggered interaction instruction, invoking a pre-trained inference model; processing the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information for executing an operation action flow related to the interaction instruction; executing a corresponding operation process based on the operation information; or, sending the operation information to a target internet device, causing the target internet device to execute the corresponding operation process based on the operation information.

[0006] In a second aspect, the present application also provides a control device for an Internet device. The control device includes: a calling module for calling a pre-trained inference model in response to a triggered interaction instruction; a processing module for processing the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the interaction instruction; an execution module for executing a corresponding operation process based on the operation information; or sending the operation information to a target Internet device so that the target Internet device executes the corresponding operation process based on the operation information.

[0007] On the third aspect, the present application also provides an intelligent cleaning device, which includes a robot body and a cleaning base station, and the robot body or the cleaning base station is provided with a control device of an Internet device, and the control device includes: a calling module, which is used to call a pre-trained reasoning model in response to a triggered interaction instruction; a processing module, which is used to process the interaction instruction through the reasoning model to obtain an reasoning result corresponding to the interaction instruction; wherein the reasoning result includes operation information of an operation action flow related to the execution of the interaction instruction; an execution module, which is used to execute a corresponding operation process based on the operation information; or, send the operation information to a target Internet device so that the target Internet device executes a corresponding operation process based on the operation information.

[0008] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed: in response to a triggered interaction instruction, calling a pre-trained inference model; processing the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the interaction instruction; executing a corresponding operation process based on the operation information; or sending the operation information to a target Internet device, so that the target Internet device executes the corresponding operation process based on the operation information.

[0009] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps: in response to a triggered interaction instruction, calling a pre-trained inference model; processing the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the interaction instruction; executing a corresponding operation process based on the operation information; or sending the operation information to a target Internet device so that the target Internet device executes the corresponding operation process based on the operation information.

[0010] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps: in response to a triggered interaction instruction, invokes a pre-trained inference model; processes the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the interaction instruction; executes a corresponding operation process based on the operation information; or, sends the operation information to a target Internet device, causing the target Internet device to execute the corresponding operation process based on the operation information.

[0011] The control method, control device, intelligent cleaning device, computer device, storage medium and computer program product of the above-mentioned Internet device, by responding to the triggered interaction instruction, calling the pre-trained inference model, and processing the interaction instruction through the inference model to obtain the inference result corresponding to the interaction instruction; wherein the inference result includes operation information based on the operation action flow related to the execution of the interaction instruction; further, executing the corresponding operation process based on the operation information; or sending the operation information to the target Internet device so that the target Internet device executes the corresponding operation process based on the operation information. Since the inference model in this application is a pre-trained model for processing the interaction instruction, the pre-trained inference model can be called to process the interaction instruction to obtain the inference result corresponding to the interaction instruction, so that the server or target Internet device can automatically execute the corresponding operation process based on the operation information of the operation action flow related to the execution of the interaction instruction included in the inference result, without the need for multiple tedious interactions with the user, which can effectively improve the convenience of controlling the Internet device, realize intelligent control, and thus effectively improve the user's control experience of the Internet device, bringing convenience to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A diagram illustrating an application environment of a method for controlling an Internet device in one embodiment;

[0013] Figure 2 1 is a flow chart of a method for controlling an Internet device in one embodiment;

[0014] Figure 3 A schematic diagram of a product-side flow chart of a method for controlling an Internet device provided in one embodiment;

[0015] Figure 4 A schematic diagram of an interactive interface displayed by an exemplary target application in one embodiment;

[0016] Figure 5 A schematic diagram of a display interface of an inference model provided in one embodiment on a product side;

[0017] Figure 6 A structural block diagram of a control device for an Internet device in one embodiment;

[0018] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0020] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0021] The control method of the Internet device provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be a target Internet device, such as a sweeping robot, and the server 104 can be a background server or base station of the robot application. The server 104 responds to the interaction instruction triggered by the (operator) and calls the pre-trained reasoning model, and processes the interaction instruction through the reasoning model to obtain the reasoning result corresponding to the interaction instruction; wherein the reasoning result includes operation information of the operation action flow related to the execution of the interaction instruction, and the server 104 can execute the corresponding operation process based on the operation information; or, the server 104 sends the operation information to the target Internet device so that the target Internet device executes the corresponding operation process based on the operation information.

[0022] The terminal 102 may be an intelligent cleaning robot (such as a sweeper), a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, or a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, or a smart car device. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, or the like.

[0023] The server 104 may be an independent physical server or a service node in a blockchain system. A peer-to-peer (P2P) network is formed between the service nodes in the blockchain system. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0024] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN, Content Delivery NetworkCDN), and big data and artificial intelligence platforms.

[0025] The terminal 102 and the server 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network communication connection methods, and this application does not limit this.

[0026] In one embodiment, Figure 2 As shown, a control method for an Internet device is provided. The method can be executed by a server or a terminal alone, or by a server and a terminal together. Figure 1 The following steps are used as an example to illustrate the server in the example:

[0027] Step 202: In response to the triggered interaction instruction, call the pre-trained inference model.

[0028] Among them, interactive instructions refer to instructions for interaction triggered by the operator (user). The interactive instructions in this application can be information instructions including different information forms. It can be understood that the interactive instructions in this application include but are not limited to instructions in the form of voice, image, text, etc. For example, Figure 3 As shown, it is a flow chart of the product side of the control method of the Internet device provided by this application. The interactive instructions in this application can be as follows Figure 3 The multimodal interaction instructions shown in include but are not limited to: user-triggered voice instructions, text instructions, gesture instructions, etc.

[0029] A pre-trained inference model refers to a pre-trained inference model. For example, the model architecture of the inference model in this application can be a multimodal model, that is, the trained inference model in this application can be used to process pure text information, and can also be used to process multimodal information containing images and text.

[0030] Step 204 : Process the interaction instruction through the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the execution of the interaction instruction.

[0031] Among them, the inference result refers to the result output after the interaction instruction is processed by the inference model. For example, the inference result output by the inference model in this application can be operation information containing a complete set of operation processes (i.e., operation action flow).

[0032] Operation information refers to information used to instruct a device on how to execute a corresponding operation process. It is understood that the operation information in this application includes, but is not limited to, text operation flow information, image operation flow information, or video operation flow information. For example, the operation information (text operation flow information) in this application includes: selecting a target app → opening the target app → clicking the first control in the app, and other operation information related to the execution of a series of operation actions determined by the interactive instructions.

[0033] An operation action flow refers to a series of operation information used to instruct a device on how to execute a corresponding operation process (in the order of the operation actions). For example, the inference result output by the inference model in this application can be a series of operation information for executing the operation action flow, that is, the series of operation information for executing the operation action flow in this application includes at least: APP name, control name, coordinate information, text filling content (automatic filling content) and other information, so that the current device or target Internet device executes the corresponding operation process based on the operation information (such as selecting the target APP → opening the target APP → clicking the first control in the APP).

[0034] Specifically, the control method of the Internet device provided in the present application can be widely used in various scenarios that require interaction with users, such as smart home, social networking, ticket purchase, travel, etc., that is, the devices used by different users (operation objects) can interact with the intelligent robot application (or base station). When the user (operation object) wants to query or purchase ticket information (such as querying or purchasing relevant ticket information for "business trip to Shenzhen tomorrow"), the user can open the intelligent robot application (Application, APP) on the terminal through a trigger operation, and enter the interactive page of the intelligent robot application through a selection operation, that is, the user can log in to the intelligent robot application (such as xx robot application) through a trigger operation. Furthermore, the user can trigger the interactive instruction in different ways on the interactive page of the xx robot application displayed on the terminal, so that the terminal responds to the interactive instruction triggered by the user, and displays or voice broadcasts the execution result corresponding to the interactive instruction. For example, in the interactive page of the xx robot application displayed on the terminal, the operating object A using the xx robot application can trigger the generation of the corresponding interactive instruction A by inputting a text instruction and clicking controls such as "Confirm" and "Search" (the interactive instruction A is a text interactive instruction obtained in the form of text input), so that the terminal responds to the interactive instruction A triggered by the operating object A through the triggering operation in the interactive page of the xx robot application, and sends the interactive instruction A to the background server of the xx robot application, so that the background server of the xx robot application responds to the interactive instruction A, calls the pre-trained inference model, and processes the interactive instruction A through the inference model to obtain the inference result A corresponding to the interactive instruction A; wherein the inference result A in this application includes operation information based on the execution of the relevant operation action flow of the interactive instruction A, that is, the background server of the xx robot application can return the obtained inference result A corresponding to the interactive instruction A to the terminal, and the terminal can display the inference result A obtained based on the interactive instruction A in real time, for example, displaying the processing status information including the inference result A in real time in the interactive page that triggers the interactive instruction A, and for another example, can also synchronously display the prompt information "processing in progress".

[0035] It can be understood that the method provided in the present application can be implemented through interaction between the terminal and the background server of the intelligent robot application, or it can be implemented through interaction between the front end and the back end of the terminal, that is, the front end of the terminal is used to display the interactive page (interact with the user), and the back end of the terminal is equivalent to the background server, which is used to call the pre-trained inference model and process the interaction instructions through the inference model to obtain the inference results corresponding to the interaction instructions.

[0036] For example, let's take the processing scenario in an application with the function of simulating user operations (automated operations) as an example. Figure 4As shown in the figure, it is a schematic diagram of the interactive interface displayed by the target application. When user A wants to turn on the air conditioner in the bedroom of his home, user A can open a specific application on the terminal through a trigger operation, that is, the user can log in to the target application through a trigger operation. Further, the user can display the target application as shown in the figure. Figure 4 In the interactive interface shown in , the interactive instruction "Hello XX, turn on the bedroom air conditioner to 26°C" is generated by triggering the operation (the interactive instruction can be a voice interactive instruction obtained in the form of voice input). For example, the terminal displays the following Figure 4 In the interactive interface of the target application shown, user A can view the specific content and related functional information displayed in the interface. Assume that user A is in the interactive interface of the target application shown in FIG. Figure 4 Click the icon corresponding to the "Voice Control" in the interface shown in the figure, then the terminal responds to the above trigger operation of user A, collects the voice command of user A, and displays the text command corresponding to the collected voice command in the figure. Figure 4 The command input box shown in is: Hello XX, turn on the bedroom air conditioner to 26℃; further, user A can Figure 4 Click the button corresponding to the "OK control" in the interface shown. The terminal generates a request A carrying the instruction A "Hello XX, turn on the bedroom air conditioner to the temperature 26°C" in response to the above-mentioned click operation of user A, and sends the request A to the background server of the target application (or the back end of the terminal). The background server of the target application (or the back end of the terminal) can call the pre-trained inference model based on the instruction A carried by the request A, namely "Hello XX, turn on the bedroom air conditioner to the temperature 26°C", and process the instruction A through the inference model to obtain the inference result A corresponding to the instruction A, that is, take the instruction A as the input parameter of the model and input it into the pre-trained inference model. After processing by the inference model, it outputs the operation information of executing a series of related operation action flows (ie, the inference result A), and returns the obtained inference result A to the terminal for display.

[0037] It can be understood that the terminal in this application can be an intelligent cleaning robot with a cleaning function (such as a sweeper), that is, the user can interact with the intelligent cleaning robot. For example, user A can display the following on the screen of the intelligent cleaning robot: Figure 4 Click the icon corresponding to "Voice Control" in the interface shown in the figure, and the intelligent cleaning robot responds to the above triggering operation of user A, collects the voice command of user A, and displays the text command corresponding to the collected voice command in the figure. Figure 4 The command input box shown in is: Hello XX, turn on the bedroom air conditioner to 26℃; further, user A can Figure 4Click the button corresponding to the "OK" control in the interface shown. The smart cleaning robot will automatically generate a request A carrying the instruction A "Hello XX, turn on the bedroom air conditioner to the temperature 26°C" in response to the above-mentioned click operation of user A, and send the request A to the background server of the target application (or the back end of the terminal) so that the background server can process the instruction A carried in the request A.

[0038] Alternatively, the intelligent cleaning robot automatically calls the pre-trained inference model based on the real-time generated instruction A "Hello XX, turn on the bedroom air conditioner to 26°C", and processes the instruction A through the inference model to obtain the inference result A corresponding to the instruction A. That is, the instruction A is used as the input parameter of the model and input into the pre-trained inference model. After processing by the inference model, the operation information of executing a series of related operation action flows (that is, the inference result A) is output, and the obtained inference result A is displayed.

[0039] Step 206: Execute the corresponding operation process based on the operation information; or, send the operation information to the target Internet device so that the target Internet device executes the corresponding operation process based on the operation information.

[0040] The term "operation flow" refers to a series of action flow operations required to be executed in response to a triggered interaction instruction. For example, the operation flow in this application may be a complete operation flow consisting of multiple action flows, including opening different types of apps and simulating a user's triggering operation to select a target control (to achieve the user's needs corresponding to the interaction instruction). For example, the operation flow in this application may be: (simulating user operation) selecting a target app → opening the target app → (simulating user operation) clicking the first control in the target app.

[0041] The target Internet device refers to a device used to execute the corresponding operation process based on the operation information. For example, the target Internet device in this application can be a robot with a cleaning function (such as a sweeper) or other smart home devices.

[0042] Specifically, the server responds to the interaction instruction triggered by the user, calls the pre-trained inference model, and processes the interaction instruction through the inference model to obtain the inference result corresponding to the interaction instruction; wherein, after the inference result includes the operation information of the operation action flow related to the execution of the interaction instruction, the server can directly execute the corresponding operation process based on the obtained operation information; or, the server can also send the obtained operation information to the target Internet device (such as a sweeper) to enable the target Internet device to execute the corresponding operation process based on the operation information, for example, to trigger the target Internet device (such as a sweeper) to execute the corresponding cleaning work based on the operation information.

[0043] For example, let's take the processing scenario in an application with the function of simulating user operations (automated operations) as an example. Figure 4 Click the icon corresponding to the "Voice Control" in the interface shown in the figure, then the terminal responds to the above trigger operation of user A, collects the voice command of user A, and displays the text command corresponding to the collected voice command in the figure. Figure 4 The command input box shown in is: Hello XX, turn on the bedroom air conditioner to 26℃; further, user A can Figure 4 Click the button corresponding to the "OK" control in the interface shown. The terminal generates a request A carrying the instruction A "Hello XX, turn on the bedroom air conditioner to 26°C" in response to the above-mentioned click operation of user A, and sends the request A to the background server of the target application (or the back end of the terminal). The background server of the target application (or the back end of the terminal) can call the pre-trained inference model based on the instruction A carried by the request A, namely "Hello XX, turn on the bedroom air conditioner to 26°C", and process the instruction A through the inference model to obtain the inference result A corresponding to the instruction A, that is, input the instruction A as the input parameter of the model into the pre-trained inference model. After processing by the inference model, the operation information (ie, the inference result A) of executing a series of related operation action flows is output, and the obtained inference result A is returned to the terminal. The terminal can directly execute the corresponding operation process based on the obtained operation information. For example, the terminal can execute the following Figure 3 The operation flow shown in FIG 1 is as follows: the terminal (simulating a user operation) selects a target app → opens the target app → (simulating a user operation) clicks the first control in the target app. The target app can be an app for managing air conditioning, and the first control can be a control for adjusting the air conditioning temperature (for example, the first control is for controlling the temperature to a constant 26°C). Alternatively, the terminal can send the obtained operation information to a target internet device (such as a robot vacuum), causing the target internet device to execute the corresponding operation flow based on the operation information. For example, the terminal can trigger the target internet device (such as a robot vacuum) to execute the operation flow (simulating a user operation) of selecting a target app → opening the target app → (simulating a user operation) clicking the first control in the target app based on the operation information. After the operation flow is automatically executed, the execution result in the target app can be obtained and returned to the terminal for display, allowing the user to determine whether to trigger other commands based on the feedback of the execution result. In this way, the corresponding operation flow for the user-triggered instruction A, namely "Hello XX, turn on the bedroom air conditioning to 26°C," can be automatically completed without repeated user interaction, achieving efficient, accurate, and intelligent control of internet devices.

[0044] In this embodiment, in response to a triggered interaction instruction, a pre-trained inference model is called, and the interaction instruction is processed by the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information based on the operation action flow related to the interaction instruction execution; further, the corresponding operation process is executed based on the operation information; or the operation information is sent to the target Internet device so that the target Internet device executes the corresponding operation process based on the operation information. Since the inference model in this application is a pre-trained model used to process the interaction instruction, the pre-trained inference model can be called to process the interaction instruction to obtain an inference result corresponding to the interaction instruction, so that the server or target Internet device can automatically execute the corresponding operation process based on the operation information of the operation action flow related to the interaction instruction execution included in the inference result, without the need for multiple tedious interactions with the user, which can effectively improve the convenience of controlling the Internet device, realize intelligent control, and thus effectively improve the user's control experience of the Internet device, bringing convenience to the user.

[0045] In one embodiment, the method further comprises:

[0046] Displaying a prompt message for identity verification required when triggering an interactive command; the prompt message is used to prompt the operator to enter a voice verification command or a text verification command according to a target command style in a preset command style;

[0047] The calling of the pre-trained inference model in response to the triggered interaction instruction includes:

[0048] In a case where the voice verification instruction or the text verification instruction input by the operator passes verification, the pre-trained inference model is called in response to the triggered interaction instruction.

[0049] Specifically, the processing scenario in an application with the function of simulating user operations (automated operations) is used as an example for explanation. Before the user triggers an interactive instruction, that is, when the user logs into the target application (such as a large model application) for the first time, the terminal can display a prompt message for the identity verification required to trigger the interactive instruction; the prompt message is used to prompt the operator (i.e., the user) to enter a voice verification instruction or a text verification instruction according to the target instruction style in the preset instruction style, so that when the voice verification instruction or text verification instruction entered by the operator (i.e., the user) passes the verification (permission acquisition and authentication operation), the terminal responds to the interactive instruction triggered by the operator (i.e., the user), calls a pre-trained inference model, and processes the interactive instruction through the inference model to obtain an inference result corresponding to the interactive instruction (including operation information based on the operation action flow related to the execution of the interactive instruction), so that the terminal can subsequently directly execute the corresponding operation process based on the operation information; or, the terminal sends the operation information to the target Internet device (such as a vacuum cleaner), so that the target Internet device executes the corresponding operation process based on the operation information.

[0050] That is, the method provided in the present application can display a prompt message for identity verification required when triggering an interactive instruction when the pre-trained inference model is called for the first time; the prompt message is used to prompt the operator to enter a voice verification instruction or a text verification instruction according to the target instruction style in the preset instruction style, so as to improve the security of the operator's control over the Internet device and reduce the risk of misoperation.

[0051] Alternatively, according to the method provided in the present application, the terminal may also display a prompt message for identity verification required when triggering an interactive instruction each time a pre-trained inference model is called; the prompt message is used to prompt the operator to enter a voice verification instruction or a text verification instruction according to the target instruction style in the preset instruction style, so as to improve the convenience of the operator in controlling the Internet device, without the need to obtain permission and authenticate operations every time, avoiding tedious interactive operations and bringing convenience to users.

[0052] In one embodiment, the interaction instruction includes at least a voice interaction instruction, a text interaction instruction, and an interaction instruction in the form of a video; in response to the triggered interaction instruction, before calling the pre-trained inference model, the method further includes:

[0053] Displays an interface for triggering interactive commands; the interface includes voice controls, command input boxes, and video capture controls;

[0054] The calling of the pre-trained inference model in response to the triggered interaction instruction includes:

[0055] In the case where a voice interaction instruction is collected through the voice control, the pre-trained inference model is called in response to the triggered voice interaction instruction; or,

[0056] In the case where a text interaction instruction is collected through the instruction input box, the pre-trained inference model is called in response to the triggered text interaction instruction; or,

[0057] In the case where an interactive instruction in the form of a video is captured through the video capture control, the pre-trained inference model is called in response to the triggered interactive instruction in the form of the video.

[0058] Among them, the voice control refers to the control used to collect voice commands. For example, the voice control in this application can be Figure 4 A control displayed as a "microphone" icon in the interactive interface shown in .

[0059] Specifically, we take the processing scenario in an application with the function of simulating user operations (automatic operations) as an example. Assume that the target application (such as a large model application) displayed on the terminal includes an interactive interface of voice control, command input box and video capture control. Figure 4 As shown in , user A can Figure 4 In the interactive interface shown, different trigger operations are used to trigger the generation of interactive instructions. For example, the terminal can display the following Figure 4 The interface for triggering interactive instructions shown in FIG, which includes a voice control, an instruction input box and a video capture control, can be used by user A to click on the following Figure 4 , and collects voice commands through the voice control. That is, when the terminal collects voice commands through the voice control, the terminal responds to the automatically triggered voice interaction command and calls the pre-trained inference model to process the voice interaction command; or when the terminal collects voice commands through the voice control, the terminal responds to user A clicking on the voice control as shown in FIG. Figure 4 The "Confirm" control displayed in the form of a button shown in the figure triggers the generated interactive instruction, and calls the pre-trained reasoning model to process the interactive instruction. It can be understood that when the terminal in the embodiment of the present application collects the voice instruction through the voice control, the terminal can automatically trigger the generation of the voice instruction, without the operator clicking the "Confirm" control to trigger the generation of the instruction, that is, the embodiment of the present application supports directly calling the pre-trained reasoning model to process the automatically generated instruction without the user clicking the "Confirm" control, and also supports the user to further trigger the step of calling the pre-trained reasoning model to process the automatically generated instruction by clicking the "Confirm" control.

[0060] For example, user A can also Figure 4Enter a text instruction in the instruction input box of the interactive interface shown in the figure, that is, when the terminal collects the text instruction through the instruction input box, the terminal responds to the above text instruction (that is, text interaction instruction) triggered by user A and calls the pre-trained inference model to process the text interaction instruction; or user A can also enter the text instruction in the command input box as shown in the figure. Figure 4 In the interactive interface shown, click the video capture control to capture interactive instructions in video form (for example, interactive instructions containing multiple frames of hand gestures) through the video capture control. That is, when interactive instructions in video form are captured through the video capture control, the terminal responds to the interactive instructions in video form triggered by user A and calls the pre-trained inference model to process the interactive instructions in video form. This allows the interaction with users to be triggered and generated by using different interactive forms, such as (voice interaction), dialogue interaction, and video interaction, thereby enhancing the interactivity with users. Even for users who have interactive ideas but cannot describe them in specific text, relevant guidance or responses can be made to these users' vague interactive demands (such as gesture instructions in video form) and relevant interface displays can be provided, which can effectively improve the control experience of these users' Internet devices and bring convenience to users.

[0061] In one embodiment, the step of processing the interaction instruction through the inference model to obtain the inference result corresponding to the interaction instruction includes:

[0062] In the case where the interaction instruction is a voice interaction instruction, the voice interaction instruction is converted into text description information, and the text description information is processed by the inference model to obtain an inference result corresponding to the voice interaction instruction; or

[0063] When the interactive instruction is in the form of a video, the interactive instruction in the form of the video is converted into multimodal information including images and texts, and the multimodal information is processed through the inference model to obtain an inference result corresponding to the interactive instruction in the form of the video.

[0064] Specifically, the control method of the Internet device provided in this application is applied to the scenario of interaction between smart home devices and users. When user A wants to turn on the air conditioner in the bedroom of his home, user A can trigger an operation to open a specific application on the terminal (a smart home device such as a vacuum cleaner), that is, the user can log in to the target application through the trigger operation. Further, the user can select the target application to display the following information: Figure 4In the interactive interface shown in , an interaction instruction 1 of “Hello XX, turn on the bedroom air conditioner to 26°C” is generated by a trigger operation (the interaction instruction 1 may be a voice interaction instruction obtained in the form of voice input), that is, when the interaction instruction 1 is a voice interaction instruction, the terminal (a smart home device such as a sweeper) may convert the voice interaction instruction into text description information, and use the converted text description information as an input parameter of the model, and input it into the inference model for processing, that is, the text description information is processed by the inference model, and the inference result corresponding to the voice interaction instruction is output as follows: (simulating user operation) Select the target APP as APP1 for managing the air conditioner → Open the target APP1 → (simulating user operation) Click the first control in the target APP1, wherein the first control may be a control for adjusting the temperature of the air conditioner (for example, the first control is to control the temperature to be a constant temperature of 26°C).

[0065] Alternatively, the user can select the target application to display Figure 4 In the interactive interface shown in , an interactive instruction 2 of "turn on the air conditioner in room A to the temperature of 26°C" is generated by a trigger operation (the interactive instruction 1 can be an interactive instruction in the form of a video obtained in the form of video input), that is, when the interactive instruction 2 is an interactive instruction in the form of a video (for example, containing multiple frames of video images, and the video images contain the layout of room A), the terminal (a smart home device such as a sweeper) can convert the interactive instruction in the form of a video into multimodal information containing images and text, and process the multimodal information through a pre-trained inference model, and output the inference result corresponding to the interactive instruction in the form of a video: (simulating user operation) select the target APP as APP1 for managing the air conditioner → open the target APP1 → (simulating user operation) click the first control in the target APP1, wherein the first control can be a control for adjusting the temperature of the air conditioner (for example, the first control is to control the temperature to a constant temperature of 26°C). As a result, by adopting different forms of interaction to trigger the automatic generation of interaction instructions, for example, by adopting (voice interaction) dialogue interaction and video interaction, the interactivity with the user is enhanced, while the convenience of controlling Internet devices can be effectively improved, and intelligent control can be achieved, thereby effectively improving the user's control experience of Internet devices and bringing convenience to users.

[0066] In one embodiment, the steps of executing a corresponding operation process based on the operation information include:

[0067] When the operation information includes identification information of a third-party application, obtaining permission information of the third-party application corresponding to the identification information for starting the third-party application;

[0068] When the permission information is the target value, the third-party application is launched;

[0069] Jump to the page displayed by the third-party application, simulate the target trigger operation of the operator on the target control, so that the third-party application responds to the target trigger operation, executes the operation process corresponding to the target trigger operation, and generates result information of executing the operation process;

[0070] The result information is used as operation result information of executing the corresponding operation process based on the operation information.

[0071] Among them, the target value can be a preset value. For example, the target value in this application can be 1, that is, when the permission information is 1, it means that it is authorized to start. For example, when the permission information is 0 (not the target value 1), it means that it is not authorized to start.

[0072] Third-party applications refer to applications released by third-party systems or manufacturers. They have ecological restrictions and cannot be launched through the current manufacturer's applications.

[0073] Specifically, when the terminal or target Internet device executes the corresponding operation process based on the operation information, when the operation information includes the identification information of the third-party application (for example, the operation information is: select the target APP566 → open the target APP566 → (simulate user operation) click the first control in the target APP566), the terminal or target Internet device can obtain the permission information of the third-party application corresponding to the identification information (APP566) for starting the third-party application, and when the permission information is the target value, the terminal or target Internet device triggers the automatic start of the third-party application and jumps to the page displayed by the third-party application, simulating the target trigger operation of the operator on the target control, so that the third-party application responds to the target trigger operation, executes the operation process corresponding to the target trigger operation, and generates result information of executing the operation process. The terminal or target Internet device uses the result information generated by the third-party application as the operation result information of executing the corresponding operation process based on the operation information. This breaks the ecological restrictions between applications released by different manufacturers, and processes interactive instructions through reasoning models that simulate human understanding and reasoning capabilities. It also automatically simulates user click controls on different APPs based on the operation action flow output by the model. For example, it automatically simulates the user opening a ticket purchase app to grab tickets and place an order, making the terminal or target Internet device (such as a sweeping robot) the control center, realizing intelligent control, and effectively improving the user's control experience of Internet devices, bringing convenience to users.

[0074] In one embodiment, when the permission information is the target value, after starting the third-party application, the method further includes:

[0075] Obtain information about failure to start a third-party application and detect the installation status of the third-party application;

[0076] When the installation status of the third-party application is not installed, other applications with similar functions to the third-party application are searched from the local database, and the other applications are regarded as third-party applications, so that the result information of executing the operation process generated by the other applications is used as the operation result information of executing the corresponding operation process based on the operation information.

[0077] Specifically, the control method of the Internet device provided in this application is applied to the scenario of interaction between smart home devices and users as an example for explanation. Assuming that the operation information includes the identification information of the third-party application (for example, the operation information is: select the target APP566 → open the target APP566 → (simulate user operation) click the first control in the target APP566), the terminal or target Internet device can obtain the permission information of the third-party application corresponding to the identification information (APP566) for starting the third-party application, and when the permission information is the target value, the terminal or target Internet device triggers the automatic start of the third-party application APP566. If the start fails, the terminal or target Internet device obtains the information that the start of the third-party application APP566 fails, and the terminal or target Internet device can detect the third-party application APP566 in real time. In the case where the installation status of the third-party application APP566 is not installed (i.e., the third-party application APP566 is not installed or is not installed correctly in the terminal or target Internet device), the terminal or target Internet device can search for other applications with similar functions to the third-party application APP566 from the local database (for example, it finds APP236 with similar functions to the third-party application APP566 and is also used to control the bedroom air conditioner), and uses the other application APP236 as the third-party application, and uses the result information of the execution operation process generated by the other application APP236 as the operation result information of the corresponding operation process executed based on the operation information. This breaks the ecological restrictions between applications released by different manufacturers, simulates human understanding and reasoning ability through the inference model to process interaction instructions, and automatically simulates the user's click control of different APPs based on the operation action flow output by the model, making the terminal or target Internet device (such as a sweeping robot) a control center, realizing intelligent control, and effectively improving the user's Internet device control experience, bringing convenience to the user.

[0078] In one embodiment, before calling the pre-trained inference model in response to the triggered interaction instruction, the method further includes:

[0079] In response to a parameter input operation triggered in a model training page, a plurality of training samples for sample interaction instructions are obtained; the training samples include text data for describing an operation action flow and video data for describing an operation action flow;

[0080] In response to a training start operation triggered in the model training page, multiple training samples of the sample interaction instructions are input into the initial model for training to obtain a trained inference model.

[0081] Specifically, if Figure 5 As shown in FIG, it is a schematic diagram of the display interface of the inference model provided by this application on the product side. That is, when the user (operation object) wants to query or purchase ticket information (for example, query or purchase relevant ticket information for "a business trip to Shenzhen tomorrow"), the user can trigger the operation to open the intelligent robot application (Application, APP) on the terminal and enter the interface by selecting the operation. Figure 5 In the interactive interface of the intelligent robot application shown in , further, user A can Figure 5 In the interactive interface shown, the model training is started by triggering the operation, for example, the terminal displays the following Figure 5 In the interface shown, user A can click Figure 5 The "training" control in the interface shown in FIG. 1 is used to make the terminal respond to the above-mentioned click operation of user A and display the following Figure 5 In the input box or input page shown on the right, when user A completes the input operation (for example, user A inputs multiple training samples for sample interaction instructions), the terminal responds to user A in the following Figure 5 The parameter input operation is triggered in the model training page shown on the right. After obtaining multiple training samples (training samples include text data and video data for describing the operation action flow) for the sample interaction instructions input by user A and the labels corresponding to each training sample (labels are used to reflect the standardized operation action flow), user A can further click on Figure 5 If the "Start" control is clicked in the model training page shown on the right, the terminal responds to user A's Figure 5The training start action triggered on the model training page (i.e., clicking the "Start" control) above) inputs multiple training samples of sample interaction instructions into the initial model for training, resulting in a trained inference model. This allows for targeted training of the initial model by constructing training samples that include text data and video data describing the flow of operational actions. This allows the trained large model (inference model) to be used to infer various user-triggered interaction instructions, improving the accuracy of inference results without compromising the user experience. This effectively improves the accuracy of control of internet devices based on interaction instructions while also enhancing the user experience.

[0082] In one embodiment, after executing the corresponding operation process based on the operation information, the method further includes:

[0083] Obtaining operation result information of executing the corresponding operation process based on the operation information; wherein the operation result information includes result information indicating successful execution and result information indicating failed execution;

[0084] Display the result of the operation; or

[0085] Displays operation result information and prompt information, where the prompt information is used to prompt the operator to enter other instructions based on the displayed operation result information.

[0086] Specifically, the control method for an Internet device provided in this application is applied to a scenario where a smart home device interacts with a user. Assuming the terminal is a robot vacuum cleaner in a smart home device, that is, after the robot vacuum cleaner executes the corresponding operation process based on the operation information, it is assumed that the operation result information A obtained by the robot vacuum cleaner for executing the corresponding operation process based on the operation information is: the instruction to turn on the air conditioner in room A to 26°C was not completed (i.e., result information indicating execution failure), and the operation result information A is displayed; or, the operation result information A is displayed along with a prompt information indicating whether the instruction to turn on the air conditioner in room A to 26°C needs to be executed again. That is, the prompt information is used to prompt the operator (user) to enter another instruction based on the displayed operation result information. For example, based on the displayed result information indicating execution failure, the operator (user) can enter the instruction "Turn on the air conditioner in room A to 26°C" again to re-execute the processing flow of calling the pre-trained inference model to infer the instruction "Turn on the air conditioner in room A to 26°C". Among them, the operation result information in the embodiment of the present application includes result information indicating successful execution and result information indicating execution failure. Therefore, by constructing training samples including text data for describing the operation action flow and video data for describing the operation action flow, the initial model is trained in a targeted manner, so that the trained large model (inference model) can be used to infer various interactive instructions triggered by users, so that the accuracy of the inference results can be improved without affecting the user experience, and after completing a series of operations based on the inference results output by the model, the results or information to be confirmed are automatically fed back to the user, effectively improving the user experience.

[0087] This application also provides an application scenario, which applies the above-mentioned control method for an Internet device. Specifically, the application of the control method for an Internet device in this application scenario is as follows:

[0088] In the process of user interaction with the sweeping robot, the control method of the Internet device mentioned above can be used, such as Figure 3 As shown in , specifically including:

[0089] 1. Train a large model to enable it to simulate clicking on various apps.

[0090] 2. The large model inference process can run on the cloud or on the client side.

[0091] 3. Applications that call on large model capabilities run on systems that support APP operation, such as Android and Hongmeng.

[0092] 4. The motherboard running the above system can be on the base station or the robot user's mobile phone.

[0093] 5. Applications that call large model capabilities need to obtain system permissions and the permissions of the user's upcoming app (such as payment, storage, photography, account information, etc.).

[0094] 6. When a user sends a command (in the form of voice, image, text, etc.) to an application that invokes the big model's capabilities, the application sends the information command to the big model for understanding and reasoning, and then provides app operations that match the command requirements.

[0095] 7. After the operation is completed, feedback the results or information to be confirmed to the user.

[0096] The beneficial effects of the technical solution of this application include:

[0097] Breaking the ecological limitations of Mijia, Hongmeng, etc., the robot vacuum becomes the control center of the home by simulating human understanding and reasoning ability through large models to control the APP by clicks.

[0098] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0099] Based on the same inventive concept, embodiments of the present application also provide an internet device control device for implementing the aforementioned internet device control method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the following embodiments of the internet device control device device can be found in the aforementioned limitations of the internet device control method and will not be further elaborated here.

[0100] In one embodiment, Figure 6 As shown, a control device for an Internet device is provided, including: a calling module 602, a processing module 604 and an execution module 608, wherein:

[0101] The calling module 602 is used to call the pre-trained reasoning model in response to the triggered interaction instruction.

[0102] The processing module 604 is used to process the interaction instruction through the reasoning model to obtain a reasoning result corresponding to the interaction instruction; wherein the reasoning result includes operation information of an operation action flow related to the execution of the interaction instruction.

[0103] The execution module 608 is configured to execute a corresponding operation process based on the operation information; or send the operation information to a target Internet device so that the target Internet device executes a corresponding operation process based on the operation information.

[0104] In one embodiment, the interaction instructions include at least voice interaction instructions, text interaction instructions, and interaction instructions in video form, and the device also includes: a display module for displaying an interface for triggering interaction instructions; the interface includes a voice control, an instruction input box, and a video acquisition control; the calling module is also used to call a pre-trained inference model in response to the triggered voice interaction instruction when a voice interaction instruction is acquired through the voice control; or, call a pre-trained inference model in response to the triggered text interaction instruction when a text interaction instruction is acquired through the instruction input box; or, call a pre-trained inference model in response to the triggered video interaction instruction when a video interaction instruction is acquired through the video acquisition control.

[0105] In one embodiment, the device further includes: a conversion module for converting the voice interaction instruction into text description information when the interaction instruction is the voice interaction instruction; the processing module is further used to process the text description information through the inference model to obtain an inference result corresponding to the voice interaction instruction; the conversion module is further used to convert the video form interaction instruction into multimodal information containing images and text when the interaction instruction is the video form interaction instruction; the processing module is further used to process the multimodal information through the inference model to obtain an inference result corresponding to the video form interaction instruction.

[0106] In one embodiment, the device also includes: an acquisition module, which is used to obtain permission information of a third-party application corresponding to the identification information of the third-party application when the operation information includes the identification information of the third-party application; a startup module, which is used to start the third-party application when the permission information is a target value; the processing module is also used to jump to the page displayed by the third-party application, simulate the target trigger operation of the operator on the target control, so that the third-party application responds to the target trigger operation, executes the operation process corresponding to the target trigger operation, and generates result information of executing the operation process; and uses the result information as operation result information for executing the corresponding operation process based on the operation information.

[0107] In one embodiment, the device also includes: a detection module, which is used to obtain information on the failure to start the third-party application and detect the installation status of the third-party application; a search module, which is used to search for other applications with similar functions to the third-party application from a local database when the installation status of the third-party application is not installed, and use the other applications as the third-party application, so as to use the result information of executing the operation process generated by the other applications as the operation result information of executing the corresponding operation process based on the operation information.

[0108] In one embodiment, the present application also provides an intelligent cleaning device, which includes a robot body and a cleaning base station. The robot body or the cleaning base station is provided with a control device of an Internet device, and the control device includes: a calling module, which is used to call a pre-trained inference model in response to a triggered interaction instruction; a processing module, which is used to process the interaction instruction through the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the execution of the interaction instruction; an execution module, which is used to execute a corresponding operation process based on the operation information; or, send the operation information to a target Internet device so that the target Internet device executes a corresponding operation process based on the operation information.

[0109] Each module in the control device of the aforementioned intelligent cleaning device and internet-connected device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a computer device memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0110] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store control data of the Internet device. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a control method of an Internet device is implemented.

[0111] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0112] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0113] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0114] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0115] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0116] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0117] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for controlling an Internet device, characterized in that: The method comprises: In response to the triggered interaction instruction, the pre-trained inference model is called; Processing the interaction instruction through the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the execution of the interaction instruction; Execute the corresponding operation process based on the operation information; or The operation information is sent to the target Internet device, so that the target Internet device executes the corresponding operation process based on the operation information.

2. The method according to claim 1, characterized in that The interactive instructions include at least voice interactive instructions, text interactive instructions, and interactive instructions in video form; Before calling the pre-trained inference model in response to the triggered interaction instruction, the method further includes: Display an interface for triggering interactive commands; the interface includes a voice control, a command input box, and a video capture control; The calling of the pre-trained inference model in response to the triggered interaction instruction includes: In the case where a voice interaction instruction is collected through the voice control, in response to the triggered voice interaction instruction, a pre-trained inference model is called; or, In the case where a text interaction instruction is collected through the instruction input box, in response to the triggered text interaction instruction, a pre-trained inference model is called; or, In the case where an interactive instruction in video form is captured by the video capture control, a pre-trained inference model is called in response to the triggered interactive instruction in video form.

3. The method according to claim 2, characterized in that The processing of the interaction instruction by the inference model to obtain an inference result corresponding to the interaction instruction includes: In the case where the interaction instruction is the voice interaction instruction, converting the voice interaction instruction into text description information, processing the text description information through the inference model, and obtaining an inference result corresponding to the voice interaction instruction; or In the case where the interaction instruction is an interaction instruction in the form of the video, the interaction instruction in the form of the video is converted into multimodal information including images and texts, and the multimodal information is processed by the inference model to obtain an inference result corresponding to the interaction instruction in the form of the video.

4. The method according to claim 1, wherein The executing a corresponding operation process based on the operation information includes: When the operation information includes identification information of a third-party application, obtaining permission information for starting the third-party application corresponding to the identification information of the third-party application; When the permission information is a target value, starting the third-party application; Jumping to the page displayed by the third-party application, simulating the target trigger operation of the operator on the target control, so that the third-party application responds to the target trigger operation, executes the operation process corresponding to the target trigger operation, and generates result information of executing the operation process; The result information is used as operation result information of executing the corresponding operation process based on the operation information.

5. The method according to claim 4, characterized in that When the permission information is at the target value, after starting the third-party application, the method further includes: Obtaining information indicating failure to start the third-party application and detecting an installation status of the third-party application; When the installation status of the third-party application is not installed, other applications with similar functions to the third-party application are searched from the local database, and the other applications are used as the third-party applications, so that the result information of executing the operation process generated by the other applications is used as the operation result information of executing the corresponding operation process based on the operation information.

6. A control device for an Internet device, characterized in that: The device comprises: A calling module, configured to call a pre-trained inference model in response to a triggered interaction instruction; a processing module, configured to process the interaction instruction using the inference model to obtain an inference result corresponding to the interaction instruction; wherein the inference result includes operation information of an operation action flow related to the execution of the interaction instruction; An execution module is used to execute a corresponding operation process based on the operation information; or, to send the operation information to a target Internet device so that the target Internet device executes a corresponding operation process based on the operation information.

7. An intelligent cleaning device, characterized in that: It comprises a robot body and a cleaning base station, wherein the control device of the Internet device according to claim 6 is provided on the robot body or the cleaning base station.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images