Equipment control method and device based on multiple modes, equipment, medium and product

Abstract

The invention relates to an equipment control method and device based on multiple modes, electronic equipment and a storage medium, and relates to the technical field of intelligent control. The method comprises the following steps: acquiring a service request of a user; performing semantic analysis on the service request through a task planning model to obtain service sub-tasks; calling a corresponding device according to the service subtask to obtain a device execution result; performing multi-modal analysis on the service request and the equipment execution result through a result analysis model to obtain an analysis result; and replying a request response result to the user. According to the method, a service request is decomposed into a plurality of service subtasks through semantic analysis, and related equipment execution results are obtained through equipment. The service request and the equipment execution result are subjected to multi-modal analysis through the result analysis model, and the equipment execution result is subjected to targeted identification analysis on the basis of understanding the service intention of the user, so that the analysis result for the service request is obtained, and the interaction efficiency and the user experience are greatly improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent control technology, and more specifically, to a multimodal device control method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, smart home and IoT devices are widely used. With the popularization of these smart devices, users can easily remotely check device status or receive event alarms via mobile applications. Building on this, natural language interaction technologies, such as service assistants, allow users to control devices or query information through dialogue. However, existing service assistant functions are still relatively limited, mainly confined to executing preset control commands, lacking real-time perception and cognition of the physical environment, and unable to understand and answer unstructured questions posed by users. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a multimodal device control method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to a first aspect of the present disclosure, a multimodal device control method is provided, the method comprising: acquiring a user's service request; performing semantic analysis on the service request using a task planning model to obtain at least one service subtask; invoking a corresponding device to execute the service subtask according to the at least one service subtask to obtain an execution result of at least one device; performing multimodal analysis on the service request and the execution result of the at least one device using a result analysis model to obtain an analysis result; and replying a request response result to the user according to the analysis result.

[0005] In some exemplary embodiments of this disclosure, the step of performing semantic analysis on the service request through a task planning model to obtain at least one service subtask includes: inputting associated device information into the task planning model; the associated device information is used to represent device information of at least one device with an associated relationship; performing semantic analysis on the service request based on the associated device information to obtain the at least one service subtask; the service subtask includes at least: device type information, device identification information, and task information.

[0006] In some exemplary embodiments of this disclosure, the step of invoking a corresponding device to execute the service subtask according to the at least one service subtask and obtaining at least one device execution result includes: sending the service subtask to the corresponding device invocation module according to the device type information; generating a device invocation instruction according to the task information; sending the device invocation instruction to the corresponding device according to the device identification information; and receiving the device execution result returned by the device.

[0007] In some exemplary embodiments of this disclosure, the step of invoking a corresponding device to execute the service subtask according to the at least one service subtask and obtaining at least one device execution result includes: sending the at least one service subtask to a device invocation proxy module; determining the target service interface corresponding to each of the service subtasks; generating a service invocation instruction corresponding to the target service interface according to the service subtask; sending the service invocation instruction to the corresponding target service interface; and receiving the device execution result returned by the target service interface.

[0008] In some exemplary embodiments of this disclosure, the service subtask includes at least service type information; determining the target service interface corresponding to each service subtask includes: determining the target service interface corresponding to each service subtask based on the service type information.

[0009] In some exemplary embodiments of this disclosure, the device invokes the proxy module and the target service interface through an MCP protocol communication connection.

[0010] In some exemplary embodiments of this disclosure, the step of replying to the user with a request response result based on the analysis result includes: processing the analysis result through a response generation model to generate the request response result; and sending the request response result to the user.

[0011] In some exemplary embodiments of this disclosure, the method further includes: performing semantic analysis on the service request through the task planning model to obtain a conditional execution task; the conditional execution task includes at least: a trigger condition and a device execution task; and controlling the corresponding device to execute the device execution task in response to the analysis result satisfying the trigger condition.

[0012] In some exemplary embodiments of this disclosure, the step of invoking a corresponding device to execute the service subtask according to the at least one service subtask and obtaining at least one device execution result includes: sending device invocation confirmation information to the user according to the at least one service subtask; and in response to the user confirming the device invocation confirmation information, invoking a corresponding device to execute the service subtask according to the at least one service subtask and obtaining at least one device execution result.

[0013] According to a second aspect of the present disclosure, a multimodal device control apparatus is provided, comprising: a service request unit for acquiring a user's service request; a task planning unit for performing semantic analysis on the service request using a task planning model to obtain at least one service sub-task; a device execution unit for invoking a corresponding device to execute the service sub-task according to the at least one service sub-task to obtain at least one device execution result; a result analysis unit for performing multimodal analysis on the service request and the at least one device execution result using a result analysis model to obtain an analysis result; and a request response unit for replying a request response result to the user according to the analysis result.

[0014] In some exemplary embodiments of this disclosure, the task planning unit is further configured to input associated device information into the task planning model; the associated device information is used to represent device information of at least one device with an associated relationship; semantic analysis is performed on the service request based on the associated device information to obtain the at least one service subtask; the service subtask includes at least: device type information, device identification information, and task information.

[0015] In some exemplary embodiments of this disclosure, the device execution unit is further configured to send the service subtask to the corresponding device invocation module according to the device type information; generate a device invocation instruction according to the task information; send the device invocation instruction to the corresponding device according to the device identification information; and receive the device execution result returned by the device.

[0016] In some exemplary embodiments of this disclosure, the device execution unit is further configured to send the at least one service subtask to the device invocation proxy module; determine the target service interface corresponding to each of the service subtasks; generate a service invocation instruction corresponding to the target service interface according to the service subtask; send the service invocation instruction to the corresponding target service interface; and receive the device execution result returned by the target service interface.

[0017] In some exemplary embodiments of this disclosure, the task planning unit is further configured to perform semantic analysis on the service request through the task planning model to obtain a conditional execution task; the conditional execution task includes at least a trigger condition and a device execution task; the device execution unit is configured to control the corresponding device to execute the device execution task in response to the analysis result satisfying the trigger condition.

[0018] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to: implement the multimodal device control method described in any one of the present disclosures.

[0019] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform any of the multimodal device control methods described in the present disclosure.

[0020] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the multimodal device control methods described in the present disclosure.

[0021] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This disclosure provides a multimodal device control method that performs semantic analysis on service requests using a task planning model to obtain service subtasks; it then invokes the corresponding device to execute the task based on the service subtask, obtaining the device execution result; finally, it performs multimodal analysis on the service request and the device execution result using a result analysis model, obtaining the analysis result; and based on the analysis result, it replies to the user with a request response. This method decomposes service requests into multiple service subtasks through semantic analysis and obtains relevant device execution results through corresponding devices. By performing multimodal analysis on service requests and device execution results using a result analysis model, it understands the user's service intent and performs targeted identification and analysis of the device execution results, thereby obtaining analysis results specific to the service request, greatly improving interaction efficiency and user experience.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0024] Figure 1This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 1 .

[0025] Figure 2 This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 2 .

[0026] Figure 3 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 1 .

[0027] Figure 4 This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 3 .

[0028] Figure 5 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 2 .

[0029] Figure 6 This is a flowchart illustrating a request-response result process according to an exemplary embodiment of the present disclosure.

[0030] Figure 7 This is a flowchart illustrating a conditional task processing procedure according to an exemplary embodiment of the present disclosure.

[0031] Figure 8 This is a block diagram illustrating a multimodal device control apparatus according to an exemplary embodiment of the present disclosure.

[0032] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0033] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0034] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] The steps of the method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples.

[0036] Figure 1 This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 1 The method described in this embodiment can be applied to electronic devices, including smart speakers, smartphones, and smart tablet terminals. For example... Figure 1 As shown, in some embodiments, the multimodal device control method provided in this disclosure may include the following steps.

[0037] In step S110, the user's service request is obtained.

[0038] In this embodiment of the disclosure, a service request (Query) initiated by the user is obtained. This service request can be a control request or a query request based on at least one device. The service request can take the form of an unstructured question that is closer to natural expression. For example, "Can you check what the cat in the living room is doing?" or "Is there a package at the door?". The service request can be voice information or text information.

[0039] In an exemplary embodiment, the service request is received through a control device. This control device is used to control various smart devices. The control device may be a smart speaker, smartphone, or smart tablet terminal, etc. The control device may include a dialogue service module for enabling dialogue interaction with the user. The control device receives the user's voice request through this dialogue service module and converts the voice request into a corresponding text-based service request.

[0040] In step S120, the service request is semantically analyzed using a task planning model to obtain at least one service subtask.

[0041] In this embodiment, a pre-trained task planning model performs natural language understanding on the service request to identify the user's intent and the related device objects and operational requirements. The task planning model decomposes the service request into multiple tasks based on the understanding of the user's intent and generates corresponding structured service subtasks. These service subtasks are predefined structured task instructions. For example, a service subtask could be a structured function call. The task planning model can be deployed in the control device or on a cloud server communicatively connected to the control device.

[0042] In an exemplary embodiment, the service subtask can be based on the device to be controlled. The task planning model pre-acquires associated device information. This associated device information represents the device information of at least one device with an associated relationship. The devices in the associated device information are pre-bound and associated with the user, making them available for invocation. The device information may include one or more of the following: device type, device ID, device location, and device function. The task planning model performs semantic analysis on the service request based on the associated device information, thereby generating a service subtask corresponding to the specific device.

[0043] In an exemplary embodiment, taking the service request "Help me see what the cat in the living room is doing?" as an example, the task planning model determines, through natural language understanding, that the service request requires calling the living room camera to take a picture, and generates the corresponding calling function capture_image(living_room_camera_001). Here, capture_image() represents calling the image capture function; living_room_camera represents the living room camera; and 001 is the device ID.

[0044] In an exemplary embodiment, taking the service request "Is there a package at the door?" as an example, the task planning model determines, through natural language understanding, that the service request requires calling the door camera to take a picture, and generates the corresponding calling function capture_image(door_camera_002). Here, capture_image() represents calling the image capture function; door_camera represents the door camera; and 002 is the device ID.

[0045] In an exemplary embodiment, the service subtasks generated by the task planning model can also be standardized device invocation requests generated based on the service request. These device invocation requests do not explicitly specify the device being invoked; instead, they express the required device type and function using a standardized request format. For example, taking the service request "Help me see what the cat in the living room is doing?" as an example, the task planning model generates a corresponding device invocation request `func(camera, living_room, capture_image)` by performing natural language understanding on the service request. This task planning model does not need to know the deployment location and function of each backend device in advance; it only needs to determine the required device type as `camera`, the device location as `living_room`, and the device function as `capture_image` based on the service request. Other backend devices are then responsible for routing the device invocation request to the corresponding device for execution. In this way, the task planning model can be decoupled from the specific backend smart devices.

[0046] In step S130, according to the at least one service subtask, the corresponding device is invoked to execute the service subtask, and the execution result of at least one device is obtained.

[0047] In this embodiment, the multiple standardized service subtasks generated by the aforementioned task planning model are transformed into control instructions that can be executed by specific physical devices. Each control instruction is sent to its corresponding device to control the device to perform the corresponding functional action. The relevant devices execute the service subtasks respectively, obtaining corresponding device execution results. The device execution results fed back by each device are collected and summarized to form at least one device execution result that satisfies the service request.

[0048] In an exemplary embodiment, the task planning model can return the generated multiple service subtasks to the dialogue service module in the control device, which then sends each subtask to the corresponding device for invocation. In this scheme, the dialogue service module acts as the central hub for the entire user service control task, responsible for invoking and forwarding relevant intermediate information. Alternatively, the task planning model can directly send the generated multiple service subtasks to the corresponding devices for invocation.

[0049] In an exemplary embodiment, the control device further includes multiple device invocation modules. Each device invocation module is responsible for invoking a device of a specific device type. Each device invocation module generates corresponding control commands based on the service subtask. For example, a user may have multiple different types of devices bound to their device, such as cameras, speakers, and temperature and humidity sensors. For each type of device, separate modules are provided for camera invocation, speaker invocation, and temperature and humidity sensor invocation. The camera invocation module manages the user's various cameras, such as a living room camera and a doorway camera. Based on the aforementioned service subtask `capture_image(living_room_camera_001)` for the camera, the camera invocation module generates corresponding control commands for the living room camera and sends them to the living room camera to control it to perform a photo-taking task.

[0050] In an exemplary embodiment, since some device calls require user confirmation before execution, a device call confirmation process may be included before sending the service subtask to each device. This may include the following steps.

[0051] Based on the at least one service subtask, send device call confirmation information to the user; In response to the user's confirmation of the device call confirmation information, the corresponding device is invoked to execute the service sub-task according to the at least one service sub-task, and the execution result of the at least one device is obtained.

[0052] In this embodiment of the disclosure, a device call confirmation message is generated and sent to the user based on the service sub-tasks generated by the task planning model. This device call confirmation message is used to confirm with the user whether permission is granted to call the corresponding device. If the user confirms the device call confirmation message, the corresponding device is called to execute the service sub-task according to the aforementioned process. If the user does not confirm the device call confirmation message, the subsequent process is terminated.

[0053] In step S140, the service request and the execution result of the at least one device are analyzed using a result analysis model to obtain the analysis results.

[0054] In this embodiment, a pre-trained result analysis model analyzes at least one device execution result collected from various devices based on the aforementioned service request. This result analysis model is a multimodal model capable of simultaneously processing and understanding multiple modalities of data, including text, images, video, and audio. Based on its multimodal processing capabilities, the model first deeply analyzes the service request (audio or text information) to understand the user's intent and focus. Then, guided by this understanding, it scans, identifies, and analyzes multiple device execution results (audio, video, and other multimedia information) from various devices. Finally, it aligns and fuses the relevant information in the device execution results with the intent expressed in the service request to obtain the analysis result specific to the service request. Therefore, this result analysis model does not simply concatenate multiple device execution results; rather, it uses multimodal processing capabilities to perform complex cognitive reasoning based on the user intent expressed in the service request to obtain the analysis result. This analysis result can be represented in a structured semantic manner, potentially including identified objects, attributes, behaviors, and the relationships between this information and the service request.

[0055] In an exemplary embodiment, a service request of "Help me see what the cat in the living room is doing?" is taken as an example. The result analysis model clarifies the user's intent as "What is the cat doing?" based on the service request. The result analysis model first identifies the target object "cat" in the photos fed back by the living room camera, and then infers the cat's behavior (such as "sleeping" or "licking its fur") by analyzing the cat's posture in the photos, finally obtaining a standardized analysis result {object: cat; location: sofa; behavior: sleeping}.

[0056] In an exemplary embodiment, the result analysis model can return the generated analysis results to the dialogue service module in the control device, and the dialogue service module can reply to the user based on the analysis results.

[0057] In an exemplary embodiment, the result analysis model can receive the execution results of each device sent by the aforementioned device invocation module, analyze the execution results of each device based on the service request, and return the obtained analysis results to the device invocation module, which then returns them to the dialogue service module.

[0058] It should be noted that the aforementioned task planning model and result analysis model can be independent artificial intelligence models trained separately, or they can be sub-models of the same model. The architectural relationships between related models can be adjusted according to actual needs, and all of these should be considered within the scope of protection of this disclosure.

[0059] In step S150, based on the analysis results, a response to the request is sent to the user.

[0060] In this embodiment of the disclosure, based on the analysis results obtained from the result analysis model for the service request, a corresponding request response result is sent to the user to reply to the user's service request.

[0061] In an exemplary embodiment, the result analysis model returns the analysis results in a standardized format to the dialogue service module. The dialogue service module then converts the analysis results into a request-response result conforming to natural language usage and replies to the user via voice or text. For example, the standardized analysis result generated by the result analysis model is {Object: Cat; Location: Sofa; Behavior: Sleeping}. The dialogue service module converts this analysis result into the text "Your cat is sleeping on the sofa" in natural language usage.

[0062] In an exemplary embodiment, the dialogue service module can invoke a pre-trained response generation model to transform the standardized analysis results into request-response results that conform to natural language habits. This response generation model focuses on the refinement, organization, and generation of language, combining the analysis results with the service request as contextual background to generate fluent, natural, and conversational request-response results.

[0063] It should be noted that the response generation model can be an independently trained artificial intelligence model, or it can be a model that integrates the aforementioned task planning model and / or result analysis model. The architectural relationships between related models can be adjusted according to actual needs, and all of these should be considered within the scope of protection of this disclosure.

[0064] This disclosure provides a multimodal device control method that performs semantic analysis on service requests using a task planning model to obtain service subtasks; it then invokes the corresponding device to execute the task based on the service subtask, obtaining the device execution result; finally, it performs multimodal analysis on the service request and the device execution result using a result analysis model, obtaining the analysis result; and based on the analysis result, it replies to the user with a request response. This method decomposes service requests into multiple service subtasks through semantic analysis and obtains relevant device execution results through corresponding devices. By performing multimodal analysis on service requests and device execution results using a result analysis model, it understands the user's service intent and performs targeted identification and analysis of the device execution results, thereby obtaining analysis results specific to the service request, greatly improving interaction efficiency and user experience.

[0065] Figure 2 This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 2 . Figure 3 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 1 .

[0066] In this embodiment of the disclosure, Figure 2 In the multimodal device control method shown, steps S210, S240, and S250 are respectively related to... Figure 1 Steps S110, S140, and S150 in the multimodal device control method shown correspond to each other and will not be repeated here.

[0067] In some exemplary embodiments, in Figure 1 Based on the multimodal device control method shown, the aforementioned step S120 may include the following steps.

[0068] In step S221, the associated device information is input into the task planning model; the associated device information is used to represent the device information of at least one device with an associated relationship.

[0069] In this embodiment, associated device information is input into the task planning model. This associated device information can be pre-stored in the task planning model or input during the task planning model processing. The associated device information represents the device information of at least one device with an associated relationship. The devices in the associated device information are pre-bound devices that are associated with the user and can be invoked. The device information may include one or more of the following: device type, device ID, device location, and device function.

[0070] In step S222, semantic analysis is performed on the service request based on the associated device information to obtain at least one service subtask; the service subtask includes at least: device type information, device identification information, and task information.

[0071] In this embodiment, the task planning model understands the user's intent based on semantic analysis of the service request and decomposes the service request into one or more decomposed tasks. According to the task objectives of each decomposed task, the model searches for devices and device information matching the corresponding decomposed task in the associated device information. The task planning model generates corresponding service sub-tasks based on the respective tasks and the matching devices and device information. Each service sub-task includes at least: device type information, device identification information, and task information. The device type information indicates the device type of the called device. The device identification information indicates the identification information of the called device. The task information indicates the execution task of the called device. The called device may have multiple device functions, such as taking photos or recording videos. The task information can indicate one or more device functions of the called device.

[0072] In an exemplary embodiment, the task planning model pre-acquires information about associated devices, including multiple cameras bound to the user. For example, a living room camera {device type: camera; device ID: 001; device location: living room; device function: taking photos, recording videos}; and a doorway camera {device type: camera; device ID: 002; device location: doorway; device function: taking photos, recording videos}.

[0073] Taking the service request "Help me see what the cat in the living room is doing?" as an example, the task planning model uses natural language understanding to determine that the service request requires calling the living room camera to take a picture, and generates the corresponding call function capture_image(living_room_camera_001). Here, capture_image() indicates calling the image capture function; camera indicates that the device type is a camera; living_room indicates that the device location is the living room; 001 indicates the device ID.

[0074] In some exemplary embodiments, in Figure 1 Based on the multimodal device control method shown, the aforementioned step S130 may include the following steps.

[0075] In step S231, the service subtask is sent to the corresponding device invocation module according to the device type information.

[0076] In this embodiment, the control device includes multiple device invocation modules. Each device invocation module is responsible for invoking a device of a specific device type. Each module generates corresponding control commands based on the service subtask. For example, a user may have multiple different types of devices bound to their account, such as cameras, speakers, and temperature and humidity sensors. Separate invocation modules are provided for each device type: a camera invocation module, a speaker invocation module, and a temperature and humidity sensor invocation module. Each device invocation module manages one or more devices according to the device type it is responsible for. For example, the camera invocation module manages both a living room camera and a doorway camera.

[0077] In this embodiment of the disclosure, the service subtask is sent to the corresponding device invocation module according to the device type information in the service subtask, so that the corresponding device can be invoked to execute the service subtask through the device invocation module.

[0078] In step S232, a device call instruction is generated based on the task information.

[0079] In this embodiment, the service subtasks generated by the task planning model are standardized task expressions. Since the control instruction formats of different hardware devices vary, the device invocation module needs to generate corresponding device invocation instructions that can be executed by the device based on the task information in the service subtask.

[0080] In step S233, the device call command is sent to the corresponding device according to the device identification information.

[0081] In this embodiment of the disclosure, the device invocation module sends the device invocation instruction to the corresponding device according to the device identification information in the service subtask, so as to control the device to execute the service subtask.

[0082] In step S234, the device execution result returned by the device is received.

[0083] In this embodiment of the disclosure, after completing the relevant task, the called device feeds back the obtained device execution result to the device calling module. The various device calling modules collect and summarize the multiple device execution results fed back by each device.

[0084] Figure 3 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 1 .

[0085] In step S301, the dialogue service module receives a service request sent by the user. If the service request is in voice format, it is converted into corresponding text information. The "service request" mentioned below refers to the corresponding text information converted by the dialogue service module.

[0086] In step S302, the dialogue service module inputs the received service request as a parameter into the task planning model. The task planning model performs semantic analysis on the semantic request, understands the user's request intent, and performs task planning, generating multiple corresponding service sub-tasks.

[0087] In step S303, the task planning model returns the generated multiple service subtasks to the dialogue service module.

[0088] In step S304, the dialogue service module invokes the corresponding device invocation module based on the service subtask returned by the task planning model. During the invocation, the dialogue service module transmits the service subtask and service request to the device invocation module.

[0089] In step S305, after receiving the service subtask, the device invocation module generates a device invocation instruction for the corresponding device and controls the corresponding device to perform the corresponding task action.

[0090] In step S306, the device invocation module receives the device execution result returned by the corresponding device after performing the corresponding task action.

[0091] In step S307, the device invocation module sends two different modalities of information, namely the service request (text information) and the related device execution result (such as multimedia information), as parameters to the result analysis model for comprehensive analysis.

[0092] In step S308, the result analysis model is a multimodal model capable of simultaneously processing and understanding various modalities of data, such as text, images, video, and audio. This result analysis model analyzes the device execution result based on the user intent expressed in the service request, obtains the analysis result specific to the service request, and returns the analysis result to the device invocation module.

[0093] In step S309, the device calling module returns the analysis results obtained from the result analysis model to the dialogue service module, which acts as the overall scheduler.

[0094] In step S310, the analysis results received by the dialogue service module are structured data. To provide the user with a response that conforms to natural language expression habits, the dialogue service module resends the service request and analysis results to the task planning model. This task planning model also includes a built-in response generation model. This response generation model processes the analysis results based on the service request to generate a response that conforms to natural language expression habits.

[0095] In step S311, the task planning model returns the request response result generated by the response generation model to the dialogue service module.

[0096] In step S312, the dialogue service module replies to the user in voice or text form based on the response to the request, thus completing the question-and-answer process for the user's service request.

[0097] The multimodal device control method disclosed herein pre-inputs associated device information into a task planning model, enabling the model to perform contextual reasoning based on an understanding of user intent and device information related to the user, thereby generating more accurate service sub-task plans. Simultaneously, by setting up device invocation modules for different device types, the system can flexibly accommodate various heterogeneous devices, enhancing overall scalability and maintainability.

[0098] Figure 4 This is a flowchart illustrating a multimodal device control method according to an exemplary embodiment of the present disclosure. Figure 3 . Figure 5 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 2 .

[0099] In this embodiment of the disclosure, Figure 4 In the multimodal device control method shown, steps S410, S420, S440, and S450 are respectively related to... Figure 1 The steps S110, S120, S140, and S150 in the multimodal device control method shown correspond to each other and will not be repeated here.

[0100] In some exemplary embodiments, in Figure 1 Based on the multimodal device control method shown, the aforementioned step S130 may include the following steps.

[0101] In step S431, the at least one service subtask is sent to the device invocation agent module.

[0102] In this embodiment, the task planning model is decoupled from the specific backend smart devices. The task planning model does not need to pre-obtain the associated device information of the backend bound devices, and the generated service subtasks do not need to specify the device identifier information for executing the task. The task planning model only needs to generate standardized service subtasks based on service requests. These service subtasks do not explicitly call the specific device, but rather express the required device type and function using a standardized request format. The device call proxy module deployed on the backend is responsible for routing the service subtask to the corresponding device for execution.

[0103] In an exemplary embodiment, such as Figure 5As shown, the task planning model returns at least one generated service subtask to the dialogue service module. The dialogue service module, acting as the overall scheduler, sends this at least one service subtask to the device invocation proxy module, which is responsible for routing the service subtask to the corresponding device for execution.

[0104] In step S432, the target service interface corresponding to each of the service subtasks is determined.

[0105] In this embodiment, the device invokes a proxy module that communicates with multiple service interfaces to manage task scheduling for each service interface. The proxy module matches a suitable service interface from among the multiple service interfaces based on each service sub-task, using it as the target service interface for that sub-task. In this way, the proxy module acts as a proxy, routing each service sub-task to the appropriate service interface for execution. By setting up this proxy module, the task planning model is decoupled from the specific backend smart devices. The task planning model does not need to concern itself with the specific implementation details and diversity of the backend services, while also providing scalability for accessing different service interfaces.

[0106] In an exemplary embodiment, the service subtask includes at least service type information. This service type information indicates the service type corresponding to the service subtask. When allocating service interfaces, the device call proxy module determines the target service interface corresponding to each service subtask based on the service type information. Here, as described above... Figure 2 , 3 The embodiments shown differ from those described above. In the former embodiments, the task planning model generates and allocates the service sub-tasks with a specific execution device as the target. However, in this embodiment, the task planning model and the device invocation proxy module generate and allocate the service sub-tasks with the service type as the target. In practical applications, service sub-task allocation can be based on either the device or the service type, depending on the needs of the application scenario.

[0107] In an exemplary embodiment, the service interfaces connected to the proxy module by the device may include at least one of the following: a summary service interface, a rule service interface, and a query service interface. The summary service interface, corresponding to the summary service type, is used to handle information summary requests. The rule service interface, corresponding to the rule service type, is used to create or manage automation rules. The query service interface, corresponding to the query service type, is used to handle query tasks that require understanding and analyzing data.

[0108] In an exemplary embodiment, the device invocation proxy module communicates with each service interface via the MCP (Model Context Protocol). The MCP protocol is a standardized communication interface protocol used to establish secure, bidirectional connections between large language models and external data sources. The device invocation proxy module acts as an MCP Client, and each service interface acts as an MCP Server. This solution provides plug-and-play scalability by employing the MCP protocol. Thanks to the standardized MCP protocol and unified MCP client gateway, other third-party service interfaces can seamlessly integrate with the device invocation proxy module. This design significantly reduces the complexity and development difficulty of integrating new features, giving the system architecture strong scalability.

[0109] In step S433, a service invocation instruction corresponding to the target service interface is generated based on the service subtask.

[0110] In this embodiment, since different service interfaces may adopt different service protocols, the device invocation proxy module generates a service invocation instruction adapted to the target service interface based on the service subtask and the determined target service interface. Based on this, the device invocation proxy module realizes protocol conversion between the task planning model and specific service interfaces. The task planning model does not need to concern itself with the protocol types supported by each backend service interface, further decoupling the task planning model from the specific backend smart devices.

[0111] In step S434, the service call instructions are sent to the corresponding target service interfaces.

[0112] In this embodiment of the disclosure, the device call proxy module sends the generated service call instructions to the corresponding target service interface to control the device connected to the service interface to execute the service sub-task.

[0113] In step S435, the device execution result returned by the target service interface is received.

[0114] In this embodiment, after completing the relevant task, the called device feeds back the obtained device execution result to the device call proxy module through the corresponding service interface. The device call proxy module collects and summarizes the multiple device execution results fed back by each device.

[0115] Figure 5 This is a schematic diagram of a multimodal device control process illustrated according to an exemplary embodiment of the present disclosure. Figure 2 .

[0116] In step S501, the dialogue service module receives a service request sent by the user. If the service request is in voice format, it is converted into corresponding text information. The "service request" mentioned below refers to the corresponding text information converted by the dialogue service module.

[0117] In step S502, the dialogue service module inputs the received service request as a parameter into the task planning model. The task planning model performs semantic analysis on the semantic request, understands the user's request intent, and performs task planning, generating multiple corresponding service sub-tasks.

[0118] In step S503, the task planning model returns the generated multiple service subtasks to the dialogue service module.

[0119] In step S504, the dialogue service module, acting as the overall scheduler, sends multiple service subtasks returned by the task planning model to the device invocation proxy module. The information sent by the dialogue service module to the device invocation proxy module includes: service subtasks and service requests.

[0120] In step S505, after receiving the service subtask, the device invocation proxy module matches a suitable target service interface from multiple service interfaces based on each service subtask. The device invocation proxy module then generates a service invocation instruction adapted to the execution of the target service interface based on the service subtask and the determined target service interface. The generated service invocation instructions are then sent to the corresponding target service interfaces.

[0121] In step S506, each target service interface executes its corresponding service subtask and obtains the corresponding device execution result. Each target service interface sends information in two different modalities—service request (text information) and related device execution result (such as multimedia information)—as parameters to the result analysis model for comprehensive analysis.

[0122] In step S507, the result analysis model is a multimodal model capable of simultaneously processing and understanding various modalities of data, such as text, images, video, and audio. This result analysis model analyzes the device's execution results based on the user intent expressed in the service request, obtains the analysis results specific to the service request, and returns these results to the corresponding target service interface.

[0123] In step S508, each target service interface returns the analysis results obtained from the result analysis model to the device call agent module.

[0124] In step S509, the device calls the agent module to further return the analysis result to the dialogue service module, which acts as the overall scheduler.

[0125] In step S510, the dialogue service module processes the analysis result based on the service request, generating a request response result that conforms to natural language expression habits. This request response result is then sent back to the user in voice or text form, completing the question-and-answer process for the user's service request.

[0126] The multimodal device control method disclosed herein decouples the task planning model from the specific backend smart devices by setting up a device invocation proxy module. The task planning model does not need to concern itself with the specific implementation details and diversity of backend services, while providing scalability for accessing different service interfaces. Thanks to the standardized MCP protocol and unified MCP client gateway, other third-party service interfaces can seamlessly connect to the device invocation proxy module. This design greatly reduces the complexity and development difficulty of integrating new functions, giving the system architecture strong scalability.

[0127] Figure 6 This is a flowchart illustrating a request-response result process according to an exemplary embodiment of the present disclosure. In this embodiment of the disclosure, in Figure 1 Based on the multimodal device control method shown, the aforementioned step S150 may include the following steps.

[0128] In step S610, the analysis results are processed by the response generation model to generate the request response result.

[0129] In this embodiment, since the analysis results generated by the result analysis model are typically in a standardized format, directly outputting data that does not conform to the user's language habits, this disclosure also includes a pre-trained response generation model to transform the standardized analysis results into request-response results that conform to natural language habits. This response generation model focuses on language refinement, organization, and generation; it combines the analysis results with the service request as contextual background to generate fluent, natural, and conversational request-response results.

[0130] In an exemplary embodiment, in addition to inputting the processed analysis results as parameters into the response generation model, the service request can also be input as a parameter into the response generation model. The response generation model can infer the user's request intent based on semantic analysis of the service request, and then process the analysis results according to the user's request intent to generate a request response result that is tailored to the user's request intent and conforms to dialogue habits.

[0131] It should be noted that the response generation model can be an independently trained artificial intelligence model, or it can be a model that integrates the aforementioned task planning model and / or result analysis model. The architectural relationships between related models can be adjusted according to actual needs, and all of these should be considered within the scope of protection of this disclosure.

[0132] In step S620, the request response result is sent to the user.

[0133] In this embodiment of the disclosure, the dialogue service module replies to the user in voice or text form based on the request response result, thus completing the question-and-answer process for the user's service request.

[0134] The multimodal device control method disclosed herein transforms the standardized analysis results output by the result analysis module into fluent, natural request-response results that conform to human dialogue habits by introducing a response generation model, which significantly improves the final user experience and the effectiveness of information delivery.

[0135] Figure 7 This is a flowchart illustrating a conditional task processing procedure according to an exemplary embodiment of the present disclosure. In this embodiment, in... Figure 1 Based on the multimodal device control method shown, the method may further include the following steps.

[0136] In step S710, the service request is semantically analyzed using the task planning model to obtain a conditional execution task; the conditional execution task includes at least: a trigger condition and a device execution task.

[0137] In this embodiment, user service requests may include not only queries about the current environmental state but also the setting of execution rules based on certain conditions met by the environmental state. For example, "Turn off the lights when no one is in the living room" or "Check if the fan is off; if not, turn it off." These service requests often include preset trigger conditions, such as "When no one is in the living room" or "Is the fan off?" They also include control instructions for the device after these preset conditions are met, such as "Turn off the lights" or "Turn off (the fan)." For such rule-setting service requests, this disclosure uses a task planning model to perform semantic analysis on the service request to understand the user intent expressed by the service request. When the task planning model identifies that the user intent includes such rule-setting requests, a conditional execution task is established. This conditional execution task includes at least: a trigger condition and a device execution task. The trigger condition refers to the preset trigger condition identified based on the service request. The device execution task refers to the execution task of a specified device that meets the trigger condition. Through the understanding of the user intent by the task planning model, the association between the trigger condition and the device execution task is established, forming the conditional execution task.

[0138] In step S720, in response to the analysis result satisfying the trigger condition, the corresponding device is controlled to execute the device execution task.

[0139] In this embodiment of the disclosure, based on the analysis results obtained from the aforementioned result analysis model, it is determined whether the analysis results satisfy the triggering conditions in the conditional execution task. In response to the analysis results satisfying the triggering conditions, the device execution task is executed based on the association between the triggering conditions established by the conditional execution task and the device execution task. That is, the corresponding device is controlled to execute the corresponding task according to the device execution task.

[0140] For example, if the analysis results of the result analysis model meet the condition that there is no one in the living room, the corresponding device will be triggered to execute a task and control the corresponding lighting switch in the living room to be turned off.

[0141] For example, if the analysis results of the result analysis model satisfy the condition that the fan is in the on state, then the corresponding device is triggered to execute a task to control and shut down the fan.

[0142] The multimodal device control method disclosed herein provides an implementation method for users to set device control rules based on the understanding of user intent through a task planning model. By binding the "trigger condition" with the subsequent "device execution task", the system can automatically drive the corresponding device to perform the specified operation when the analysis results output by the result analysis model meet the preset conditions. This facilitates users in setting control rules for smart devices and greatly enhances the practical value and scenario adaptability of smart home systems.

[0143] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0144] Figure 8 This is a block diagram illustrating a multimodal device control apparatus according to an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be applied to electronic devices, including smart speakers, smartphones, and smart tablet terminal devices. Figure 8 As shown, the multimodal device control device 800 may include: a service request unit 810, a task planning unit 820, a device execution unit 830, a result analysis unit 840, and a request response unit 850.

[0145] Service request unit 810 is used to obtain the user's service request.

[0146] The task planning unit 820 is used to perform semantic analysis on the service request through a task planning model to obtain at least one service subtask.

[0147] The device execution unit 830 is used to call the corresponding device to execute the service sub-task according to the at least one service sub-task, and obtain the execution result of at least one device.

[0148] The result analysis unit 840 is used to perform multimodal analysis on the service request and the execution result of the at least one device through the result analysis model to obtain the analysis results.

[0149] The request response unit 850 is used to reply to the user with a request response result based on the analysis result.

[0150] In some exemplary embodiments of this disclosure, the task planning unit 820 is further configured to input associated device information into the task planning model; the associated device information is used to represent device information of at least one device with an associated relationship; semantic analysis is performed on the service request based on the associated device information to obtain the at least one service subtask; the service subtask includes at least: device type information, device identification information and task information.

[0151] In some exemplary embodiments of this disclosure, the device execution unit 830 is further configured to send the service subtask to the corresponding device invocation module according to the device type information; generate a device invocation instruction according to the task information; send the device invocation instruction to the corresponding device according to the device identification information; and receive the device execution result returned by the device.

[0152] In some exemplary embodiments of this disclosure, the device execution unit 830 is further configured to send the at least one service subtask to the device invocation proxy module; determine the target service interface corresponding to each of the service subtasks; generate a service invocation instruction corresponding to the target service interface according to the service subtask; send the service invocation instruction to the corresponding target service interface; and receive the device execution result returned by the target service interface.

[0153] In some exemplary embodiments of this disclosure, the service subtask includes at least service type information. The device execution unit 830 is further configured to have the device invocation proxy module determine the target service interface corresponding to each service subtask based on the service type information.

[0154] In some exemplary embodiments of this disclosure, the device invokes the proxy module and the target service interface through an MCP protocol communication connection.

[0155] In some exemplary embodiments of this disclosure, the request response unit 850 is further configured to process the analysis results through a response generation model to generate the request response result; and send the request response result to the user.

[0156] In some exemplary embodiments of this disclosure, the task planning unit 820 is further configured to perform semantic analysis on the service request through the task planning model to obtain a conditional execution task; the conditional execution task includes at least a trigger condition and a device execution task. The device execution unit 830 is further configured to control the corresponding device to execute the device execution task in response to the analysis result satisfying the trigger condition.

[0157] In some exemplary embodiments of this disclosure, the device execution unit 830 is further configured to send device call confirmation information to the user according to the at least one service subtask; in response to the user confirming the device call confirmation information, the device executes the service subtask according to the at least one service subtask and obtains the execution result of the at least one device.

[0158] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0159] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0160] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0161] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0162] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0163] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0164] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0165] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0166] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0167] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in position of device 900 or a component of device 900, the presence or absence of user contact with device 900, orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0168] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 916 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0169] In some embodiments of this disclosure, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0170] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions that can be executed by a processor 920 of device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0171] In some embodiments of this disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to perform the above-described method.

[0172] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0173] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A device control method based on multimodal operation, characterized in that, The method includes: Obtain the user's service request; The service request is semantically analyzed using a task planning model to obtain at least one service subtask. Based on the at least one service subtask, the corresponding device is invoked to execute the service subtask, and the execution result of at least one device is obtained; The service request and the execution result of the at least one device are analyzed using a result analysis model to obtain the analysis results; Based on the analysis results, a response is sent to the user.

2. The method according to claim 1, characterized in that, The step of performing semantic analysis on the service request using a task planning model to obtain at least one service sub-task includes: Input the associated device information into the task planning model; the associated device information is used to represent the device information of at least one device with an associated relationship; Based on the associated device information, semantic analysis is performed on the service request to obtain at least one service subtask; the service subtask includes at least: device type information, device identification information, and task information.

3. The method according to claim 2, characterized in that, The step of invoking the corresponding device to execute the service subtask according to the at least one service subtask, and obtaining the execution result of at least one device, includes: Based on the device type information, the service subtask is sent to the corresponding device invocation module; Based on the task information, generate a device call command; Based on the device identification information, the device call command is sent to the corresponding device; Receive the execution result returned by the device.

4. The method according to claim 1, characterized in that, The step of invoking the corresponding device to execute the service subtask according to the at least one service subtask, and obtaining the execution result of at least one device, includes: Send the at least one service subtask to the device call agent module; Determine the target service interface corresponding to each of the aforementioned service subtasks; Based on the service subtask, generate a service invocation instruction corresponding to the target service interface; The service invocation commands are sent to the corresponding target service interfaces respectively; Receive the device execution result returned by the target service interface.

5. The method according to claim 4, characterized in that, The service subtask shall include at least service type information; The step of determining the target service interface corresponding to each of the service subtasks includes: Based on the service type information, the target service interface corresponding to each service subtask is determined.

6. The method according to claim 4, characterized in that, The device invokes the proxy module and the target service interface through the MCP protocol for communication connection.

7. The method according to claim 1, characterized in that, The step of replying to the user with a request response result based on the analysis results includes: The analysis results are processed by the response generation model to generate the request response result; Send the response result of the request to the user.

8. The method according to claim 1, characterized in that, The method further includes: The service request is semantically analyzed using the task planning model to obtain a conditional execution task; the conditional execution task includes at least: a triggering condition and a device execution task. In response to the analysis result satisfying the triggering condition, the corresponding device is controlled to execute the device execution task.

9. The method according to claim 1, characterized in that, The step of invoking the corresponding device to execute the service subtask according to the at least one service subtask, and obtaining the execution result of at least one device, includes: Based on the at least one service subtask, send device call confirmation information to the user; In response to the user's confirmation of the device call confirmation information, the corresponding device is invoked to execute the service sub-task according to the at least one service sub-task, and the execution result of the at least one device is obtained.

10. A multimodal device control apparatus, characterized in that, include: The service request unit is used to obtain the user's service request; The task planning unit is used to perform semantic analysis on the service request through a task planning model to obtain at least one service subtask. The device execution unit is configured to invoke the corresponding device to execute the service sub-task according to the at least one service sub-task, and obtain at least one device execution result; The result analysis unit is used to perform multimodal analysis on the service request and the execution result of the at least one device through a result analysis model to obtain analysis results; The request response unit is used to reply to the user with a request response result based on the analysis results.

11. The apparatus according to claim 10, characterized in that, The task planning unit is further configured to input associated device information into the task planning model; the associated device information is configured to represent device information of at least one device with an associated relationship. Based on the associated device information, semantic analysis is performed on the service request to obtain the at least one service sub-task; The service subtask includes at least: device type information, device identification information, and task information.

12. The apparatus according to claim 11, characterized in that, The device execution unit is further configured to: send the service subtask to the corresponding device invocation module according to the device type information; generate a device invocation instruction according to the task information; send the device invocation instruction to the corresponding device according to the device identification information; and receive the device execution result returned by the device.

13. The apparatus according to claim 10, characterized in that, The device execution unit is further configured to send the at least one service subtask to the device invocation proxy module; determine the target service interface corresponding to each service subtask; generate a service invocation instruction corresponding to the target service interface according to the service subtask; send the service invocation instruction to the corresponding target service interface; and receive the device execution result returned by the target service interface.

14. The apparatus according to claim 10, characterized in that, The task planning unit is also used to perform semantic analysis on the service request through the task planning model to obtain conditional execution tasks; The conditional execution task includes at least: a triggering condition and a device execution task; The device execution unit is used to control the corresponding device to execute the device execution task in response to the analysis result satisfying the trigger condition.

15. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the multimodal device control method according to any one of claims 1 to 9.

16. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform the multimodal device control method according to any one of claims 1 to 9.

17. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the multimodal device control method as described in any one of claims 1 to 9.

Images