Task processing method, control method of operating device, and electronic device

Abstract

The present application discloses a task processing method, a control method for an operating device, and an electronic device, and relates to large model technology and the field of task processing. The method includes: obtaining a task processing request and task environment data; analyzing the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; at least based on the initial state and the target state, a target action sequence corresponding to the task processing request is generated, wherein at least one action in the target action sequence is used to operate the target object. The present application solves the technical problem of low accuracy of action plan generation in the related art.

Description

Technical Field

[0001] The present application relates to the field of large model technology and task processing, and more specifically, to a task processing method, a control method for an operating device, and an electronic device. Background Art

[0002] Currently, large multimodal models have demonstrated powerful problem-solving capabilities in vision and language tasks, as well as the potential to encode large-scale world knowledge. However, in actual applications, due to the complexity of the real environment and the diversity of data, the data used to train large models is usually difficult to cover different data in different environments. When using large multimodal models to perceive and analyze data in the environment to plan action plans for changing the state of the environment, inaccurate action plans may be planned. Summary of the invention

[0003] The embodiments of the present application provide a task processing method, a control method of an operating device, and an electronic device to at least solve the technical problem of low accuracy in action plan generation in the related art.

[0004] According to one aspect of an embodiment of the present application, a task processing method is provided, including: obtaining a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of a target environment; using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment.

[0005] According to one aspect of an embodiment of the present application, a control method for an operating device is also provided, including: obtaining a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; and controlling the operating device to execute at least one action in the target action sequence in sequence.

[0006] According to another aspect of an embodiment of the present application, a control method for an operating device is also provided, including: receiving a task processing request and task environment data sent by the operating device, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; and sending the target action sequence to the operating device.

[0007] According to another aspect of an embodiment of the present application, a task processing method is also provided, including: responding to an input instruction acting on an operation interface, displaying a task processing request and task environment data on the operation interface, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; responding to a processing instruction acting on the operation interface, displaying a target action sequence corresponding to the task processing request on the operation interface, wherein at least one action in the target action sequence is used to operate a target object in the target environment, and the target action sequence is an action sequence generated based on at least an initial state and a target state of the target environment, the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the initial state and the target state are states obtained by analyzing the target task information and the target environment image using a multimodal reasoning model, and the multimodal reasoning model is a natural language deep learning model.

[0008] According to another aspect of an embodiment of the present application, a task processing method is also provided, including: obtaining a task processing request and task environment data by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the task processing request and the task environment data, the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; outputting the target action sequence by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target action sequence.

[0009] According to one aspect of an embodiment of the present application, a task processing device is provided, including: a data acquisition module, used to acquire a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of a target environment; a state determination module, used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; a sequence generation module, used to generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment.

[0010] According to one aspect of an embodiment of the present application, a control device for an operating device is also provided, including: a first acquisition module, used to acquire a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; a first determination module, used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; a first generation module, used to generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; a device control module, used to control the operating device to sequentially execute at least one action in the target action sequence.

[0011] According to another aspect of an embodiment of the present application, a control device for an operating device is also provided, including: a data receiving module, used to receive a task processing request and task environment data sent by the operating device, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; a second determination module, used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; a second generation module, used to generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; a sequence sending module, used to send the target action sequence to the operating device.

[0012] According to another aspect of an embodiment of the present application, a task processing device is also provided, including: a first display module, used to respond to an input instruction acting on an operation interface, and display a task processing request and task environment data on the operation interface, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; a second display module, used to respond to a processing instruction acting on the operation interface, and display a target action sequence corresponding to the task processing request on the operation interface, wherein at least one action in the target action sequence is used to operate a target object in the target environment, and the target action sequence is an action sequence generated based on at least an initial state and a target state of the target environment, the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the initial state and the target state are states obtained by analyzing the target task information and the target environment image using a multimodal reasoning model, and the multimodal reasoning model is a natural language deep learning model.

[0013] According to another aspect of an embodiment of the present application, a task processing device is also provided, including: a second acquisition module, used to obtain a task processing request and task environment data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the task processing request and the task environment data, the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; a third determination module, used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; a third generation module, used to generate a target action sequence corresponding to the task processing request based on at least the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; a sequence output module, used to output the target action sequence by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target action sequence.

[0014] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0015] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0018] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0019] In an embodiment of the present application, a task processing request and task environment data are obtained; a multimodal reasoning model is used to analyze the target task information and the target environment image to obtain the initial state and the target state of the target environment; and a target action sequence corresponding to the task processing request is generated based at least on the initial state and the target state. Before generating the target action sequence, the initial state and the target state of the target environment are determined based on the target environment image in the target environment and the corresponding target task information, and then the target action sequence is generated based on these two states. This can ensure the degree of correlation between the generated target action sequence and the target environment, so that the target action sequence can be supported by the target environment, enhance the perception and understanding ability of the multimodal reasoning model, thereby improving the accuracy of the generated target action sequence, and thus solving the technical problem of low accuracy of action plan generation in related technologies.

[0020] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a schematic diagram of a task processing scenario according to the present application;

[0023] Figure 2 is a flowchart of a task processing method according to the present application;

[0024] Figure 3 is a schematic diagram of a task processing process according to the present application;

[0025] Figure 4 is a flow chart of a control method of an operating device according to the present application;

[0026] Figure 5 is a flow chart of another control method of an operating device according to the present application;

[0027] Figure 6 is a flowchart of another task processing according to the present application;

[0028] Figure 7 is a flowchart of another task processing method according to the present application;

[0029] Figure 8 is a structural block diagram of a task processing device according to the present application;

[0030] Fig. 9 is a structural block diagram of a control device for operating equipment according to the present application;

[0031] Fig.10 is a structural block diagram of another control device for operating equipment shown in the present application;

[0032] Fig.11 is a structural block diagram of another task processing device according to the present application;

[0033] Fig.12 is a structural block diagram of another task processing device according to the present application;

[0034] Fig.13 It is a structural block diagram of an electronic device according to the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The technical solution provided in this application is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (LLM), multi-modal pre-training model, etc.

[0038] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In an embodiment of the present application, in the scenario of operating a target object in a target environment, multimodal data, such as task processing requests and task environment data, are processed by using a large model such as a multimodal reasoning model or a data augmentation model, and the construction of a target action sequence for converting the target environment from an initial state to a target state is used as an example for explanation.

[0039] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0040] Embodied planning: Embodied planning is a process of developing and executing action plans in a physical or virtual environment by sensing the environment, understanding the mission objectives, and combining one's own capabilities.

[0041] Neuro-Symbolic: Neuro-symbolic, an artificial intelligence approach that combines neural networks and symbolic reasoning, can be used to build a neural symbolic engine.

[0042] According to an embodiment of the present application, a task processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above task processing method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of a task processing scenario according to the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0044] In the embodiment of the present application, the system composed of the client device and the server can perform the following steps: the client device performs the steps of obtaining the task processing request and the task environment data, displaying the task processing request and the task environment data to the user, and displaying the target action sequence to the user, and the server performs the steps of analyzing the target task information and the target environment image using the multimodal reasoning model, obtaining the initial state and the target state of the target environment, and generating the target action sequence according to the initial state and the target state. It should be noted that the embodiment of the present application can be performed in the client device when the operating resources of the client device can meet the deployment and operation conditions of the large model.

[0045] Under the above operating environment, this application provides Figure 2 The task processing method shown. Figure 2 is a flowchart of a task processing method according to the present application. Figure 2 As shown, the method may include the following steps:

[0046] Step S202, obtaining a task processing request and task environment data.

[0047] The task processing request includes target task information, and the task environment data includes: a target environment image of the target environment;

[0048] The above-mentioned target environment may refer to the environment in which the task processing system (referred to as the processing system for short) is located, which may be a physical environment in real life or a virtual environment in a virtual network, wherein the processing system may refer to a system for perceiving, analyzing, and planning data in the target environment to generate an action plan for controlling the behavior of different objects in the target environment. For example, when the target environment is a user's residence, the corresponding processing system may refer to a smart home system, which may generate an action plan for controlling these devices according to user instructions and real-time data generated by devices such as air conditioners, refrigerators, temperature sensors, and sweeping robots in the residence, so as to realize operations such as controlling the switching of the air conditioner, adjusting the temperature, controlling the refrigeration and freezing switches of the refrigerator, and controlling the operation of the sweeping robot; when the target environment is a virtual network environment, the corresponding processing system may be a network smart housekeeper, which may monitor the real-time data changes of different module components in the network in real time, so as to generate an action plan for controlling the operating parameters of different module components according to user instructions or data change trends, thereby realizing operations to ensure the security of the network environment and reduce the operating costs of the network environment. The target environment image can be obtained by photographing the target environment with an image capturing device or an image generating device, or by generating an image based on data generated by different objects in the target environment. The target task information can be a description of the operation task for controlling different objects in the environment pre-configured in the processing system, or target task information corresponding to the instructions input by the user for controlling different objects in the target environment. The target environment image and target task information can be used to assist the processing system in generating a corresponding action plan to ensure the accuracy of the generated action plan.

[0049] At present, there are three common ways to generate action plans. The first is to directly plan the baseline, which can directly input the environmental data into the pre-trained model to generate an action plan corresponding to the environmental data. However, this method cannot take into account the rationality of the action plan during execution, such as whether it matches the current state of different objects in the environment, nor does it take into account the current environmental state of the target environment, such as whether it supports the execution of the action plan, resulting in the generated action plan may not be executed and poor accuracy. The second is to compare the pre-built action plan framework with the thinking chain as a planning method to determine a reasonable action plan through the thinking chain method, but this method is usually used in task reasoning and problem analysis and processing, and it cannot clarify the environmental state of the target environment. It is separated from the target environment itself, resulting in a lack of basis during the execution of the action plan, resulting in the failure of the action plan execution. The third is RAP (Reasoning Via Planning, reasoning through planning), which generates action plans by using world models and reasoning agent planning. However, in this method, there are usually inaccuracies in the intermediate steps, resulting in the wrong direction of generating the action plan, and ultimately failing to generate an accurate action plan.

[0050] Based on this, in order to reasonably and accurately generate corresponding action plans for different objects in the target environment, the environmental state of the target environment can be determined in real time based on the real-time data generated by different objects to improve the degree of correlation between the generated action plan and the target environment, thereby ensuring that the generated action plan can be supported by the target environment to improve the rationality and feasibility of the action plan. Therefore, when generating an action plan, the processing system can obtain in real time multimodal data related to the target object currently required to be controlled in the target environment, such as a task processing request for controlling the target object, and task environment data that can reflect the environmental state of the environment in which the target object is currently located. Based on this, the processing system can shoot the target environment through an image shooting device or an image generating device to obtain an environmental image that can reflect the environmental state of the target environment, that is, the above-mentioned target environment image, and construct the above-mentioned task environment data based on the target environment image. At the same time, the task-related description pre-configured in the processing system or the instruction text corresponding to the control instruction input by the user in real time can be analyzed to determine the target task information used to operate the target object, and construct the above-mentioned task processing request based on the target task information.

[0051] Step S204, using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment.

[0052] The initial state is used to represent the state of the target environment before executing the task processing request, and the target state is used to represent the state of the target environment after executing the task processing request.

[0053] The multimodal reasoning model may refer to a pre-trained natural language deep learning model for determining the initial state and the target state. The state of the target environment may be determined by the real-time state of different objects in the target environment. For example, assuming that the current target environment is the user's residence, and there are devices such as a sweeping robot, a sofa, a TV, and an air conditioner in the target environment, the state of the target environment may be a state determined by the real-time state of the sweeping robot, the real-time state of the sofa, the real-time state of the TV, and the real-time state of the air conditioner. Correspondingly, the real-time state of different objects may be determined by the real-time operating parameters of different objects. Taking the sweeping robot as an example, the real-time state of the sweeping robot may be constructed based on the working state, working parameters, working position, power, room cleaning results, and other data of the sweeping robot at the current moment. The initial state may refer to the state of the target environment before the processing system executes the task processing request, which may be determined based on the current operating state of different objects in the target environment. The target state may refer to the state of the target environment after the processing system executes the task processing request, which may be determined based on the control parameters contained in the operation task currently required. To facilitate understanding, assuming that the processing system detects that the battery of the sweeping robot is too low and needs to be charged, the operation task of charging the sweeping robot will be triggered in the processing system. The corresponding action plan may be to charge the sweeping robot. At this time, the above-mentioned initial state may be the environmental state determined by the current state of the sweeping robot, sofa, TV, air conditioner and other equipment before executing the action plan, that is, when it is determined that the sweeping robot needs to be charged. The target state may be the environmental state determined by the current state of the sweeping robot, sofa, TV, air conditioner and other equipment after executing the action plan, that is, after the sweeping robot is charged. During this process, the operating state of the sofa, TV, air conditioner and other equipment may remain unchanged, while the operating state of the sweeping robot will change, and the corresponding environmental state of the target environment will also change during the action plan.

[0054] In an optional scheme of this embodiment, as shown above, considering that the environmental state of the target environment in which the processing system is located will have a great impact on the rationality, accuracy and other properties of the formulated action plan, therefore, in order to accurately and reasonably formulate the action plan, the processing system can first determine the initial state and target state of the target environment based on the acquired target task information and target environment image. In order to improve the accuracy of the determined initial state and target state, the target task information and target environment image can be processed using the above-mentioned multimodal reasoning model to determine the initial state and target state of the target environment.

[0055] Continuing with the above-mentioned sweeping robot as an example, in the initial state, the state of the sweeping robot can be a state determined by analyzing the target environment image related to the sweeping robot in the multimodal data. For example, the target environment image can be analyzed to determine the current position of the sweeping robot in the user's residence, the trend of power changes, the area of ​​the ground that has been cleaned and other data, and then the state of the sweeping robot before executing the action plan is determined based on these data; in the target state, the state of the sweeping robot can be a state determined by analyzing the image related to the charging pile of the sweeping robot in the target environment image and the information related to the charging process of the sweeping robot in the target task information. For example, the target environment image can be analyzed to determine the location of the charging pile, the charging angle and other data, and at the same time, the target task information can be analyzed to determine the target power of the sweeping robot when charging, the charging time and other data, and then the state of the sweeping robot after executing the action plan is determined based on these data.

[0056] It should be noted that the examples mentioned above to explain the initial state and the target state are only illustrative and are not intended to be limiting.

[0057] Step S206, generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state.

[0058] At least one action in the target action sequence is used to operate a target object in the target environment.

[0059] The above-mentioned target action sequence may refer to an action sequence used to represent an action plan, which may be used to operate a target object in a target environment so that the state of the target environment is converted from an initial state to a target state. Generally, a plurality of actions may be included in the target action sequence, and different actions may be used to operate the target object. Continuing with the example of a sweeping robot in a user's residence, when the processing system detects that the sweeping robot needs to be charged, the actions included in the above-mentioned target action sequence may include but are not limited to: actions for planning the driving path, driving speed, driving angle and other data of the sweeping robot from the current position to the charging pile position, and actions for planning the charging time, charging power, target power and other data of the sweeping robot when it reaches the charging pile for charging.

[0060] In an optional solution of this embodiment, after determining the initial state and target state of the target environment, the processing system can generate a target action sequence for operating the target object in the target environment based on the task processing request in an embedded planning manner according to the two states. For example, the processing system can match the generated initial state and target state to determine the target object that will change state in the target environment, and then determine the operation to be performed on the target object and the corresponding operation parameters according to the target task information used to describe the operation task, and finally generate a target action sequence for controlling the target object according to the operation parameters. After generating the target action sequence, the processing system can also display the sequence to the user in a preset operation interface to facilitate user viewing and modification.

[0061] Considering that the target action sequence constructed only based on the initial state and the target state may conflict with the current action of the target object, for example, if the determined initial state is that the sweeping robot is cleaning the floor, the target state is that the sweeping robot is fully charged, and the target task information shows that the sweeping robot is charged when the power of the sweeping robot is lower than the preset power threshold, then the corresponding target action sequence can be an action plan for controlling the sweeping robot to move to the charging pile for charging. If the current power of the sweeping robot is sufficient to complete the cleaning of the remaining floor, the target action sequence can actually include an action plan for continuing to clean the floor. Without this action plan, it may take longer to clean the floor and increase the cost. Therefore, in addition to generating a target action sequence that matches the task processing request based on the initial state and the target state, the processing system can also combine the current parameters of the target object itself and the parameters of other objects that have an associated relationship with the target object to improve the generated target action sequence to ensure the accuracy of the generated target action sequence.

[0062] In an embodiment of the present application, a method is adopted in which multimodal data is acquired; based on the multimodal data, an initial state and a target state of a target environment are generated; and at least based on the initial state and the target state, a target action sequence corresponding to a task processing request is generated. Before generating the target action sequence, the initial state and the target state of the target environment are generated according to the multimodal perception data in the target environment, and then the target action sequence is generated based on these two states. This can ensure the degree of correlation between the generated target action sequence and the target environment, so that the target action sequence can be supported by the target environment, enhance the perception and understanding ability of the multimodal reasoning model, thereby improving the accuracy of the generated target action sequence, and further solving the technical problem of low accuracy of action plan generation in related technologies.

[0063] In an embodiment of the present application, a multimodal reasoning model is used to analyze target task information and a target environment image to obtain an initial state and a target state of the target environment, including: inputting the target task information and the target environment image into the multimodal reasoning model, and obtaining the initial state output by the multimodal reasoning model; inputting the target task information, the target environment image, and the initial state into the multimodal reasoning model, and obtaining the target state output by the multimodal reasoning model.

[0064] In an optional scheme of the present embodiment, considering that the initial state is an environmental state that can be clearly determined, and the target state can be regarded as an environmental state obtained by predicting the future state of the target environment, if the target state is determined only according to the target task information, the determined target state may lack relevance to the target environment, resulting in the target action sequence generated according to the initial state and the target state being unable to be smoothly executed in the target environment. Therefore, in order to ensure the rationality and accuracy of the determined initial state and target state, the processing system may first input the target task information and the target environment image into the above-mentioned multimodal reasoning model to determine the initial state of the target environment according to the target environment image that can reflect the current state of different objects in the target environment and the description related to the current state of different objects in the target task information, and then input the target task information, the target environment image and the multimodal data into the multimodal reasoning model again to determine the target state of the target environment according to the determined initial state, the state of other objects unrelated to the target object in the target environment image, and the description related to the target state of the target object in the target task information, so as to further ensure the authenticity of the determined initial state and the degree of relevance between the determined target state and the target environment.

[0065] In an embodiment of the present application, a multimodal reasoning model is used to analyze target task information and a target environment image to obtain an initial state and a target state of the target environment, including: determining environmental identification data of the target environment, wherein the environmental identification data includes location identifications of different locations in the target environment and object identifications of different objects in the target environment; and using a multimodal reasoning model to analyze target task information, a target environment image and environmental identification data to obtain an initial state and a target state.

[0066] The above-mentioned environment identification data may refer to area identifications set for different areas in the target environment, and object identifications set for different objects in the target environment.

[0067] In an optional scheme of the present embodiment, in order to avoid the situation where the initial state or target state of the determined target environment is abnormal due to the single multimodal data failing to include data corresponding to all objects in the target environment, such as object location, object type and other data, when using the multimodal reasoning model to analyze the acquired target task information and target environment image, the processing system may first obtain environmental data related to multiple objects in the target environment, i.e., the above-mentioned environmental identification data, and based on the environmental identification data, use the multimodal reasoning model to analyze the target task information and the target environment image to ensure the accuracy of the determined initial state and target state. Among them, in order to facilitate the determination of the environmental identification data of the target environment, the target environment can be divided into multiple different location areas according to the environmental parameters of the target environment, such as the environment size, the environment area type and other parameters, and different location identifications can be set for different location areas. The corresponding processing system can directly use the location identification to indicate the position of objects in the environment, and in order to facilitate the distinction between different objects in the same position, different object identifications can also be set for different objects. Correspondingly, when obtaining the environmental identification data, the processing system can first determine the different location areas included in the target environment, and then determine the different objects included in the different location areas, and finally construct the above-mentioned environmental identification data according to the location identifications of different location areas and the object identifications of different objects.

[0068] In an embodiment of the present application, a target action sequence corresponding to a task processing request is generated based at least on an initial state and a target state, including one of the following: inputting target task information, a target environment image, an initial state, and a target state into a multimodal reasoning model, and obtaining a target action sequence output by the multimodal reasoning model; inputting the initial state and the target state into a data enhancement model, and obtaining a target action sequence output by the data enhancement model, wherein the data enhancement model is used to symbolize the initial state and the target state, and generate a target action sequence based on the symbolized initial state and the target state.

[0069] In an optional scheme of the present embodiment, in order to ensure the accuracy of the generated target action sequence, the processing system may input the target task information, the target environment image, the initial state and the target state into the multimodal reasoning model, and first determine the target object that currently needs to be operated from the multiple objects contained in the target environment according to the initial state and the target state, and then determine the target task information that matches the object identification from the target task information and the target task information contained in the target environment image according to the object identification of the target object, and finally generate a corresponding action plan according to the target task information to obtain the above-mentioned target action sequence. Alternatively, in order to avoid the limitations of data perception, analysis and planning that may exist in the multimodal reasoning model, the above-mentioned initial state and The target state is input into the pre-trained data enhancement model, and the initial state and the target state are symbolized by the data enhancement model to obtain the above-mentioned symbolized initial state and the symbolized target state. Finally, the target action sequence is generated according to the symbolized initial state and the symbolized target state by using the method of generating the target action sequence using the neural symbolic framework in the relevant literature. For example, the processing system can use the above-mentioned data enhancement model to generate multiple action sequences according to the two symbolized states, and score and sort the multiple action sequences, and then use the action sequence with the highest score as the above-mentioned target action sequence, so as to ensure the feasibility, rationality and accuracy of the obtained target action sequence.

[0070] In an embodiment of the present application, after the initial state and the target state are input into the data enhancement model, the method further includes: in response to the normal operation process of the data enhancement model, obtaining the target action sequence output by the data enhancement model; in response to an abnormality in the operation process of the data enhancement model, inputting the target task information, the target environment image, the initial state and the target state into the multimodal reasoning model, and obtaining the target action sequence output by the multimodal reasoning model.

[0071] In an optional scheme of the present embodiment, after the initial state and the target state are input into the data enhancement model, the processing system can monitor the running state of the data enhancement model in real time. If the running process of the data enhancement model is normal, the action sequence output by the data enhancement model can be directly obtained, and the action sequence can be determined as the above-mentioned target action sequence; if the running process of the data enhancement model is abnormal, it means that the action sequence currently output by the data enhancement model may be wrong. At this time, the processing system can re-input the target task information, target environment image, initial state and target state into the above-mentioned multimodal reasoning model, and use the multimodal reasoning model to regenerate the above-mentioned target action sequence, thereby ensuring the accuracy of the generated target action sequence.

[0072] In an embodiment of the present application, after generating a target action sequence corresponding to a task processing request based at least on an initial state and a target state, the method further includes: based on the initial state, simulating operations on a target object according to the target action sequence to determine a result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; when the result state does not match the target state, adjusting the model parameters of the multimodal reasoning model.

[0073] In an optional solution of the present embodiment, after generating the target action sequence corresponding to the multimodal data, the processing system may further simulate the target action sequence of the target object in the virtual environment based on the initial state to simulate the environmental state of the target environment after the target object is currently operated according to the target action sequence in the target environment, that is, determine the result state of the above-mentioned target environment, and then match the result state with the aforementioned generated target state to determine whether the target action sequence is feasible and accurate. If the two states match, it means that the current operation of the target object according to the target action sequence can accurately reach the target state, and the corresponding processing system can perform the operation according to the target action sequence. operate on the target object; if the two states do not match, it means that errors may occur when operating the target object according to the target action sequence. At this time, the processing system can adjust the model parameters of the multimodal reasoning model according to the difference between the result state and the target state to ensure the accuracy of the multimodal reasoning model. Alternatively, the processing system can also display the target action sequence to the user in a preset operation interface. After the user adjusts the target action sequence, the processing system adjusts the model parameters of the data processing model according to the adjusted target action sequence, so as to ensure the accuracy of the multimodal reasoning model, thereby improving the accuracy of the initial state, target state, target action sequence and other data subsequently generated by the multimodal reasoning model.

[0074] To facilitate understanding of the above task processing method, Figure 3is a schematic diagram of a task processing process according to the present application, wherein 301 represents a target environment image corresponding to multiple objects existing in the target environment, which can be obtained by photographing with an image capturing device, 302 represents target task information corresponding to an operation task for operating a target object in the target environment, which can be a control instruction input by a user, for example, it can include: putting grapes in a bowl, putting durian next to a pot, 303 represents a multimodal reasoning model, which can be used to generate an initial state and a target state of the target environment according to multimodal data such as the target task information and the target environment image, as well as a target action sequence for converting the state of the target environment from the initial state to the target state, 304 represents an initial state generated by the multimodal reasoning model, which can be determined by the target environment image corresponding to 301, for example, the initial state can at least include: apples on a table, strawberries on a plate, grapes next to a frying pan, blackberries next to a box, and durian next to a bowl, 305 represents a target state generated by the multimodal reasoning model The generated target state can be determined by the target task information corresponding to 302. For example, the target state can at least include: grapes in the bowl, durian next to the pot. 306 represents the action sequence generated by the multimodal reasoning model directly according to the initial state and the target state. For example, the target action sequence can at least include: move to the side of the pan, pick up grapes, move to the side of the bowl, put grapes in the bowl, pick up durian, move to the side of the pan, pick up durian, move to the side of the bowl, put down durian. 307 can be a data enhancement model, which can be a Neuro-Symbolic neural symbolic engine, and can also be used to generate a target action sequence according to the initial state and the target state. 308 represents the action sequence generated by the data enhancement model. For example, the target action training can include: move to the side of the pan, pick up grapes, move to the side of the bowl, put grapes in the bowl, pick up durian, move to the side of the pan, and put down durian. 309 represents the target action sequence determined from the two action sequences 306 and 308. Figure 3As shown, when generating an action plan, the processing system can first obtain the target environment image corresponding to different objects in the target environment, and the target task information for operating the target object. According to the target environment image and the target task information, the multimodal data of the target environment can be constructed, and the multimodal data can be input into the multimodal reasoning model to obtain the initial state and target state of the target environment, as well as the action plan for operating the target object, that is, the action sequence. At the same time, the initial state and the target state can also be input into the data enhancement model to regenerate the action sequence. After the two models generate the action sequence respectively, the processing system can also simulate and verify the two action sequences to select the target action sequence that meets the current environment state. For example, the action sequence generated by the above-mentioned data enhancement model can be determined as the target action sequence, so as to ensure the accuracy of the generated target action sequence. Finally, the target action sequence is executed on the target object, which can ensure the stability of converting the state of the target environment from the initial state to the target state. It should be noted that the above-mentioned target environment image, target task information, action sequences generated by different models, etc. are all exemplary displays and are not limited here.

[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0076] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0077] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0078] According to an embodiment of the present application, a control method for operating a device is also provided. Figure 4 is a flow chart of a control method for an operating device according to the present application, such as Figure 4 As shown, the method may include the following steps:

[0079] Step S402, obtaining a task processing request and task environment data.

[0080] The task processing request includes target task information, and the task environment data is a target environment image obtained by photographing the target environment where the operating device is located.

[0081] The above-mentioned operating device can be a device deployed in a real scene and used to operate the target object in the target environment. For example, it can be a mobile device for moving tables, chairs and other furnishings, or a control device for controlling electrical appliances such as air conditioners and refrigerators.

[0082] In an optional solution of this embodiment, when the operating device is only used as a medium for operating the target object, that is, the operating device will be controlled by the control system of the operating device (hereinafter referred to as the control system), in order to improve the rationality and accuracy of controlling the operating device, the control system can determine the environmental state of the target environment in which the operating device is located in real time based on the real-time data generated by different operating devices, so as to improve the degree of correlation between the action plan generated for operating the operating device and the target environment, thereby ensuring that the generated action plan can be supported by the target environment, so as to improve the rationality and feasibility of executing the action plan. Based on this, when generating the action plan, the control system can obtain multimodal data contained in the target environment in real time, that is, the above-mentioned task processing request and task environment data. For example, the control system can shoot the target environment in which the operating device is located through an image shooting device to obtain the above-mentioned target environment image, and construct the above-mentioned task environment data based on the target environment image. At the same time, the control system can analyze the target task information received by the operating device for operating the target object in the target environment to obtain the construction of the above-mentioned task processing request.

[0083] Step S404: Analyze the target task information and the target environment image using the multimodal reasoning model to obtain the initial state and target state of the target environment.

[0084] Among them, the initial state is used to characterize the state of the target environment before executing the task processing request, the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model.

[0085] The multimodal reasoning model may refer to a pre-trained natural language deep learning model for determining an initial state and a target state. The state of the target environment may be determined by the real-time states of different objects in the target environment. The initial state may refer to the state of the target environment before the control system executes a task processing request, which may be determined based on the current operating states of different objects in the target environment. The target state may refer to the state of the target environment after the control system executes a task processing request, which may be determined based on the control parameters contained in the operation task currently required.

[0086] In an optional scheme of the present embodiment, considering that the environmental state of the target environment in which the control system is located will have a significant impact on the rationality, accuracy and other properties of the formulated action plan, therefore, in order to accurately and reasonably formulate an action plan, the control system can use a pre-trained multimodal reasoning model to analyze the acquired target task information and target environment image to determine the initial state and target state of the target environment.

[0087] Step S406: Generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state.

[0088] At least one action in the target action sequence is used to operate a target object in the target environment.

[0089] The target action sequence may refer to an action sequence used to represent an action plan, and may be used to operate a target object in a target environment so that the state of the target environment is converted from an initial state to a target state.

[0090] In an optional scheme of the present embodiment, after determining the initial state and target state of the target environment, the control system can generate a target action sequence for operating the target object in the target environment based on the task processing request according to these two states. For example, the control system can match the generated initial state and target state to determine the target object that will change state in the target environment, and then determine the operation to be performed on the target object and the corresponding operation parameters according to the target task information used to describe the operation task, and finally generate a target action sequence for controlling the target object according to the operation parameters. After generating the target action sequence, the control system can also display the sequence to the user in a preset operation interface to facilitate user viewing and modification.

[0091] In an optional scheme of the present embodiment, it is taken into account that the target action sequence constructed only according to the initial state and the target state may conflict with the current action of the target object. For example, if the table needs to be moved to the target position, the corresponding position of the table in the initial state is at the initial position, and the position of the table in the target state is at the target position. When the operating device moves the table only according to the initial position and the target position, the table may collide with other objects during the movement, resulting in the inability to move the table, or the operating device may move too fast, causing the table to detach from the operating device and cause damage. Therefore, in addition to directly constructing target action training based on the initial state and the target state, the control system can also obtain object-related parameters of different objects in the target environment, and apply the parameters to construct the above-mentioned target action sequence, thereby ensuring the rationality and accuracy of the constructed target action sequence.

[0092] Step S408: controlling the operating device to sequentially execute at least one action in the target action sequence.

[0093] In an optional solution of the present embodiment, after constructing the target action sequence, the control system can control the above-mentioned operating device to execute at least one action included in the sequence at one time, so as to achieve the purpose of operating the target object in the target environment. For example, assuming that the operating device is a mobile device and the target device to be moved is a table, the actions included in the target action sequence may at least include: moving from the current standby position to the initial position of the table according to a first route and a first speed, raising the robotic arm to a first height after the robotic arm is close to the bottom of the table, maintaining the current height of the robotic arm and moving to the target position according to a second route and a second speed, lowering the robotic arm at the target position, leaving the target position, etc. The corresponding operating device can move the table to the target position according to these actions, thereby realizing the operation of moving the table.

[0094] In an embodiment of the present application, the control operation device sequentially executes at least one action in the target action sequence, including: based on the initial state, simulating the operation of the target object according to the target action sequence to determine the result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; when the result state matches the target state, the control operation device sequentially executes at least one action in the target action sequence.

[0095] In an optional solution of this embodiment, when the control operation device sequentially executes the actions in the target action sequence, the control system can further simulate the actions included in the target action sequence based on the above initial state to simulate the environmental state of the target environment after the current control operation device operates the target object according to the target action sequence, that is, determine the result state of the above target environment, and then match the result state with the aforementioned generated target state to determine whether the target action sequence is feasible and accurate. If the two states match, it means that the operation device is currently operating the target object according to the target action sequence and can accurately reach the target state. The corresponding control system can control the operation device to operate the target object according to the target action sequence. object to operate; if the two states do not match, it means that the current operation of the target object according to the target action sequence may cause errors. At this time, the control system can adjust the model parameters of the multimodal reasoning model used to generate the target action sequence according to the difference between the result state and the target state to ensure the accuracy of the multimodal reasoning model. Alternatively, the control system can also display the target action sequence to the user in a preset operation interface. After the user adjusts the target action sequence, the control system adjusts the model parameters of the data processing model according to the adjusted target action sequence, so as to ensure the accuracy of the multimodal reasoning model, thereby improving the accuracy of the initial state, target state, target action sequence and other data subsequently generated by the multimodal reasoning model.

[0096] According to an embodiment of the present application, a control method for operating a device is also provided. Figure 5 is a flow chart of another control method of an operating device according to the present application, such as Figure 5 As shown, the method may include the following steps:

[0097] Step S502: receiving a task processing request and task environment data sent by an operating device.

[0098] The task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment where the operating device is located.

[0099] In an optional solution of this embodiment, when the operating device is an independent device for operating the target object, that is, the operating device can run independently without being actively controlled by the operating system, in order to improve the rationality and accuracy of controlling the operating device, the control system can determine in real time the environmental state of the target environment in which the operating device is located, based on the real-time data generated by different operating devices, so as to improve the degree of correlation between the action plan generated for the operating device and the target environment, thereby ensuring that the generated action plan can be supported by the target environment, so as to improve the rationality and feasibility of the operating device when executing the action plan. Based on this, when generating the action plan, the control system can first receive the multimodal data contained in the target environment obtained in real time by the operating device, that is, the above-mentioned task processing request and task environment data. For example, the operating device can first use an image capturing device to capture the target environment in which the operating device is located to obtain the above-mentioned target environment image, and construct the above-mentioned task environment data based on the target environment image. At the same time, the above-mentioned task processing request is obtained by analyzing the received target task information for operating the target object in the target environment.

[0100] Step S504: Analyze the target task information and the target environment image using the multimodal reasoning model to obtain the initial state and target state of the target environment.

[0101] Among them, the initial state is used to characterize the state of the target environment before executing the task processing request, the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model.

[0102] The multimodal reasoning model may refer to a pre-trained natural language deep learning model for determining an initial state and a target state. The state of the target environment may be determined by the real-time states of different objects in the target environment. The initial state may refer to the state of the target environment before the control system executes a task processing request, which may be determined based on the current operating states of different objects in the target environment. The target state may refer to the state of the target environment after the control system executes a task processing request, which may be determined based on the control parameters contained in the operation task currently required.

[0103] In an optional scheme of the present embodiment, considering that the environmental state of the target environment in which the control system is located will have a significant impact on the rationality, accuracy and other properties of the formulated action plan, therefore, in order to accurately and reasonably formulate an action plan, the control system can use a pre-trained multimodal reasoning model to analyze the acquired target task information and target environment image to determine the initial state and target state of the target environment.

[0104] Step S506: Generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state.

[0105] At least one action in the target action sequence is used to operate the target object.

[0106] The target action sequence may refer to an action sequence used to represent an action plan, and may be used to operate a target object in a target environment so that the state of the target environment is converted from an initial state to a target state.

[0107] In an optional scheme of the present embodiment, after determining the initial state and target state of the target environment, the control system can generate a target action sequence for operating the target object in the target environment based on the task processing request according to these two states. For example, the control system can match the generated initial state and target state to determine the target object that will change state in the target environment, and then determine the operation to be performed on the target object and the corresponding operation parameters according to the target task information used to describe the operation task, and finally generate a target action sequence for controlling the target object according to the operation parameters. After generating the target action sequence, the control system can also display the sequence to the user in a preset operation interface to facilitate user viewing and modification.

[0108] In an optional scheme of the present embodiment, it is taken into account that the target action sequence constructed only according to the initial state and the target state may conflict with the current action of the target object. For example, if the table needs to be moved to the target position, the corresponding position of the table in the initial state is at the initial position, and the position of the table in the target state is at the target position. When the operating device moves the table only according to the initial position and the target position, the table may collide with other objects during the movement, resulting in the inability to move the table, or the operating device may move too fast, causing the table to detach from the operating device and cause damage. Therefore, in addition to directly constructing target action training based on the initial state and the target state, the control system can also obtain object-related parameters of different objects in the target environment, and apply the parameters to construct the above-mentioned target action sequence, thereby ensuring the rationality and accuracy of the constructed target action sequence.

[0109] Step S508: sending the target action sequence to the operating device.

[0110] In an optional scheme of the present embodiment, after constructing the target action sequence, the control system can send the generated target action sequence to the operating device, and the operating device can actively choose whether to execute the target action sequence, thereby avoiding the limitations of the operating device directly controlling the control system to execute the target action sequence, such as the single operation mode of the operating device, poor flexibility, and the inability to adapt the action plan to be executed in real time according to changes in the environment.

[0111] In an embodiment of the present application, sending a target action sequence to an operating device includes: based on an initial state, performing a simulated operation on a target object according to the target action sequence to determine a result state of a target environment, wherein the result state is used to characterize a state of the target environment after the target object is operated according to the target action sequence; when the result state matches the target state, sending the target action sequence to the operating device.

[0112] In an optional scheme of the present embodiment, in order to avoid the situation where the operating device fails to execute the target action sequence, the control system can also simulate the actions included in the target action sequence based on the above-mentioned initial state before sending the target action sequence to the operating device, so as to simulate the environmental state of the target environment after the operating device operates the target object according to the target action sequence, that is, determine the result state of the above-mentioned target environment, and then match the result state with the aforementioned generated target state to determine whether the target action sequence is feasible and accurate. If the two states match, it means that the operating device can accurately reach the target state by operating the target object according to the target action sequence, and the corresponding control system can send the target action sequence to the operating device; if the two states do not match, it means that an error may occur when the operating device operates the target object according to the target action sequence. At this time, the control system may not send the target action sequence to the operating device, and adjust the model parameters of the multimodal reasoning model used to generate the target action sequence according to the difference between the result state and the target state of the above simulation to ensure the accuracy of the multimodal reasoning model. Alternatively, the control system can also display the target action sequence to the user in a preset operation interface. After the user adjusts the target action sequence, the control system adjusts the model parameters of the data processing model according to the adjusted target action sequence, thereby ensuring the accuracy of the multimodal reasoning model, thereby improving the accuracy of the initial state, target state, target action sequence and other data subsequently generated by the multimodal reasoning model.

[0113] According to an embodiment of the present application, a task processing method is also provided. Figure 6 is a flowchart of another task processing according to the present application, such as Figure 6 As shown, the method may include the following steps:

[0114] Step 602, in response to an input instruction on the operation interface, displaying the task processing request and the task environment data on the operation interface.

[0115] The task processing request includes target task information, and the task environment data includes: a target environment image of the target environment.

[0116] The above-mentioned input instructions may refer to instructions for acquiring multimodal data. The above-mentioned target environment may refer to the environment in which the processing system is located. The above-mentioned target environment image may be obtained by photographing the target environment through an image capturing device or an image generating device, or by generating images based on data generated by different objects in the target environment. The above-mentioned target task information may be a description of the operation task for controlling different objects in the environment pre-configured in the processing system, or target task information corresponding to the instructions input by the user for controlling different objects in the target environment. The target environment image and target task information may be used to assist the processing system in generating a corresponding action plan to ensure the accuracy of the generated action plan.

[0117] In an optional solution of this embodiment, in order to reasonably and accurately generate corresponding action plans for different objects in the target environment, the environmental state of the target environment can be determined in real time based on the real-time data generated by different objects to improve the correlation between the generated action plan and the target environment, thereby ensuring that the generated action plan can be supported by the target environment to improve the rationality and feasibility of the action plan. Based on this, when generating an action plan, if an input instruction input by the user on the operation interface is detected, the processing system can first obtain the multimodal data contained in the target environment, and display the multimodal data in the operation interface, that is, the above-mentioned task processing request and task environment data, so as to facilitate user viewing. For example, the control system can obtain the target environment image of the above-mentioned target environment through an image capture device or an image generation device, and construct the above-mentioned task environment data according to the target environment image, and at the same time, analyze the task-related description pre-configured in the processing system, or the instruction text corresponding to the control instruction input by the user in real time, so as to obtain the target task information of the operation task for operating the target object in the target environment, and construct the above-mentioned task processing request according to the target task information.

[0118] Step 604, in response to the processing instruction acting on the operation interface, the target action sequence corresponding to the target task information is displayed on the operation interface.

[0119] Among them, at least one action in the target action sequence is used to operate the target object in the target environment. The target action sequence is an action sequence generated based on at least an initial state and a target state of the target environment. The initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request. The initial state and the target state are states obtained by analyzing the target task information and the target environment image using a multimodal reasoning model. The multimodal reasoning model is a natural language deep learning model.

[0120] The above-mentioned multimodal reasoning model may refer to a pre-trained natural language deep learning model for determining the initial state and the target state. The above-mentioned processing instructions may refer to instructions for generating a target action sequence. The above-mentioned target action sequence may refer to an action sequence for representing an action plan, which can be used to operate a target object in a target environment so that the state of the target environment is converted from an initial state to a target state. The state of the above-mentioned target environment may be determined by the real-time state of different objects in the target environment. The above-mentioned initial state may refer to the state of the target environment before the processing system executes the task processing request, which may be determined based on the current operating state of different objects in the target environment. The above-mentioned target state may refer to the state of the target environment after the processing system executes the task processing request, which may be determined based on the control parameters contained in the operation task currently required to be performed.

[0121] In an optional solution of this embodiment, in order to accurately and reasonably formulate an action plan, the processing system can first use the pre-trained multimodal reasoning model to analyze the acquired target task information and target environment image to determine the initial state and target state of the target environment, and then generate a target action sequence for operating the target object in the target environment based on the task processing request according to these two states. After generating the target action sequence, the processing system can output the target action sequence in the above-mentioned operation interface to facilitate user viewing and adjustment.

[0122] According to an embodiment of the present application, a task processing method is also provided. Figure 7 is a flowchart of another task processing method according to the present application, such as Figure 7 As shown, the method may include the following steps:

[0123] Step S702: Acquire the task processing request and task environment data by calling the first interface.

[0124] The first interface includes a first parameter, a parameter value of the first parameter includes a task processing request and task environment data, the task processing request includes target task information, and the task environment data includes: a target environment image of a target environment.

[0125] The above-mentioned first interface may refer to an interface for retrieving a first parameter, and the first parameter may include at least a parameter value for representing multimodal data. The above-mentioned input instruction may refer to an instruction for obtaining multimodal data. The above-mentioned target environment may refer to the environment in which the processing system is located. The above-mentioned target environment image may be an image obtained by photographing the target environment by an image capturing device or an image generating device, or by image generation based on data generated by different objects in the target environment. The above-mentioned target task information may be a relevant description of an operation task for controlling different objects in the environment pre-configured in the processing system, or text information corresponding to an instruction input by a user for controlling different objects in the target environment. The target environment image and the target task information may be used to assist the processing system in generating a corresponding action plan to ensure the accuracy of the generated action plan.

[0126] In an optional solution of this embodiment, in order to reasonably and accurately generate corresponding action plans for different objects in the target environment, the environmental state of the target environment can be determined in real time based on the real-time data generated by different objects to improve the correlation between the generated action plan and the target environment, thereby ensuring that the generated action plan can be supported by the target environment to improve the rationality and feasibility of the action plan. Based on this, when generating an action plan, if an input instruction input by a user on the operation interface is detected, the processing system can first obtain the parameter value of the first parameter included in the first interface by calling the first interface, that is, the multimodal data contained in the target environment. For example, the control system can obtain the target environment image of the above-mentioned target environment through an image capture device or an image generation device, and at the same time, analyze the task-related description pre-configured in the processing system, or the instruction text corresponding to the control instruction input by the user in real time, so as to obtain the target task information of the operation task for operating the target object in the target environment, and then integrate the above-mentioned target environment image and the target task information to obtain the above-mentioned multimodal data.

[0127] Step S704: Analyze the target task information and the target environment image using the multimodal reasoning model to obtain the initial state and target state of the target environment.

[0128] The initial state is used to represent the state of the target environment before executing the task processing request, and the target state is used to represent the state of the target environment after executing the task processing request.

[0129] The above-mentioned multimodal reasoning model may refer to a pre-trained natural language deep learning model for determining the initial state and the target state. The state of the above-mentioned target environment may be determined by the real-time state of different objects in the target environment. The above-mentioned initial state may refer to the state of the target environment before the processing system executes the task processing request, which may be determined based on the current operating state of different objects in the target environment. The above-mentioned target state may refer to the state of the target environment after the processing system executes the task processing request, which may be determined based on the control parameters contained in the operation task currently required.

[0130] In an optional scheme of the present embodiment, as shown above, taking into account the environmental state of the target environment in which the processing system is located, it will have a great impact on the rationality, accuracy and other properties of the formulated action plan. Therefore, in order to accurately and reasonably formulate an action plan, the processing system can first determine the initial state and target state of the target environment based on the acquired target task information and target environment image. In order to improve the accuracy of the determined initial state and target state, the above-mentioned multimodal reasoning model can be used to process the target task information and target environment image to determine the initial state and target state of the target environment.

[0131] Step S706: Generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state.

[0132] At least one action in the target action sequence is used to operate a target object in the target environment.

[0133] The target action sequence may refer to an action sequence used to represent an action plan, and may be used to operate a target object in a target environment so that the state of the target environment is converted from an initial state to a target state.

[0134] In an optional scheme of the present embodiment, after determining the initial state and target state of the target environment, the processing system can generate a target action sequence for operating the target object in the target environment based on the task processing request according to these two states. For example, the processing system can match the generated initial state and target state to determine the target object that will change state in the target environment, and then determine the operation to be performed on the target object and the corresponding operation parameters according to the target task information used to describe the operation task, and finally generate a target action sequence for controlling the target object according to the operation parameters. After generating the target action sequence, the processing system can also display the sequence to the user in a preset operation interface to facilitate user viewing and modification.

[0135] Step S708: output the target action sequence by calling the second interface.

[0136] The second interface includes a second parameter, and a parameter value of the second parameter includes a target action sequence.

[0137] The second interface may refer to an interface for outputting a target action sequence, and the second parameter may include at least a parameter value for representing the target action sequence.

[0138] In an optional solution of this embodiment, after generating the target action sequence, the processing system may call the second interface to output the parameter value of the second parameter, that is, output the generated target action sequence, for user viewing convenience.

[0139] According to an embodiment of the present application, a task processing device for implementing the above task processing method is also provided. Figure 8 is a structural block diagram of a task processing device according to the present application, such as Figure 8 As shown, the device may include: a data acquisition module 802 , a state determination module 804 and a sequence generation module 806 .

[0140] Among them, the data acquisition module 802 is used to obtain the task processing request and task environment data, wherein the task processing request includes the target task information, and the task environment data includes: the target environment image of the target environment; the state determination module 804 is used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain the initial state and target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; the sequence generation module 806 is used to generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate the target object in the target environment.

[0141] In an embodiment of the present application, the state determination module 804 includes: a first determination unit, used to input the target task information and the target environment image into the multimodal reasoning model, and obtain the initial state output by the multimodal reasoning model; a second determination unit, used to input the target task information, the target environment image and the initial state into the multimodal reasoning model, and obtain the target state output by the multimodal reasoning model.

[0142] In an embodiment of the present application, the state determination module 804 also includes: a data determination unit, used to determine environmental identification data of the target environment, wherein the environmental identification data includes location identifications of different locations in the target environment and object identifications of different objects in the target environment; a state determination unit, used to analyze target task information, target environment images and environmental identification data using a multimodal reasoning model to obtain an initial state and a target state.

[0143] In an embodiment of the present application, the sequence generation module 806 includes one of the following: a first generation unit, used to input target task information, target environment image, initial state and target state into the multimodal reasoning model, and obtain the target action sequence output by the multimodal reasoning model; a second generation unit, used to input the initial state and target state into the data enhancement model, and obtain the target action sequence output by the data enhancement model, wherein the data enhancement model is used to symbolize the initial state and the target state, and generate a target action sequence based on the symbolized initial state and target state.

[0144] In an embodiment of the present application, the above-mentioned device also includes: a first sequence acquisition module, which is used to obtain the target action sequence output by the data enhancement model in response to the normal operation process of the data enhancement model; a second sequence acquisition module, which is used to input the target task information, target environment image, initial state and target state into the multimodal reasoning model in response to an abnormality in the operation process of the data enhancement model, and obtain the target action sequence output by the multimodal reasoning model.

[0145] In an embodiment of the present application, the above-mentioned device also includes: based on the initial state, simulating the operation of the target object according to the target action sequence to determine the result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; when the result state does not match the target state, the model parameters of the multimodal reasoning model are adjusted.

[0146] In an embodiment of the present application, the above-mentioned device also includes: a state simulation module, which is used to simulate the operation of the target object according to the target action sequence on the basis of the initial state to determine the result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; a parameter adjustment module, which is used to adjust the model parameters of the multimodal reasoning model when the result state does not match the target state.

[0147] It should be noted that the above data acquisition module 802, state determination module 804 and sequence generation module 806 correspond to steps S202 to S206 in the above embodiment, and the three modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0148] According to an embodiment of the present application, a control device for an operating device for implementing the control method of the operating device is also provided. Fig. 9 is a structural block diagram of a control device for operating equipment shown in the present application, such as Fig. 9 As shown, the apparatus may include: a first acquisition module 902 , a first determination module 904 , a first generation module 906 and a device control module 908 .

[0149] Among them, the first acquisition module 902 is used to obtain the task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; the first determination module 904 is used to use a multimodal reasoning model to analyze the target task information and the target environment image to obtain the initial state and target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; the first generation module 906 is used to generate a target action sequence corresponding to the task processing request based on at least the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; the device control module 908 is used to control the operating device to execute at least one action in the target action sequence in sequence.

[0150] In an embodiment of the present application, the device control module 908 includes: a first determination unit, which is used to simulate the operation of the target object according to the target action sequence based on the initial state to determine the result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; and a device control unit, which is used to control the operating device to execute at least one action in the target action sequence in sequence when the result state matches the target state.

[0151] It should be noted that the first acquisition module 902, the first determination module 904, the first generation module 906 and the device control module 908 correspond to steps S402 to S408 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0152] According to an embodiment of the present application, a control device for an operating device for implementing the control method of the operating device is also provided. Fig.10is a structural block diagram of another control device for operating equipment shown in the present application, such as Fig.10 As shown, the apparatus may include: a data receiving module 1002 , a second determining module 1004 , a second generating module 1006 and a sequence sending module 1008 .

[0153] Among them, the data receiving module 1002 is used to receive the task processing request and task environment data sent by the operating device, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment in which the operating device is located; the second determination module 1004 is used to use a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; the second generation module 1006 is used to generate a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; the sequence sending module 1008 is used to send the target action sequence to the operating device.

[0154] In an embodiment of the present application, the sequence sending module 1008 includes: a second determination unit, which is used to simulate the operation of the target object according to the target action sequence on the basis of the initial state to determine the result state of the target environment, wherein the result state is used to characterize the state of the target environment after the target object is operated according to the target action sequence; a sequence sending unit, which is used to send the target action sequence to the operating device when the result state matches the target state.

[0155] It should be noted that the data receiving module 1002, the second determining module 1004, the second generating module 1006 and the sequence sending module 1008 correspond to steps S502 to S508 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0156] According to an embodiment of the present application, a task processing device for implementing the above task processing is also provided. Fig.11 is a structural block diagram of another task processing device according to the present application, such as Fig.11 As shown, the device may include: a first display module 1102 and a second display module 1104 .

[0157] Among them, the first display module 1102 is used to respond to the input instructions acting on the operation interface, and display the task processing request and task environment data on the operation interface, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; the second display module 1104 is used to respond to the processing instructions acting on the operation interface, and display the target action sequence corresponding to the task processing request on the operation interface, wherein at least one action in the target action sequence is used to operate the target object in the target environment, and the target action sequence is an action sequence generated based on at least an initial state and a target state of the target environment, the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the initial state and the target state are states obtained by analyzing the target task information and the target environment image using a multimodal reasoning model, and the multimodal reasoning model is a natural language deep learning model.

[0158] It should be noted that the first display module 1102 and the second display module 1104 correspond to steps S602 to S604 in the above embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0159] According to an embodiment of the present application, a task processing device for implementing the above task processing is also provided. Fig.12 is a structural block diagram of another task processing device according to the present application, such as Fig.12 As shown, the apparatus may include: a second acquisition module 1202 , a third determination module 1204 , a third generation module 1206 and a sequence output module 1208 .

[0160] Among them, the second acquisition module 1202 is used to obtain the task processing request and task environment data by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the task processing request and the task environment data, the task processing request includes the target task information, and the task environment data includes: a target environment image of the target environment; the third determination module 1204 is used to analyze the target task information and the target environment image using a multimodal reasoning model to obtain the initial state and the target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; the third generation module 1206 is used to generate a target action sequence corresponding to the task processing request based on at least the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; the sequence output module 1208 is used to output the target action sequence by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target action sequence.

[0161] It should be noted that the second acquisition module 1202, the third determination module 1204, the third generation module 1206 and the sequence output module 1208 correspond to steps S702 to S708 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0162] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0163] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0164] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0165] In this embodiment, the electronic device can execute the program code in the method.

[0166] Optionally, Fig.13This is a structural block diagram of an electronic device according to the present application. As shown in the figure, the electronic device A may include: one or more (only one is shown in the figure) processors 1302, a memory 1304, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0167] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0168] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; use a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; based at least on the initial state and the target state, generate a target action sequence corresponding to the task processing request, wherein at least one action in the target action sequence is used to operate a target object in the target environment.

[0169] Those skilled in the art will appreciate that the structure shown in the figure is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Fig.13 It does not limit the structure of the above electronic device. For example, the electronic device A may also include Fig.13 More or fewer components (such as network interfaces, display devices, etc.) shown in the figure, or having Fig.13 Different configurations are shown.

[0170] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0171] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0172] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0173] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of the target environment; using a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize the state of the target environment before executing the task processing request, and the target state is used to characterize the state of the target environment after executing the task processing request, and the multimodal reasoning model is a natural language deep learning model; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment.

[0174] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0175] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0176] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0177] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0182] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A task processing method, characterized in that: include: Acquire a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image of a target environment; Analyzing the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize a state of the target environment before executing the task processing request, and the target state is used to characterize a state of the target environment after executing the task processing request, the multimodal reasoning model is a natural language deep learning model, and the target state is jointly determined by the target task information and the target environment image; Based at least on the initial state and the target state, a target action sequence corresponding to the task processing request is generated, wherein at least one action in the target action sequence is used to operate a target object in the target environment.

2. The method according to claim 1, characterized in that The using of a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment includes: Inputting the target task information and the target environment image into the multimodal reasoning model, and obtaining the initial state output by the multimodal reasoning model; The target task information, the target environment image and the initial state are input into the multimodal reasoning model, and the target state output by the multimodal reasoning model is obtained.

3. The method according to claim 1, characterized in that The using of a multimodal reasoning model to analyze the target task information and the target environment image to obtain an initial state and a target state of the target environment includes: Determining environment identification data of the target environment, wherein the environment identification data includes location identifications of different locations in the target environment and object identifications of different objects in the target environment; The target task information, the target environment image and the environment identification data are analyzed using the multimodal reasoning model to obtain the initial state and the target state.

4. The method according to claim 1, characterized in that: The generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state comprises one of the following: Inputting the target task information, the target environment image, the initial state and the target state into the multimodal reasoning model, and obtaining the target action sequence output by the multimodal reasoning model; The initial state and the target state are input into a data enhancement model, and the target action sequence output by the data enhancement model is obtained, wherein the data enhancement model is used to symbolize the initial state and the target state, and generate the target action sequence based on the symbolized initial state and target state.

5. The method according to claim 4, characterized in that After inputting the initial state and the target state into the data augmentation model, the method further includes: In response to the running process of the data enhancement model being normal, obtaining the target action sequence output by the data enhancement model; In response to an abnormality in the operation process of the data augmentation model, the target task information, the target environment image, the initial state and the target state are input into the multimodal reasoning model, and the target action sequence output by the multimodal reasoning model is obtained.

6. The method according to any one of claims 1 to 5, characterized in that After generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, the method further includes: Based on the initial state, simulate the operation on the target object according to the target action sequence to determine the result state of the target environment, wherein the result state is used to represent the state of the target environment after the operation on the target object according to the target action sequence; In a case where the result state does not match the target state, model parameters of the multimodal reasoning model are adjusted.

7. A method for controlling an operating device, characterized in that: include: Acquire a task processing request and task environment data, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing the target environment where the operating device is located; Analyzing the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize a state of the target environment before executing the task processing request, and the target state is used to characterize a state of the target environment after executing the task processing request, the multimodal reasoning model is a natural language deep learning model, and the target state is jointly determined by the target task information and the target environment image; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; The operating device is controlled to sequentially execute the at least one action in the target action sequence.

8. The method according to claim 7, characterized in that The controlling the operating device to sequentially execute the at least one action in the target action sequence includes: Based on the initial state, simulate the operation on the target object according to the target action sequence to determine the result state of the target environment, wherein the result state is used to represent the state of the target environment after the operation on the target object according to the target action sequence; When the result state matches the target state, the operating device is controlled to sequentially execute the at least one action in the target action sequence.

9. A method for controlling an operating device, characterized in that: include: Receiving a task processing request and task environment data sent by an operating device, wherein the task processing request includes target task information, and the task environment data includes: a target environment image obtained by photographing a target environment where the operating device is located; Analyzing the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize a state of the target environment before executing the task processing request, and the target state is used to characterize a state of the target environment after executing the task processing request, the multimodal reasoning model is a natural language deep learning model, and the target state is jointly determined by the target task information and the target environment image; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; The target action sequence is sent to the operating device.

10. The method according to claim 9, characterized in that The sending the target action sequence to the operating device includes: Based on the initial state, simulate the operation on the target object according to the target action sequence to determine the result state of the target environment, wherein the result state is used to represent the state of the target environment after the operation on the target object according to the target action sequence; In the case where the result state matches the target state, the target action sequence is sent to the operating device.

11. A task processing method, characterized in that: include: Acquire a task processing request and task environment data by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the task processing request and the task environment data, the task processing request includes target task information, and the task environment data includes: a target environment image of a target environment; Analyzing the target task information and the target environment image using a multimodal reasoning model to obtain an initial state and a target state of the target environment, wherein the initial state is used to characterize a state of the target environment before executing the task processing request, and the target state is used to characterize a state of the target environment after executing the task processing request, the multimodal reasoning model is a natural language deep learning model, and the target state is jointly determined by the target task information and the target environment image; generating a target action sequence corresponding to the task processing request based at least on the initial state and the target state, wherein at least one action in the target action sequence is used to operate a target object in the target environment; The target action sequence is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target action sequence.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 11 when running.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images