Industrial Embodied Intelligence Data Construction and Training Methods and Simulation Experiment Equipment

Abstract

This application provides a method and simulation experimental equipment for constructing and training industrial embodied intelligence data, relating to the field of intelligent manufacturing technology. The method includes: selecting task support data from an industrial scenario knowledge base based on the acquired task execution instructions; collecting task environment data through environmental sensors based on the task support data; constructing first execution parameters for the industrial task based on the task execution instructions, task support data, and task environment data; decomposing the industrial task into multiple consecutive industrial sub-tasks using a thought chain reasoning technique, obtaining second execution parameters for each industrial sub-task; and executing multiple industrial sub-tasks one by one through a sub-task execution model based on the second execution parameters of each industrial sub-task. The method of this application provides a simulation experimental equipment that can flexibly adapt to different industrial scenarios, execute diverse industrial tasks, and fully integrate embodied intelligence characteristics.

Description

Technical Field

[0001] This application relates to the field of intelligent manufacturing, and in particular to a method and simulation experimental equipment for constructing and training industrial embodied intelligent data. Background Technology

[0002] In the field of intelligent manufacturing, simulation experimental equipment has become an indispensable part. However, the deployment of traditional simulation experimental equipment often involves a cumbersome and time-consuming installation and debugging process, and relies on fixed programming to perform industrial operations, making it difficult for them to adapt to flexible production or flexibly changing production scenarios.

[0003] Embodied intelligence can endow simulation experimental equipment with the ability to interact with the environment, as well as the characteristics of self-learning and adaptation, and has broad application prospects in the field of intelligent manufacturing. However, current academic research on embodied intelligence mainly focuses on the development of humanoid robots and advanced algorithms, while research in teaching and scientific research fields is relatively insufficient.

[0004] In fields such as teaching and research, solving real-world industrial problems requires simulating various industrial scenarios to determine the performance of simulation experimental equipment in different industrial tasks. Therefore, how to provide a simulation experimental device that can flexibly adapt to different industrial scenarios, perform diverse industrial tasks, and fully integrate embodied intelligence is a problem that this application urgently needs to solve. Summary of the Invention

[0005] This application provides a method and simulation experimental device for constructing and training industrial embodied intelligence data, and provides a simulation experimental device that can flexibly adapt to different industrial scenarios, perform diverse industrial tasks, and fully integrate embodied intelligence characteristics.

[0006] A first aspect of this application provides a method for constructing and training industrial embodied intelligence data, the method comprising:

[0007] When a task execution instruction is obtained to instruct the simulation experimental equipment to perform an industrial task, task support data is selected from a pre-set industrial scenario knowledge base according to the task execution instruction, and task environment data is collected through a pre-set environmental sensor according to the task support data.

[0008] Based on the task execution instructions, task support data, and task environment data, the first execution parameters of the industrial task are constructed.

[0009] Using the thought chain reasoning technique, the industrial task is decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task are obtained based on the first execution parameter.

[0010] Based on the second execution parameters of each industrial subtask, multiple industrial subtasks are executed one by one using a pre-trained subtask execution model; wherein, the subtask execution model is trained using expert demonstration data.

[0011] In one possible design, the first execution parameters of the industrial task are constructed based on the task execution instructions, task support data, and task environment data, including:

[0012] Supplement task execution instructions based on task support data;

[0013] Align the task environment data with the supplemented task execution instructions, and obtain the first execution parameters based on the supplemented task execution instructions and the aligned task environment data.

[0014] In one possible design, if the industrial scenario knowledge base stores entity model information of the industrial scenario, then the supplemented task execution instructions include the entity model information.

[0015] Align the task environment data with the supplemented task execution instructions, including:

[0016] Based on the task environment data, the first entity point cloud information of the industrial scene is obtained;

[0017] Based on the entity model information and the first entity point cloud information, the entities indicated by the task environment data are aligned with the entities indicated by the supplemented task execution instructions through data fusion and comparison technology.

[0018] For the entity to be identified, after obtaining the first entity point cloud information of the industrial scene based on the task environment data, the method further includes:

[0019] Based on the first entity point cloud information, the second entity point cloud information of the entity to be identified is obtained;

[0020] Based on the entity model information and the second entity point cloud information, the entity to be identified is identified through data fusion and comparison technology.

[0021] In one possible design, task environment data is acquired through pre-set environmental sensors based on task support data, including:

[0022] Based on the task support data, industrial scenarios are selected from multiple preset scenarios;

[0023] Based on the industrial scenario, identify the entities used to interact with the simulation experimental equipment, as well as the environmental sensors used to detect the simulation experimental equipment and / or entities.

[0024] The mission environment data is collected through environmental sensors.

[0025] In one possible design, the subtask execution model stores multiple subtask execution templates, as well as a third execution parameter for each subtask execution template;

[0026] The target industrial subtask is any one of multiple industrial subtasks; for the target industrial subtask, based on the second execution parameters of each industrial subtask, multiple industrial subtasks are executed one by one using a pre-trained subtask execution model, including:

[0027] Based on the target industrial sub-task, select the target sub-task execution template that matches the target industrial sub-task from multiple sub-task execution templates;

[0028] Based on the second execution parameters of the target industrial subtask and the third execution parameters of the target subtask execution template, the fourth execution parameters of the target industrial subtask are obtained.

[0029] Execute the target industrial subtask according to the fourth execution parameter of the target industrial subtask.

[0030] In one possible design, after executing the target industrial subtask according to the fourth execution parameter of the target industrial subtask, the method further includes:

[0031] Obtain the execution results of the target industrial subtask;

[0032] When the execution result indicates that the target industrial subtask has failed and the task environment data is no longer aligned with the supplemented task execution instructions, the second execution parameters of the target industrial subtask are updated using model predictive control technology so that the target industrial subtask can be re-executed based on the updated second execution parameters.

[0033] In one possible design, the solid model information is a three-dimensional model;

[0034] For the target industrial subtask, after executing multiple industrial subtasks one by one using the pre-trained subtask execution model based on the second execution parameters of each industrial subtask, the method further includes:

[0035] Based on the fourth execution parameter and execution result of the target industrial subtask, the entity model information is driven to realize the digital twin of the target industrial subtask.

[0036] In one possible design, before executing multiple industrial subtasks one by one using a pre-trained subtask execution model based on the second execution parameters of each industrial subtask, the method further includes:

[0037] Expert demonstration data was obtained; the expert demonstration data was obtained through the remote master-slave mode of the simulation experimental equipment; the expert demonstration data was labeled.

[0038] Add at least one of several simulated interference factors to the expert demonstration data; among them, the simulated interference factors include simulated lighting interference, simulated occlusion interference, and simulated texture interference;

[0039] The subtask execution model was trained based on expert demonstration data with simulated interference factors added.

[0040] The second aspect of this application provides a simulation experimental device for performing the industrial embodied intelligence data construction and training method of any one of the first aspects;

[0041] The simulation experimental equipment includes: an experimental workbench, and multiple actuators that are detachably mounted on the experimental workbench; among them, the multiple actuators are used together to perform industrial tasks instructed by task execution commands.

[0042] In one possible design, the experimental workbench is composed of multiple movable workbenches spliced ​​together, and each movable workbench has multiple mounting holes.

[0043] The simulation experimental equipment also includes: an interactive mechanism for demonstrating digital twins of industrial tasks;

[0044] The interactive mechanism and each actuator are mounted on the experimental workbench through at least one mounting hole.

[0045] A third aspect of this application provides an industrial embodied intelligence data construction and training apparatus, the apparatus comprising:

[0046] The data acquisition module is used to, when it receives a task execution instruction that instructs the simulation experimental equipment to perform an industrial task, filter out task support data from a pre-set industrial scenario knowledge base according to the task execution instruction, and collect task environment data through a pre-set environmental sensor according to the task support data.

[0047] The first parameter module is used to construct the first execution parameters of the industrial task based on the task execution instructions, task support data, and task environment data.

[0048] The task decomposition module is used to decompose an industrial task into multiple consecutive industrial sub-tasks using the thinking chain reasoning technology, and to obtain the second execution parameters for each industrial sub-task based on the first execution parameters.

[0049] The first execution module is used to execute multiple industrial sub-tasks one by one according to the second execution parameters of each industrial sub-task and the pre-trained sub-task execution model; wherein the sub-task execution model is trained using expert demonstration data.

[0050] A fourth aspect of this application provides an electronic device, including: a memory, and a memory communicatively connected to a processor;

[0051] The memory stores the instructions that the computer executes;

[0052] When the processor executes computer execution instructions stored in memory, it is used to implement the industrial embodied intelligence data construction and training method of any of the first aspects.

[0053] The fifth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the industrial embodied intelligence data construction and training method of any one of the first aspects.

[0054] The sixth aspect of this application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the industrial embodied intelligence data construction and training method of any one of the first aspects.

[0055] This application provides a method and simulation experimental device for constructing and training industrial embodied intelligence data. The method includes: selecting task support data from an industrial scenario knowledge base based on the acquired task execution instructions, and collecting task environment data through environmental sensors based on the task support data; constructing first execution parameters for the industrial task based on the task execution instructions, task support data, and task environment data; decomposing the industrial task into multiple consecutive industrial sub-tasks using a thought chain reasoning technique to obtain second execution parameters for each industrial sub-task; and executing multiple industrial sub-tasks one by one through a sub-task execution model based on the second execution parameters of each industrial sub-task. The following technical effects were achieved: Task support data was obtained by filtering through a pre-set industrial scenario knowledge base, and task environment data was collected through pre-set environmental sensors. This facilitated the construction of the first execution parameters of the industrial task based on the task execution instructions, task support data, and task environment data, thus realizing the construction of embodied intelligent data in the industrial field. Compared with directly obtaining execution parameters from task execution instructions, this improved the ability of the simulation experimental equipment to interact with the physical entity, thereby enhancing the embodied intelligence capability of the simulation experimental equipment. The industrial task was decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task were determined based on the first execution parameters, reducing the execution difficulty of the industrial task. The industrial sub-tasks were executed through a pre-trained sub-task execution model, improving the intelligence level and execution efficiency of the simulation experimental equipment in executing industrial sub-tasks. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A schematic diagram illustrating a scenario for the industrial embodied intelligence data construction and training method provided in this application embodiment;

[0058] Figure 2 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 1 ;

[0059] Figure 3 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 2 ;

[0060] Figure 4 A flowchart illustrating methods for constructing and training intelligent data for industry. Figure 3 ;

[0061] Figure 5 A flowchart illustrating methods for constructing and training intelligent data for industry. Figure 4 ;

[0062] Figure 6 This is a schematic diagram of the structure of the simulation experimental equipment provided in the embodiments of this application;

[0063] Figure 7 A schematic diagram of the structure of the industrial embodied intelligence data construction and training device provided in the embodiments of this application;

[0064] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0065] Figure label:

[0066] 110 - Simulation experimental equipment; 120 - Entity; 130 - Environmental sensors; 141 - First server; 142 - Second server;

[0067] 610 - Experimental workbench; 611 - Movable workbench; 612 - Mounting hole; 620 - Actuator; 630 - Interactive mechanism;

[0068] 710 - Data Acquisition Module; 720 - First Parameter Module; 730 - Task Decomposition Module; 740 - First Execution Module;

[0069] 810 - Processor; 820 - Memory; 830 - Communication components; 840 - Bus. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0071] In this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, nor do they necessarily imply difference. It should be noted that in this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In this application, "at least one" means one or more, and "more than one" means two or more.

[0072] It should be noted that the phrase "at the moment" in this application can refer to the instant a certain situation occurs, or to a period of time after the occurrence of a certain situation; this application does not impose a specific limitation in this regard. Furthermore, the industrial embodied intelligence data construction and training method and simulation experimental equipment provided in this application are merely examples, and may include more or less content. The user information (including but not limited to user device information and user personal information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in one or more embodiments of this application are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0073] To facilitate a clear description of the technical solution of this application, some of the terms and technologies involved in this application are briefly introduced below:

[0074] Simulation experimental equipment: This is a mechatronic device that achieves automatic control through programming and can repeatedly perform tasks. Simulation experimental equipment typically has three or more degrees of freedom, enabling it to simulate human arm movements to complete industrial operations such as handling, welding, or assembly.

[0075] Flexible manufacturing is a production model centered on "the manufacturing system's rapid response to changes in the internal and external environment." It emphasizes the use of intelligent equipment such as simulation labs and dynamic management to achieve flexible switching between production of multiple varieties and small batches of products.

[0076] Embodied intelligence refers to intelligent systems that possess physical entities and can interact with their environment through a perception-action closed loop. The core of embodied intelligence lies in combining artificial intelligence algorithms, such as large-scale models, with robotic hardware to achieve autonomous decision-making and adaptive learning.

[0077] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0078] In the field of intelligent manufacturing, simulation experimental equipment has become an indispensable part, widely used in scenarios such as handling, sorting, welding, and assembly. However, the deployment of traditional simulation experimental equipment often involves a cumbersome and time-consuming installation and debugging process, and relies on fixed programming to perform industrial operations, making it difficult for them to adapt to flexible production or flexibly changing production scenarios. This limitation is mainly reflected in the fact that once the production process or product specifications change, the simulation experimental equipment needs to undergo a tedious reprogramming and debugging process, which is not only time-consuming and labor-intensive, but also increases production costs and reduces production efficiency.

[0079] To overcome these limitations, the concept of embodied intelligence has emerged. Embodied intelligence endows simulation experimental equipment with the ability to interact with the environment, as well as the characteristics of self-learning and adaptation, and has broad application prospects in the field of intelligent manufacturing. This form of intelligence transcends traditional automation based on preset programs, enabling simulation experimental equipment to flexibly adjust its operating strategies according to real-time environmental feedback and task requirements, thereby greatly improving flexibility and response speed.

[0080] However, despite the enormous theoretical potential of embodied intelligence, it still faces numerous challenges in practical applications. This is because current academic research on embodied intelligence primarily focuses on the development of humanoid robots and advanced algorithms, while research in areas such as teaching and scientific research is relatively insufficient.

[0081] In fields such as teaching and research, solving real-world industrial problems requires simulating various industrial scenarios to determine the performance of simulation experimental equipment in different industrial tasks. Therefore, how to provide a simulation experimental device that can flexibly adapt to different industrial scenarios, perform diverse industrial tasks, and fully integrate embodied intelligence is a problem that this application urgently needs to solve.

[0082] Therefore, to address the aforementioned issues, the research found that directly solving complex industrial tasks is quite difficult. To resolve this, complex industrial tasks are decomposed into multiple simple industrial sub-tasks, and execution parameters for each sub-task are defined to reduce the difficulty of execution. Furthermore, pre-stored entity models may not accurately reflect the position and shape of entities. To address this, data from different sources are aligned and fused to enable interaction between the simulation equipment and the environment. Finally, the lack of execution guidelines during industrial task execution is addressed by obtaining expert demonstration data through a remote master-slave mode. Sub-task execution models are then trained based on this data to facilitate the execution of the aforementioned multiple industrial sub-tasks.

[0083] Based on the above-mentioned inventive discovery, the technical solution of this application is proposed.

[0084] The following section introduces the application scenarios for industrial task execution provided in this application.

[0085] Figure 1 This is a schematic diagram illustrating a scenario for the industrial embodied intelligence data construction and training method provided in an embodiment of this application. It should be noted that... Figure 1 The examples shown are merely examples of scenarios in which this application can be applied, to help those skilled in the art understand the technical content of this application, but do not mean that this application cannot be used in other devices, systems, environments or scenarios.

[0086] like Figure 1 As shown, the application scenario includes: simulation experimental equipment 110, entity 120, and environmental sensor 130. Simulation experimental equipment 110 may include a processing mechanism and an execution mechanism. The processing mechanism is communicatively connected to the environmental sensor 130, and the execution mechanism interacts with the entity 120. The processing mechanism is used to acquire task environment data collected by the environmental sensor 130, and based on the task environment data and the task execution instructions it acquires, controls the execution mechanism to execute the industrial tasks indicated by the task execution instructions.

[0087] Furthermore, this application scenario also includes a first server 141, which stores an industrial scenario knowledge base. The processing unit is communicatively connected to the first server 141 and is used to filter and obtain task support data from the industrial scenario knowledge base according to task execution instructions.

[0088] Furthermore, this application scenario also includes a second server 142, which stores pre-trained subtask execution models. The processing unit communicates with the second server 142 to obtain the execution parameters output by the subtask execution models.

[0089] It should be noted that the first server 141 and the second server 142 can be different physical servers, or they can be different virtual partitions of the same physical server. In one possible scenario, the first server 141 and / or the second server 142 are virtual partitions of the processing organization.

[0090] Entity 120 may include an execution entity and an environmental entity, wherein the execution entity is the object of the industrial task; the environmental sensor 130 may include binocular cameras, millimeter-wave radar, lidar, and programmable logic controllers (PLCs), etc., and the environmental sensor 130 may be integrated on the execution mechanism and / or entity 120. In different scenarios, simulation experimental equipment 110, entity 120, and environmental sensor 130 may refer to different devices.

[0091] For example, when the industrial scenario to be simulated is an assembly scenario, the actuator may include a rotating chuck and fastening tools, the execution entity refers to the parts to be assembled, the environmental entity may include a conveyor belt and a container, and the environmental sensor 130 may also include a force sensor and a proximity sensor.

[0092] For example, when the industrial scenario to be simulated is a logistics scenario, the actuator may include a drive mechanism and a hoisting component, the execution entity refers to the part to be transferred, the environmental entity may include driving roads, parking spaces and containers, and the environmental sensor 130 may also include a position sensor and a speed sensor.

[0093] For example, when the industrial scenario to be simulated is a welding scenario, the actuator may include a two-finger gripper and a welding torch, the execution entity refers to the workpiece to be welded, the environmental entity may include a welding power source and a fence, and the environmental sensor 130 may also include a weld seam tracking sensor and a temperature sensor.

[0094] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0095] Figure 2 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 1 .like Figure 2As shown in the embodiments of this application, the executing entity can be an industrial embodied intelligence data construction and training device. This device can be located in an electronic device, and can be a simulation experimental device, specifically the processing mechanism of the simulation experimental device. The industrial embodied intelligence data construction and training method provided in the embodiments of this application includes the following steps:

[0096] S201. When a task execution instruction is obtained to instruct the simulation experimental equipment to perform an industrial task, task support data is selected from a pre-set industrial scenario knowledge base according to the task execution instruction, and task environment data is collected by a pre-set environmental sensor according to the task support data.

[0097] Specifically, task execution instructions can be formulated and input into the simulation equipment by technicians, formulated and input into the simulation equipment by an application based on a task file, or obtained by the simulation equipment from pre-defined automated execution tasks. Task execution instructions are used to instruct the simulation equipment to perform industrial tasks, which may include, but are not limited to, scenario type, execution entity type, and operational requirements.

[0098] Upon receiving a task execution instruction, the system filters relevant task support data from a pre-built industrial scenario knowledge base. This knowledge base contains data assets related to various industrial tasks, stored across various information systems, including Product Lifecycle Management (PLM), Enterprise Resource Planning (ERP), Manufacturing Execution System (MES), and Warehouse Management System (WMS). Through this filtering process, the system obtains task support data that is helpful for the industrial task. This includes parameters of simulation equipment, execution entities, and environmental entities; location information, engineering drawings, and model information of the execution entities; and layout and model information of the environmental entities.

[0099] Upon receiving the task execution command, the system also collects task environment data through pre-set environmental sensors based on the task support data. This task environment data can be real-time, reflecting the status of the simulation equipment, the execution entity, and the environmental entity to ensure the smooth execution of the industrial task. The task environment data may include color mode data and depth data obtained through a binocular camera, a 3D point cloud model obtained through millimeter-wave radar and / or lidar, position information of the simulation equipment obtained through position sensors, and operational information of the simulation equipment obtained through the PLC.

[0100] S202. Based on the task execution instructions, task support data, and task environment data, the first execution parameters of the industrial task are constructed.

[0101] Specifically, the simulation experimental equipment is an object of embodied intelligent control. The executing entity typically cannot directly issue commands for control, but can achieve indirect control through interaction with the simulation experimental equipment. Environmental entities include embodied intelligent interference objects or obstacles, which can be avoided through interaction with the simulation experimental equipment. Based on this, the first execution parameters of the industrial task are constructed through data supplementation, alignment, and fusion operations, using task execution instructions, task support data, and task environment data. The first execution parameters refer to global task planning parameters used to guide the simulation experimental equipment in executing the industrial task, including but not limited to the position, speed, and actions of the simulation experimental equipment at each moment, as well as the constraints of each action. Based on the first execution parameters, the simulation experimental equipment can complete the execution of the industrial task.

[0102] S203. Using the thinking chain reasoning technique, the industrial task is decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task are obtained based on the first execution parameters.

[0103] Specifically, Chain of Thought (CoT) reasoning technology breaks down complex industrial tasks into multiple simple and sequential industrial sub-tasks. CoT reasoning is a method based on logical reasoning and rule matching that reduces the difficulty of executing industrial tasks, helps simulation equipment understand the complexity and hierarchy of these tasks, and thus improves the adaptability and intelligence of the simulation equipment.

[0104] Subsequently, based on the first execution parameters, a second execution parameter is determined for each industrial sub-model to ensure that each industrial sub-model can execute accurately and efficiently. The second execution parameter refers to the task planning parameters used to guide the simulation experimental equipment in executing the corresponding industrial sub-task, including but not limited to the position, speed, and actions performed by the simulation experimental equipment at each moment when executing the corresponding industrial sub-task, as well as the constraints of each action.

[0105] For example, an industrial subtask might involve grasping an entity. Based on the previously obtained task execution instructions, task support data, and task environment data, the geometric model of the entity and the grasping strategy can be derived. The second execution parameters for this industrial subtask would then include: the optimal contact point pixel coordinates of the entity, and the joint angles obtained by numerical inverse kinematics or deep learning models based on the optimal contact point pixel coordinates and the tool center point (TCP).

[0106] S204. Based on the second execution parameters of each industrial subtask, execute multiple industrial subtasks one by one using the pre-trained subtask execution model.

[0107] Specifically, the subtask execution model is trained using expert demonstration data. This data contains standard execution processes and results for various industrial subtasks. Through machine learning algorithms, the execution patterns and characteristics of these industrial subtasks can be learned. When the simulation equipment executes a specific industrial subtask, it invokes the subtask execution model based on the second execution parameter of that subtask.

[0108] During the execution of industrial sub-tasks, simulation equipment and environmental sensors monitor and record changes in various parameters of the equipment and entities in real time. This data is not only displayed in real-time during task execution, ensuring immediate access for technicians and / or third-party data processing servers, but is also summarized and displayed after the task concludes for more in-depth analysis and review. Through this series of monitoring, recording, display, and analysis, technicians and / or third-party data processing servers can understand the performance of the simulation equipment in executing the industrial task within the specific industrial scenario, providing data support for solving real-world industrial problems.

[0109] This application provides an industrial task execution method, which includes: selecting task support data from an industrial scenario knowledge base based on an acquired task execution instruction, and collecting task environment data through environmental sensors based on the task support data; constructing first execution parameters for the industrial task based on the task execution instruction, task support data, and task environment data; decomposing the industrial task into multiple consecutive industrial sub-tasks using a thought chain reasoning technique to obtain second execution parameters for each industrial sub-task; and executing multiple industrial sub-tasks one by one according to the second execution parameters of each industrial sub-task through a sub-task execution model. The following technical effects were achieved: Task support data was obtained by filtering through a pre-set industrial scenario knowledge base, and task environment data was collected through pre-set environmental sensors. This facilitated the construction of the first execution parameters of the industrial task based on the task execution instructions, task support data, and task environment data, thus realizing the construction of embodied intelligent data in the industrial field. Compared with directly obtaining execution parameters from task execution instructions, this improved the ability of the simulation experimental equipment to interact with the physical entity, thereby enhancing the embodied intelligence capability of the simulation experimental equipment. The industrial task was decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task were determined based on the first execution parameters, reducing the execution difficulty of the industrial task. The industrial sub-tasks were executed through a pre-trained sub-task execution model, improving the intelligence level and execution efficiency of the simulation experimental equipment in executing industrial sub-tasks.

[0110] Figure 3 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 2 .like Figure 3 As shown, the industrial task execution method provided in this application embodiment is... Figure 2 Based on the industrial task execution method provided in the embodiments, this method is further refined. When a task execution instruction is obtained to instruct the simulation experimental equipment to perform an industrial task, the industrial task execution method provided in this application includes the following steps.

[0111] S301. Based on the task execution instructions, select and obtain task support data from the pre-set industrial scenario knowledge base.

[0112] Furthermore, before obtaining task-supporting data, the search scope of the industrial scenario knowledge base can be narrowed based on user identity, the type and location of simulation experimental equipment, and process information to improve screening efficiency.

[0113] S302. Based on the task support data, select industrial scenarios from multiple preset scenarios.

[0114] Specifically, based on the task support data obtained from the above embodiments, the industrial scenario that best matches the current industrial task is selected from multiple preset scenarios, such as assembly scenarios, logistics scenarios, and welding scenarios.

[0115] S303. Based on the industrial scenario, determine the entities used to interact with the simulation experimental equipment, and the environmental sensors used to detect the simulation experimental equipment and / or the entities.

[0116] Specifically, based on the industrial scenario, it can be determined which execution entities and environmental entities the simulation experimental equipment needs to interact with, and which environmental sensors are needed to monitor the simulation experimental equipment, execution entities, and environmental entities.

[0117] S304. Obtain mission environment data through environmental sensors.

[0118] S305. Supplement task execution instructions based on task support data.

[0119] Specifically, after receiving the task execution instruction, the system first uses an attention mechanism to dynamically capture contextual information about the task execution instruction and other user input text, such as the currently viewed order and the parameters of the simulation experimental equipment.

[0120] Secondly, it integrates with industrial scenario knowledge bases such as PLM, ERP, MES, and WMS. For example, it can access PLM to query the design parameters and processes of execution entities, access ERP to query order delivery times and bills of materials (BOM), access MES to query the equipment status and product production progress of simulation experimental equipment, and access WMS to query the layout diagram of environmental entities. Among these, the process can reveal the execution entities that the simulation experimental equipment can process and handle; the BOM can reveal which raw materials the execution entities are made from or which parts they are assembled from; and the layout diagram can reveal the activity range and routes of the simulation experimental equipment in the industrial scenario.

[0121] Finally, task execution instructions are supplemented based on the screened task support data. Specifically, the natural language instructions in the task support data are mapped to the industrial operation template where the task execution instructions are located to supplement the missing task execution instructions.

[0122] The industrial scenario knowledge base stores entity model information of industrial scenarios, such as two-dimensional planar models or three-dimensional solid models. Therefore, the supplementary task execution instructions include this entity model information. After executing S305, execution continues with S306.

[0123] S306. Based on the task environment data, obtain the first entity point cloud information of the industrial scene.

[0124] Specifically, task environment data and task execution instructions come from different hardware devices, thus requiring time synchronization and spatial alignment. Time synchronization can be achieved through timestamps, ensuring their timestamps are consistent; spatial alignment can be achieved through coordinate transformation, ensuring their coordinate systems are aligned. The time-synchronized and spatially aligned data are then fused to obtain the initial execution parameters for the industrial task.

[0125] During spatial alignment, it is necessary to first construct the first entity point cloud information of the industrial scene. Specifically, using the task environment data such as color mode data, depth data, 3D point cloud model, and position information obtained in the above embodiments, the first entity point cloud information of the industrial scene, including simulation experimental equipment, execution entities, and environmental entities, can be constructed.

[0126] S307. Based on the entity model information and the first entity point cloud information, the entity indicated by the task environment data is aligned with the entity indicated by the supplemented task execution instructions through data fusion and comparison technology.

[0127] Specifically, data fusion and comparison technology is a technique used to integrate and compare data from different sources. Through data fusion and comparison technology, entities indicated by task environment data can be aligned with entities indicated by supplementary task execution instructions, thereby achieving spatial alignment between task environment data indications and supplementary task execution instructions.

[0128] In one possible design, the first entity point cloud information, in addition to achieving the aforementioned spatial alignment, can also enable the identification of the entity to be identified. Specifically, the entity perceived through task environment data may have errors in sample identification. Therefore, for the entity to be identified, after executing S306, the following technical solution can be continued:

[0129] Based on the first entity point cloud information, the second entity point cloud information of the entity to be identified is obtained;

[0130] Based on the entity model information and the second entity point cloud information, the entity to be identified is identified through data fusion and comparison technology.

[0131] Specifically, through data fusion and comparison technology, the three-dimensional point cloud model of the executing entity can be aligned with its two-dimensional planar model or three-dimensional solid model.

[0132] Furthermore, to improve data fusion and recognition efficiency, we can first obtain the two-dimensional first entity point cloud information or the second entity point cloud information, i.e., the two-dimensional shape outline, and compare it with the two-dimensional planar model in the industrial scene knowledge base. If we cannot align them precisely, we can then compare the key features in the two-dimensional shape outline and the planar model, such as aperture parameters. If we still cannot locate them precisely, we can then compare the three-dimensional first entity point cloud information or the second entity point cloud information with the three-dimensional stereo model in the industrial scene knowledge base.

[0133] S308. Obtain the first execution parameters based on the supplemented task execution instructions and the aligned task environment data.

[0134] S309. Using the thinking chain reasoning technique, the industrial task is decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task are obtained based on the first execution parameters.

[0135] S310. Based on the second execution parameters of each industrial subtask, execute multiple industrial subtasks one by one using the pre-trained subtask execution model.

[0136] The technical effect of this application embodiment is that: the entity indicated by the task environment data is aligned with the entity indicated by the supplemented task execution instruction, thereby realizing the alignment of the entity model and the entity point cloud; the first execution parameter is obtained according to the supplemented task execution instruction and the aligned task environment data, thereby realizing the alignment and fusion of data from different sources and the matching of execution entities.

[0137] In one possible design, Figure 4 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 3 The subtask execution model stores multiple subtask execution templates and a third execution parameter for each template; the target industrial subtask is any one of these industrial subtasks. For the target industrial subtask, such as... Figure 4 As shown, S310 includes:

[0138] S401. Based on the target industrial sub-task, select the target sub-task execution template that matches the target industrial sub-task from multiple sub-task execution templates.

[0139] Specifically, the simulation equipment selects from multiple pre-stored sub-task execution templates based on the target industrial sub-task to be executed. These sub-task execution templates are pre-designed, each corresponding to a specific type of industrial sub-task and containing a series of parameters and steps required to execute that sub-task, i.e., the third execution parameters. By comparing the template with the target industrial sub-task, the simulation equipment selects the target sub-task execution template that matches the target industrial sub-task, providing guidance for subsequent execution.

[0140] S402. Based on the second execution parameters of the target industrial subtask and the third execution parameters of the target subtask execution template, the fourth execution parameters of the target industrial subtask are obtained.

[0141] Specifically, after determining the execution template for the target sub-task, the simulation experimental equipment will conduct a comprehensive analysis and adjustment based on the second execution parameters of the target industrial sub-task and the third execution parameters of the target sub-task execution template, ultimately obtaining a fourth execution parameter for the target industrial sub-task. This fourth execution parameter will be directly used to guide the simulation experimental equipment in executing the target industrial sub-task.

[0142] S403. Execute the target industrial subtask according to the fourth execution parameter of the target industrial subtask.

[0143] Specifically, the simulation equipment executes the target industrial sub-task based on this fourth execution parameter. For example, the processing mechanism generates Robot Operating System (ROS) code based on these four execution parameters, so as to control the actuator to execute the target industrial sub-task according to the ROS code; during this process, the actuator adjusts its own motion trajectory and operating force based on the ROS code to ensure the accurate execution of the target industrial sub-task.

[0144] S404. Obtain the execution results of the target industrial subtask.

[0145] S405. When the execution result indicates that the target industrial subtask has failed and the task environment data is no longer aligned with the supplemented task execution instructions, the second execution parameters of the target industrial subtask are updated using model predictive control technology so that the target industrial subtask can be re-executed according to the updated second execution parameters of the target industrial subtask.

[0146] Specifically, the simulation experimental equipment connects industrial sub-tasks to ensure the transmission of the execution results of the industrial sub-tasks and the handling of execution failures.

[0147] When the execution result indicates that the target industrial subtask has been successfully executed, continue to execute the next industrial subtask following the steps described above;

[0148] The retry mechanism for the target industrial subtask is triggered when the execution result indicates that the target industrial subtask has failed, and the task environment data is still aligned with the supplemented task execution instructions.

[0149] When the execution result indicates that the target industrial subtask has failed, and the task environment data is no longer aligned with the supplemented task execution instructions (this misalignment may be due to changes in the environmental entities, or at least one malfunction in the simulation equipment, the execution entity, or the environmental sensors), the simulation equipment will take a series of measures to address this. One effective method is to update the second execution parameters of the target industrial subtask using Model Predictive Control (MPC) technology. MPC is an advanced control strategy that can cope with uncertainties and dynamically changing environments by predicting future system states and optimizing the control strategy. The simulation equipment will then re-execute the target industrial subtask based on the updated second execution parameters.

[0150] Furthermore, by updating the first execution parameters and even the task environment data through MPC, the second execution parameters of the target industrial subtask are updated.

[0151] The technical effect of this application embodiment is that multiple industrial sub-tasks are executed by storing multiple sub-task execution templates in the sub-task execution model, and the execution results of each industrial sub-task are monitored, which improves the accuracy and reliability of the simulation experimental equipment when executing complex industrial tasks.

[0152] The following is a specific example of an industrial task execution method provided in the embodiments of this application, which includes the following steps.

[0153] First, the simulation equipment receives task execution instructions formulated and input by technicians. These instructions include: "Find the missing parts of the gearbox prototype."

[0154] Secondly, the simulation experimental equipment filters out task support data based on the task execution instructions and supplements the task execution instructions based on the task support data. The supplemented task execution instructions include: the industrial scenario is a logistics scenario; the execution mechanism includes a drive mechanism and a two-finger gripper; the execution entity includes a prototype X-type gearbox, as well as the missing Y-type bearing and Z-type gear; the environmental entities include shelves, driving roads, and workbenches; and the environmental sensors include a binocular camera, millimeter-wave radar, lidar, and position sensors.

[0155] Furthermore, the simulation equipment decomposes the industrial task into multiple consecutive industrial sub-tasks and obtains the fourth execution parameter for each sub-task. These multiple industrial sub-tasks include:

[0156] (1) The AGV moves to the shelf on the driving road;

[0157] (2) Storage cabinet with two-finger gripper to pull open the shelf;

[0158] (3) Two-finger grippers grasp Y-type bearings;

[0159] (4) The AGV moves to the workbench on the driving road;

[0160] (5) Place Y-type bearings using two-finger grippers;

[0161] ...

[0162] Taking the industrial subtask (2) as an example, the fourth execution parameter of the industrial subtask (2) includes: the angle of the two-finger gripper is always greater than 60°, and the initial position and end position of the two-finger gripper and the storage cabinet respectively.

[0163] Finally, the simulation equipment executes multiple industrial sub-tasks one by one according to the fourth execution parameter of each industrial sub-task.

[0164] In one possible design, the solid model information is a three-dimensional model. Therefore, after S405, the method further includes:

[0165] S406. Based on the fourth execution parameter and execution result of the target industrial sub-task, drive the entity model information to realize the digital twin of the target industrial sub-task.

[0166] Specifically, high-precision 3D models are pre-built for the simulation experimental equipment, execution entities, environmental entities, and environmental sensors. These entity model information recreates their actual dimensions, motion joints, and physical properties. Simultaneously, the entity model information is driven by the fourth execution parameter and execution result of the target industrial sub-task, realizing a digital twin of the industrial scene.

[0167] Digital twins support multi-view observation and interactive operation, enabling the extension of local industrial scenarios to front-end and back-end processes to simulate the complete industrial task execution process.

[0168] It is understood that the target industrial subtask is any one of multiple industrial subtasks. Therefore, the execution method of the simulation experimental equipment to execute other industrial subtasks besides the target industrial subtask is similar to the execution method of the target robot to execute the target industrial subtask, and will not be repeated in the embodiments of this application.

[0169] In one possible design, Figure 5 A flowchart illustrating the industrial embodied intelligence data construction and training method provided in the embodiments of this application. Figure 4 .like Figure 5 As shown, prior to S310, the method further includes:

[0170] S501. Obtain expert demonstration data.

[0171] Specifically, expert demonstration data is obtained through a remote master-slave mode of the simulation equipment. Using this mode, technicians (the master) operate and control the simulation equipment (the slave) to execute industrial sub-tasks, generating high-quality expert demonstration data. Multiple rounds of remote operation are used, covering different industrial scenarios and operational strategies, thus generating diverse expert demonstration data. Each round of remote operation can target different execution entities and parameters to ensure data diversity and generalization capabilities. Afterward, the collected expert demonstration data is annotated; for example, in an assembly scenario, annotations may include gripping points, assembly poses, and transport paths.

[0172] S502, Add at least one of a variety of simulated interference factors to the expert demonstration data.

[0173] Specifically, various simulated interference factors include simulated lighting interference, simulated occlusion interference, and simulated texture interference. By adding simulated interference factors, the adaptability of the sub-task execution model to disturbances is improved. For example, different lighting environments such as simulated natural light or simulated lamps are added, with different brightness levels set by ±30%, as simulated lighting interference; another example is using randomly shaped dynamic obstacles to randomly occlude 10%-50% of the target object's area as simulated occlusion interference; yet another example is setting the texture features of the target object as injection-molded material, metal material, or spray-painted material, etc., as simulated texture interference.

[0174] S503. Based on the expert demonstration data after adding simulated interference factors, a subtask execution model is trained.

[0175] Specifically, a policy network is trained by imitation learning algorithm, enabling the simulation experimental equipment to mimic the operation of an expert. The output of the policy network is then converted into control signals for the simulation experimental equipment through inverse kinematics or reinforcement learning methods, thereby training a sub-task execution model. This sub-task execution model is used in the industrial task execution method described in the above embodiments.

[0176] The technical effects of this application embodiment are: expert demonstration data is obtained through remote master-slave mode, and a sub-task execution model is trained based on the expert demonstration data, providing a training method for the sub-task execution model; by adding simulated interference factors, the adaptability of the sub-task execution model to disturbances is improved.

[0177] In one possible design, comprehensive consideration is given to task success rewards used to incentivize simulation equipment to complete industrial sub-tasks. Robust rewards used to encourage simulation equipment to maintain stable performance under disturbances. And task efficiency rewards used to encourage simulation experimental equipment to perform industrial sub-tasks efficiently. And based on this, a robust objective function is set for the subtask execution model. ,but The calculation formula is:

[0178]

[0179] in, These are the weighting coefficients. These are used to balance the contributions of different rewards. Typically, these three parameters are developed and entered by technical staff based on their experience.

[0180] The technical effect of this application embodiment is that, through the robustness objective function, the contributions of task success reward, robustness reward and task efficiency reward to the sub-task execution model are comprehensively considered.

[0181] This application also provides a simulation experiment device for performing the industrial embodied intelligence data construction and training method described above. Figure 6 This is a schematic diagram of the simulation experimental equipment provided in an embodiment of this application. Figure 6 As shown, the simulation experimental equipment includes: an experimental workbench 610, and multiple actuators 620 detachably mounted on the experimental workbench 610; wherein, the multiple actuators 620 are used together to perform industrial tasks indicated by task execution instructions.

[0182] Specifically, the experimental workbench 610 is the basic platform for the operation and task execution of the simulation experimental equipment. Its design allows for the detachable installation of multiple actuators 620, so as to be flexibly configured according to industrial scenarios and industrial tasks.

[0183] Furthermore, the simulation experimental equipment also includes processing mechanisms installed inside the experimental workbench 610, such as edge computing units, PLCs, power supplies, and wiring. The cabinet door of the experimental workbench 610 is designed to be transparent, allowing observation of the operating status indicators and wiring of the internal processing mechanisms.

[0184] Furthermore, at least one of the environmental entity, the execution entity, and the environmental sensor is detachably mounted on the experimental workbench 610.

[0185] In one possible design, the experimental workbench 610 is composed of multiple movable workbenches 611 spliced ​​together, and each movable workbench 611 has multiple mounting holes 612.

[0186] The simulation experimental equipment also includes: an interactive mechanism 630 for demonstrating digital twins of industrial tasks;

[0187] The interactive mechanism 630 and each actuator 620 are mounted on the experimental workbench 610 through at least one mounting hole 612.

[0188] Specifically, the experimental workbench 610 is composed of at least one movable workbench 611, which is convenient for installation, disassembly and transportation. The table surface of each movable workbench 611 is designed with multiple mounting holes 612, including small holes for installation and disassembly and large holes for wiring, which facilitates the quick installation and disassembly of the actuator 620, so as to customize and quickly build new industrial scenarios according to needs.

[0189] The interactive mechanism 630 may include a display, a voice interaction module, and an audible and visual alarm module. Technicians can use the interactive mechanism 630 to understand the digital twin of the industrial task in real time. In addition, they can also understand the changes in various parameters of the simulation experimental equipment and the physical entity in real time.

[0190] Figure 7 This is a schematic diagram of the structure of the industrial embodied intelligence data construction and training device provided in an embodiment of this application. Figure 7 As shown in this embodiment, the industrial task execution device can be located in an electronic device. The industrial embodied intelligence data construction and training device includes:

[0191] The data acquisition module 710 is used to, when it receives a task execution instruction for instructing the simulation experimental equipment to perform an industrial task, filter out task support data from a pre-set industrial scenario knowledge base according to the task execution instruction, and collect task environment data through a pre-set environmental sensor according to the task support data.

[0192] The first parameter module 720 is used to construct the first execution parameters of the industrial task based on the task execution instructions, task support data and task environment data;

[0193] The task decomposition module 730 is used to decompose an industrial task into multiple consecutive industrial sub-tasks using the thinking chain reasoning technology, and to obtain the second execution parameters of each industrial sub-task based on the first execution parameters.

[0194] The first execution module 740 is used to execute multiple industrial sub-tasks one by one according to the second execution parameters of each industrial sub-task and through a pre-trained sub-task execution model; wherein the sub-task execution model is trained using expert demonstration data.

[0195] The industrial embodied intelligence data construction and training device provided in this application embodiment can perform... Figure 2 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figure 2 The methods shown in the embodiments are similar, and will not be described again in the embodiments of this application.

[0196] Meanwhile, the industrial embodied intelligence data construction and training device provided in this application embodiment is a further refinement of the industrial embodied intelligence data construction and training device provided in the previous application embodiment.

[0197] In one possible design, parameter first module 720 includes:

[0198] The data supplementation module is used to supplement task execution instructions based on task support data;

[0199] The data alignment module is used to align the task environment data with the supplemented task execution instructions, and obtain the first execution parameters based on the supplemented task execution instructions and the aligned task environment data.

[0200] In one possible design, if the industrial scenario knowledge base stores entity model information of the industrial scenario, then the supplemented task execution instructions include the entity model information.

[0201] The data alignment module includes:

[0202] The point cloud acquisition module is used to obtain the first entity point cloud information in the industrial scene based on the task environment data;

[0203] The fusion comparison module is used to align the entities indicated by the task environment data with the entities indicated by the supplemented task execution instructions based on the entity model information and the first entity point cloud information through data fusion comparison technology.

[0204] For the entity to be identified, the point cloud acquisition module is also used to obtain the second entity point cloud information of the entity to be identified based on the first entity point cloud information;

[0205] The fusion comparison module is used to identify the entity to be identified based on the entity model information and the second entity point cloud information through data fusion comparison technology.

[0206] In one possible design, the data acquisition module 710 includes:

[0207] The scene filtering module is used to filter out industrial scenes from multiple preset scenes based on task support data;

[0208] The device determination module is used to determine, based on the industrial scenario, the entities that interact with the simulation experimental equipment, as well as the environmental sensors used to detect the simulation experimental equipment and / or the entities.

[0209] The data acquisition module is used to collect mission environment data through environmental sensors.

[0210] In one possible design, the subtask execution model stores multiple subtask execution templates, as well as a third execution parameter for each subtask execution template;

[0211] The target industrial subtask is any one of multiple industrial subtasks; for the target industrial subtask, the first execution module 740 includes:

[0212] The template filtering module is used to filter out the target subtask execution template that matches the target industrial subtask from multiple subtask execution templates.

[0213] The second parameter module is used to obtain the fourth execution parameter of the target industrial subtask based on the second execution parameter of the target industrial subtask and the third execution parameter of the target subtask execution template.

[0214] The second execution module is used to execute the target industrial subtask according to the fourth execution parameter of the target industrial subtask.

[0215] In one possible design, the industrial embodied intelligence data building and training device also includes:

[0216] The result acquisition module is used to obtain the execution results of the target industrial sub-task;

[0217] The re-execution module is used to update the second execution parameters of the target industrial subtask using model predictive control technology when the execution result indicates that the target industrial subtask has failed and the task environment data is no longer aligned with the supplemented task execution instructions. This allows the target industrial subtask to be re-executed based on the updated second execution parameters.

[0218] In one possible design, the solid model information is a three-dimensional model;

[0219] For the target industrial subtask, the first execution module 740 is also used to drive the entity model information according to the fourth execution parameter and execution result of the target industrial subtask to realize the digital twin of the target industrial subtask.

[0220] In one possible design, the industrial embodied intelligence data building and training device also includes:

[0221] The remote master-slave module is used to acquire expert demonstration data; the expert demonstration data is obtained through the remote master-slave mode of the simulation experimental equipment; the expert demonstration data has been annotated.

[0222] The interference addition module is used to add at least one of a variety of simulated interference factors to expert demonstration data; among them, the various simulated interference factors include simulated lighting interference, simulated occlusion interference, and simulated texture interference;

[0223] The model training module is used to train a subtask execution model based on expert demonstration data with simulated interference factors added.

[0224] The industrial embodied intelligence data construction and training device provided in this application embodiment can perform... Figures 2 to 5 The technical solution of the method embodiment shown has the same implementation principle and technical effect as... Figures 2 to 5 The methods shown in the embodiments are similar, and will not be described again in the embodiments of this application.

[0225] This application also provides an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device includes at least one processor 810 and a memory 820. The electronic device also includes a communication component 830. The processor 810, memory 820, and communication component 830 are connected via a bus 840.

[0226] In the specific implementation process, at least one processor 810 executes computer execution instructions stored in memory 820, so that at least one processor 810 is used to implement the industrial embodied intelligence data construction and training method of the above embodiment.

[0227] The specific implementation process of processor 810 can be found in the above method embodiments, and its implementation principle and technical effect are similar. The embodiments of this application will not be repeated here.

[0228] In the above embodiments, it should be understood that the processor 810 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0229] The memory 820 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage.

[0230] Bus 840 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 840 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 840 in the accompanying drawings of this application is not limited to only one bus or one type of bus.

[0231] The above description addresses the functions implemented by electronic devices and main control devices, and introduces the solutions provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments disclosed in this application, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of this application.

[0232] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these instructions are used to implement the industrial embodied intelligence data construction and training method described above. In the specific implementation of the aforementioned industrial embodied intelligence data construction and training method, each module can be implemented as a processor.

[0233] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0234] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0235] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the industrial embodied intelligence data construction and training method described above.

[0236] The computer program is stored in a readable storage medium, and at least one processor can read the computer program from the readable storage medium and execute the computer program to perform the scheme provided in any of the above embodiments.

[0237] Those skilled in the art will understand that all or part of the steps in the above-described embodiments can be implemented using hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps included in the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0238] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An industrial embodied intelligence data construction and training method, characterized in that, The method includes: When a task execution instruction is obtained to instruct the simulation experimental equipment to perform an industrial task, task support data is selected from a pre-set industrial scenario knowledge base according to the task execution instruction, and task environment data is collected by a preset environmental sensor according to the task support data. Based on the task execution instructions, the task support data, and the task environment data, the first execution parameters of the industrial task are constructed. Using the thought chain reasoning technique, the industrial task is decomposed into multiple consecutive industrial sub-tasks, and the second execution parameters of each industrial sub-task are obtained based on the first execution parameters. Based on the second execution parameters of each industrial subtask, the multiple industrial subtasks are executed one by one using a pre-trained subtask execution model; wherein the subtask execution model is trained using expert demonstration data.

2. The method of claim 1, wherein, The first execution parameters for the industrial task are constructed based on the task execution instructions, the task support data, and the task environment data, including: The task execution instructions are supplemented based on the task support data. Align the task environment data with the supplemented task execution instructions, and obtain the first execution parameters based on the supplemented task execution instructions and the aligned task environment data.

3. The method according to claim 2, characterized in that, The industrial scenario knowledge base stores entity model information of industrial scenarios, so the supplemented task execution instructions include the entity model information; Aligning the task environment data with the supplemented task execution instructions includes: Based on the task environment data, the first entity point cloud information of the industrial scene is obtained; Based on the entity model information and the first entity point cloud information, the entity indicated by the task environment data is aligned with the entity indicated by the supplemented task execution instruction through data fusion and comparison technology. For the entity to be identified, after obtaining the first entity point cloud information of the industrial scene based on the task environment data, the method further includes: Based on the first entity point cloud information, the second entity point cloud information of the entity to be identified is obtained; Based on the entity model information and the second entity point cloud information, the entity to be identified is identified through the data fusion and comparison technology.

4. The method according to claim 3, characterized in that, The step of acquiring task environment data through preset environmental sensors based on the task support data includes: Based on the task support data, the industrial scenario is selected from multiple preset scenarios; Based on the industrial scenario, determine the entity for interacting with the simulation experimental equipment, and the environmental sensors for detecting the simulation experimental equipment and / or the entity; The mission environment data is collected through the environmental sensors.

5. The method according to claim 3, characterized in that, The subtask execution model stores multiple subtask execution templates and a third execution parameter for each subtask execution template; The target industrial subtask is any one of the plurality of industrial subtasks; For the target industrial sub-task, the step of executing the multiple industrial sub-tasks one by one according to the second execution parameters of each industrial sub-task and through a pre-trained sub-task execution model includes: Based on the target industrial sub-task, select the target sub-task execution template that matches the target industrial sub-task from the plurality of sub-task execution templates; The fourth execution parameter of the target industrial subtask is obtained based on the second execution parameter of the target industrial subtask and the third execution parameter of the target subtask execution template. The target industrial subtask is executed according to the fourth execution parameter of the target industrial subtask.

6. The method according to claim 5, characterized in that, After executing the target industrial subtask according to the fourth execution parameter of the target industrial subtask, the method further includes: Obtain the execution result of the target industrial subtask; When the execution result indicates that the target industrial subtask has failed and the task environment data is no longer aligned with the supplemented task execution instructions, the second execution parameters of the target industrial subtask are updated using model predictive control technology so that the target industrial subtask can be re-executed according to the updated second execution parameters.

7. The method according to claim 5, characterized in that, The entity model information is a three-dimensional model; For the target industrial sub-task, after executing the plurality of industrial sub-tasks one by one using a pre-trained sub-task execution model based on the second execution parameters of each industrial sub-task, the method further includes: Based on the fourth execution parameter and execution result of the target industrial subtask, the entity model information is driven to realize the digital twin of the target industrial subtask.

8. The method according to claim 1, characterized in that, Before executing the plurality of industrial sub-tasks one by one according to the second execution parameters of each industrial sub-task and through the pre-trained sub-task execution model, the method further includes: The expert demonstration data is obtained through the remote master-slave mode of the simulation experimental equipment; the expert demonstration data has been annotated. At least one of a variety of simulated interference factors is added to the expert demonstration data; wherein, the variety of simulated interference factors includes simulated lighting interference, simulated occlusion interference, and simulated texture interference; The subtask execution model was trained based on expert demonstration data with simulated interference factors added.

9. A simulation experimental device, characterized in that, The simulation experimental equipment is used to execute the industrial embodied intelligence data construction and training method as described in any one of claims 1 to 8; The simulation experimental equipment includes: an experimental workbench, and multiple actuators detachably mounted on the experimental workbench; wherein, the multiple actuators are used together to perform industrial tasks indicated by task execution instructions.

10. The simulation experimental equipment according to claim 9, characterized in that, The experimental workbench is composed of multiple movable workbenches spliced ​​together, and each movable workbench has multiple mounting holes. The simulation experimental equipment also includes: an interactive mechanism for displaying a digital twin of the industrial task; The interactive mechanism and each of the actuators are mounted on the experimental workbench through at least one of the mounting holes.

Images