Motion planning method, training method and system for embodied intelligence model

Abstract

The embodiment of the present application provides a method, training method and system for motion planning of an embodied intelligence model, including: obtaining an embodied intelligence initial model; wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer; obtaining a training set of the model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language sub-sample, an in vitro perception sub-sample, an ontology cognition sub-sample, and a motion planning sub-sample; wherein the natural language sub-sample, in vitro perception sub-sample, ontology cognition sub-sample, and motion planning sub-sample in each training sample correspond to each other one by one; according to the training set of the model, the embodied intelligence initial model is trained to obtain an embodied intelligence model. This method is used to achieve the effect of improving the accuracy of motion control based on the embodied intelligence model obtained through training.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, training method and system for motion planning of an embodied intelligent model. Background Art

[0002] With the development of artificial intelligence technology, embodied intelligence technology has appeared in various fields of production and life, providing more convenience for people's production and life.

[0003] In the prior art, the embodied intelligence model mainly completes the corresponding motion planning based on natural language instructions and environmental perception information.

[0004] However, for different embodied intelligent terminals, adopting the embodied intelligent model described in the prior art will lead to the problem of inaccurate motion planning of the embodied intelligence. Summary of the invention

[0005] The embodiments of the present application provide a method, a training method, and a system for motion planning of an embodied intelligent model, so as to achieve the effect of improving the accuracy of motion control of the embodied intelligent model obtained through training.

[0006] In a first aspect, an embodiment of the present application provides a training method for an embodied intelligence model for motion planning, which is applied to an embodied intelligence server, including:

[0007] Acquire an initial model of embodied intelligence; wherein the initial model of embodied intelligence includes: an initial natural language understanding layer, an initial external perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer;

[0008] Acquire a training set of a model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language subsample, an in vitro perception subsample, an ontology recognition subsample, and a motion planning subsample; wherein the natural language subsample, the in vitro perception subsample, the ontology recognition subsample, and the motion planning subsample in each training sample correspond to each other;

[0009] The embodied intelligence initial model is trained according to the training set of the model to obtain an embodied intelligence model.

[0010] In a possible implementation, the embodied intelligence initial model is trained according to the training set of the model to obtain the embodied intelligence model, including: extracting a training sample from the training set of the model, and training the initial natural language understanding layer, the initial extracorporeal perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model according to the extracted training sample and the fusion layer in the embodied intelligence initial model to obtain the trained embodied intelligence initial model; if each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions, the embodied intelligence model is obtained; otherwise, repeating the above training process until each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions.

[0011] In a possible implementation, the initial natural language understanding layer, the initial extracorporeal perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model are trained according to the extracted training samples and the fusion layer in the embodied intelligence initial model to obtain the trained embodied intelligence initial model, including: obtaining a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value according to the extracted training samples and the embodied intelligence initial model; wherein the first loss value is the loss value corresponding to the initial natural language understanding layer; the second loss value is the loss value corresponding to the initial extracorporeal perception layer; The third loss value is the loss value corresponding to the initial ontology cognition layer; the fourth loss value is the loss value corresponding to the initial motion planning layer; the fifth loss value is the loss value obtained according to the fusion layer; according to the first loss value and the fifth loss value, the parameters of the initial natural language understanding layer are adjusted, according to the second loss value and the fifth loss value, the parameters of the initial in-vitro perception layer are adjusted, according to the third loss value and the fifth loss value, the parameters of the initial ontology cognition layer are adjusted, according to the fourth loss value and the fifth loss value, the parameters of the initial motion planning layer are adjusted, so as to obtain the trained embodied intelligence initial model.

[0012] In a possible implementation, the natural language subsample includes user instructions and an embodied intelligent action sequence corresponding to the user instructions; the extracorporeal perception subsample includes environmental information and object categories and object postures corresponding to the environmental information; the ontology cognition subsample includes ontology parameter information and a terminal target posture corresponding to the ontology parameter information; and the motion planning subsample includes a joint angle planning sequence.

[0013] In a possible implementation, according to the extracted training samples and the embodied intelligence initial model, a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value are obtained, including: according to the user instructions, the environmental information, the entity parameter information, and the embodied intelligence initial model included in the extracted training samples, a predicted embodied intelligence action sequence, a predicted item category, a predicted item posture, a predicted terminal target posture, a predicted joint angle planning sequence, and an instruction feature vector, an environmental feature vector, an entity feature vector, and a posture feature vector are obtained; according to the predicted embodied intelligence action sequence and the embodied intelligence action sequence, the first loss value is determined; according to the predicted item category, the predicted item posture, and the item category and the item posture, the second loss value is determined; according to the predicted terminal target posture and the terminal target posture, the third loss value is determined; according to the predicted joint angle planning sequence and the joint angle planning sequence, the fourth loss value is determined; according to the instruction feature vector, the environmental feature vector, the entity feature vector, and the posture feature vector, the fifth loss value is determined.

[0014] In a possible implementation, based on the user instructions, the environmental information, the ontology parameter information, and the embodied intelligence initial model included in the extracted training samples, a predicted embodied intelligence action sequence, a predicted object category, a predicted object posture, a predicted terminal target posture, a predicted joint angle planning sequence, and an instruction feature vector, an environmental feature vector, an ontology feature vector, and a posture feature vector are obtained, including: based on the user instructions and the initial natural language understanding layer, determining the instruction feature vector of the user instructions and the predicted embodied intelligence action sequence corresponding to the user instructions; based on the environmental information and the initial natural language understanding layer, determining the instruction feature vector of the user instructions and the predicted embodied intelligence action sequence corresponding to the user instructions; The initial external perception layer determines the environmental feature vector of the environmental information, and the predicted object type and predicted object posture corresponding to the environmental information; according to the ontological parameter information, the embodied intelligent action sequence, the object type, the object posture, and the initial ontological cognitive layer, the ontological feature vector of the ontological parameter information and the predicted terminal target posture corresponding to the ontological parameter information are determined; according to the predicted terminal target posture, the user instruction, the environmental information, and the initial motion planning layer, the posture feature vector of the predicted terminal target posture and the predicted joint angle planning sequence of the embodied intelligent terminal are determined.

[0015] In a possible implementation, the fifth loss value is determined based on the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector, including: inputting the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector into the fusion layer, and outputting a fused feature vector; and determining the fifth loss value based on the fused feature vector.

[0016] In a possible implementation, the method further includes: if the type of the embodied intelligent terminal increases, updating the training set of the model, and updating the training of the embodied intelligent model according to the updated model training set.

[0017] In a second aspect, an embodiment of the present application provides a method for motion planning based on an embodied intelligence model, which is applied to an embodied intelligence server, including:

[0018] In response to a current user instruction sent by the embodied intelligent terminal, obtaining current environment information and current entity parameter information of the embodied intelligent terminal;

[0019] Determine and output motion planning instructions according to the current user instructions, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model;

[0020] Among them, the embodied intelligence model is a model trained based on the first aspect and / or various possible implementation methods of the first aspect as described above.

[0021] In a possible implementation, according to the current user instruction, the current environment information, the current ontology parameter information, and the natural language understanding layer, the extracorporeal perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model, a motion planning instruction is determined and output, including: determining a current embodied intelligent action sequence corresponding to the current user instruction according to the current user instruction and the natural language understanding layer; determining a current object type and a current object posture corresponding to the current environment information according to the current environment information and the extracorporeal perception layer; determining a current terminal target posture corresponding to the current ontology parameter information according to the current ontology parameter information, the current embodied intelligent action sequence, the current object type, the current object posture, and the ontology cognition layer; determining a current joint angle planning sequence of the embodied intelligent terminal according to the current terminal target posture, the current user instruction, the current environment information, and the motion planning layer; determining and outputting a motion planning instruction according to the current joint angle planning sequence of the embodied intelligent terminal.

[0022] In a third aspect, an embodiment of the present application provides a training device for an embodied intelligent model for motion planning, comprising:

[0023] A first acquisition module is used to acquire an embodied intelligence initial model; wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial external perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer;

[0024] A second acquisition module is used to acquire a training set of a model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language subsample, an in vitro perception subsample, an ontology cognition subsample, and a motion planning subsample; wherein the natural language subsample, the in vitro perception subsample, the ontology cognition subsample, and the motion planning subsample in each training sample correspond to each other;

[0025] The training module is used to train the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model.

[0026] In a fourth aspect, an embodiment of the present application provides a device for motion planning based on an embodied intelligent model, comprising:

[0027] A third acquisition module, configured to acquire current environment information and current entity parameter information of the embodied intelligent terminal in response to a current user instruction sent by the embodied intelligent terminal;

[0028] A planning module, configured to determine and output a motion planning instruction according to the current user instruction, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model;

[0029] Among them, the embodied intelligence model is a model trained based on the first aspect and / or various possible implementation methods of the first aspect as described above.

[0030] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0031] The memory stores computer-executable instructions;

[0032] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect, or the above second aspect and / or various possible implementations of the second aspect.

[0033] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the first aspect and / or various possible implementation methods of the first aspect as above, or the second aspect and / or various possible implementation methods of the second aspect as above.

[0034] In the seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the first aspect above and / or various possible implementations of the first aspect, or the second aspect above and / or various possible implementations of the second aspect.

[0035] In an eighth aspect, an embodiment of the present application provides an embodied intelligent system, comprising: an embodied intelligent server and at least one embodied intelligent terminal; wherein each of the at least one embodied intelligent terminal is connected to the embodied intelligent server;

[0036] The embodied intelligent server is used to store and train the embodied intelligent model; wherein the embodied intelligent model is obtained by training based on each embodied intelligent terminal in the at least one embodied intelligent terminal;

[0037] The embodied intelligent terminal is used to call the embodied intelligent model stored in the embodied intelligent server in response to user instructions to implement the function corresponding to the user instructions.

[0038] In a possible implementation, the embodied intelligent terminal is used to call the embodied intelligent model stored in the embodied intelligent server in response to a user instruction to implement the function corresponding to the user instruction, including:

[0039] The embodied intelligent terminal is used to send the user instruction to the embodied intelligent server, so that the embodied intelligent server generates and sends an information collection instruction to the embodied intelligent terminal based on the stored embodied intelligent model;

[0040] The embodied intelligent terminal is used to obtain and send corresponding terminal information to the embodied intelligent server in response to the information collection instruction, so that the embodied intelligent server generates and sends corresponding execution instructions to the embodied intelligent terminal based on the stored embodied intelligent model;

[0041] The embodied intelligent terminal is used to respond to the execution instruction and implement the function corresponding to the execution instruction.

[0042] In a possible implementation, the embodied intelligent terminal includes a terminal communication module and a sensor module; the terminal communication module is connected to the sensor module and the embodied intelligent server respectively;

[0043] Wherein, the information collection instruction includes a main body information collection instruction and an external body information collection instruction; the terminal information includes main body information and external body information;

[0044] The embodied intelligent terminal is used to respond to the information collection instruction, obtain and send corresponding terminal information to the embodied intelligent server, so that the embodied intelligent server generates and sends corresponding execution instructions to the embodied intelligent terminal based on the stored embodied intelligent model, including:

[0045] The terminal communication module is used to respond to the received body information collection instruction, obtain and send the body information to the embodied intelligent server; and send the received external body information collection instruction to the sensor module;

[0046] The sensing module is used to respond to the external body information collection instruction received through the terminal communication module, obtain and send the external body information to the embodied intelligent server through the terminal communication module;

[0047] The embodied intelligent server is used to generate and send corresponding execution instructions to the embodied intelligent terminal according to the body information and the external body information and based on the stored embodied intelligent model.

[0048] In a possible implementation, the sensing module includes at least one sensing unit; the external body information collection instruction includes at least one external body sub-information collection instruction; the external body information includes at least one external body sub-information; wherein the sensing unit corresponds to the external body sub-information collection instruction and the external body sub-information in a one-to-one manner;

[0049] The sensing module is used to respond to the external body information collection instruction received through the terminal communication module, obtain the external body information and send it to the embodied intelligent server through the terminal communication module, including:

[0050] The sensing unit is used to respond to the corresponding external body sub-information collection instruction received through the terminal communication module, obtain and send the corresponding external body sub-information to the embodied intelligent server through the terminal communication module.

[0051] In a possible implementation, the embodied intelligent terminal further includes an execution module; the execution module is connected to the terminal communication module;

[0052] The embodied intelligent terminal is used to respond to the execution instruction and implement the function corresponding to the execution instruction, including:

[0053] The terminal communication module is used to send the received execution instruction to the execution module;

[0054] The execution module is used to respond to the execution instruction and implement the function corresponding to the execution instruction.

[0055] In a possible implementation, the execution module includes at least one execution unit; the execution instruction includes at least one sub-execution instruction; wherein the execution unit corresponds to the sub-execution instruction one by one; the external body information collection instruction includes at least one external body sub-information collection instruction;

[0056] The terminal communication module is used to send the received execution instruction to the execution module, including:

[0057] The terminal communication module is used to send the received sub-execution instruction to the corresponding execution unit;

[0058] The execution module is used to implement the function corresponding to the execution instruction in response to the execution instruction, including:

[0059] The execution unit is used to implement the function corresponding to the corresponding sub-execution instruction in response to the corresponding sub-execution instruction.

[0060] In a possible implementation, the embodied intelligent server includes a functional module;

[0061] The embodied intelligence server is used to store and train the embodied intelligence model, including:

[0062] The functional module is used to store and train the embodied intelligence model.

[0063] In a possible implementation, the embodied intelligent server further includes a server communication module; the server communication module is connected to the embodied intelligent terminal and the functional module respectively;

[0064] The embodied intelligent terminal is used to send the user instruction to the embodied intelligent server, so that the embodied intelligent server generates and sends the information collection instruction to the embodied intelligent terminal based on the stored embodied intelligent model, including:

[0065] The embodied intelligent terminal is used to send the user instruction to the server communication module;

[0066] The server communication module is used to send the received user instruction to the functional module;

[0067] The functional module is used to determine the corresponding embodied intelligent model in response to the received user instruction, and generate the information collection instruction based on the corresponding embodied intelligent module; and send it to the embodied intelligent terminal through the server communication module.

[0068] In a possible implementation, the embodied intelligent terminal is used to respond to the information collection instruction, obtain and send corresponding terminal information to the embodied intelligent server, so that the embodied intelligent server generates and sends corresponding execution instructions to the embodied intelligent terminal based on the stored embodied intelligent model, including:

[0069] The embodied intelligent terminal is used to respond to the information collection instruction, obtain and send corresponding terminal information to the server communication module;

[0070] The server communication module is used to send the received terminal information to the functional module;

[0071] The functional module is used to determine the corresponding embodied intelligent model in response to the received terminal information, and generate the execution instruction based on the corresponding embodied intelligent module; and send it to the embodied intelligent terminal through the server communication module.

[0072] In a possible implementation, the embodied intelligent server further includes a storage module; the storage module is connected to the functional module and the communication module respectively;

[0073] The storage module is used to store the terminal data transmitted by the server communication module; wherein the terminal data is the data of the embodied intelligent terminal during operation.

[0074] In a possible implementation, the storage module includes at least one first storage unit; the terminal data includes at least one terminal sub-data; wherein the terminal sub-data is the terminal data corresponding to the embodied intelligent terminal of each embodied intelligent scene; wherein the first storage unit has a one-to-one correspondence with the terminal sub-data and the embodied intelligent scene;

[0075] The storage module is used to store the terminal data transmitted by the server communication module, including:

[0076] The first storage unit is used to store the corresponding terminal sub-data transmitted by the server communication module.

[0077] In a possible implementation, the storage module is also used to store embodied intelligence knowledge data; wherein the embodied intelligence knowledge data includes at least one embodied intelligence knowledge sub-data; the embodied intelligence knowledge sub-data is industrial knowledge data or digital model data; wherein the industrial knowledge data is relevant knowledge of the industrial field corresponding to all embodied intelligence scenarios; the digital model data is the industrial field corresponding to all embodied intelligence scenarios, and relevant models of the basic industrial field.

[0078] In a possible implementation, the storage module further includes at least one second storage unit; wherein the second storage unit corresponds to the embodied intelligent knowledge sub-data one by one;

[0079] The second storage unit is used to store the embodied intelligent knowledge sub-data.

[0080] In a possible implementation, the storage module is also used to store the corresponding historical terminal data transmitted by the server communication module; wherein the historical terminal data is the data of the embodied intelligent terminal during the historical operation process.

[0081] In a possible implementation, the process of training the embodied intelligence model includes:

[0082] The functional module is used to obtain the terminal data, the embodied intelligence knowledge data, and the historical terminal data from the storage module in real time, and to train the stored embodied intelligence model in real time based on the acquired terminal data, the embodied intelligence knowledge data, and the historical terminal data.

[0083] The training method of the embodied intelligence model of motion planning provided in the embodiment of the present application obtains an embodied intelligence initial model, obtains a training set of the model, and trains the embodied intelligence initial model according to the training set of the model to obtain an embodied intelligence model, wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in-vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer. Adding the ontology cognition layer to the embodied intelligence initial model can improve the ontology cognition of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model. In the training process of the embodied intelligence model, the alignment function of the fusion layer is combined to improve the accuracy of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0085] Figure 1 Schematic diagram of the training method of the embodied intelligent model for motion planning provided in this application Figure 1 ;

[0086] Figure 2 Schematic diagram of the training method of the embodied intelligent model for motion planning provided in this application Figure 2 ;

[0087] Figure 3 A flowchart of a method for motion planning based on an embodied intelligence model provided in this application;

[0088] Figure 4 A schematic diagram of the structure of a training device for an embodied intelligent model for motion planning provided in this application;

[0089] Figure 5 A schematic diagram of the structure of a device for model-based motion planning provided in this application;

[0090] Figure 6 A schematic diagram of the structure of the electronic device provided for this application;

[0091] Figure 7 Schematic diagram of the structure of the embodied intelligent system provided for this application Figure 1 ;

[0092] Figure 8 Schematic diagram of the structure of the embodied intelligent system provided for this application Figure 2 ;

[0093] Fig. 9 A schematic diagram of the structure of the embodied intelligent terminal provided for this application;

[0094] Fig.10 A schematic diagram of the structure of the embodied intelligent server provided for this application;

[0095] Among them, 100-embodied intelligent system; 1-embodied intelligent server; 2-embodied intelligent terminal; 3-terminal communication module; 4-sensing module; 5-execution module; 6-functional module; 7-server communication module; 8-storage module; 9-sensing unit; 10-execution unit; 11-first storage unit; 12-second storage unit; 13-functional unit.

[0096] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0097] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0098] In the prior art, the embodied intelligence model mainly completes the corresponding motion planning based on natural language instructions and environmental perception information. However, for different embodied intelligent terminals, using the embodied intelligence model described in the prior art will lead to the problem of inaccurate motion planning of the embodied intelligence.

[0099] The present application provides a method for training an embodied intelligence model for motion planning, which obtains an embodied intelligence initial model and a training set of the model, and trains the embodied intelligence initial model according to the training set of the model to obtain an embodied intelligence model, wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in-vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer. Adding an ontology cognition layer to the embodied intelligence initial model can improve the ontology cognition of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model. In the training process of the embodied intelligence model, the alignment function of the fusion layer is combined to improve the accuracy of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model.

[0100] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0101] Figure 1 Schematic diagram of the training method of the embodied intelligent model for motion planning provided in this application Figure 1 , the execution subject is the embodied intelligent server, which can generally be implemented through software, hardware, or a combination of software and hardware. Figure 1 As shown, the method includes:

[0102] Step S101, obtaining an initial model of embodied intelligence.

[0103] Specifically, an embodied intelligence initial model may be obtained, wherein the present application does not limit the process of obtaining the embodied intelligence initial model, and optionally, a pre-stored embodied intelligence initial model may be extracted from an embodied intelligence server.

[0104] Among them, the embodied intelligence initial model is the initial model used to train the embodied intelligence model, wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in-vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer.

[0105] Specifically, the initial natural language understanding layer, the initial in-vitro perception layer, the initial ontology cognition layer, the initial motion planning layer, and the fusion layer are network layers of the embodied intelligence initial model, wherein the initial natural language understanding layer is the network layer that understands user instructions during the model training process; the initial in-vitro perception layer is the network layer that perceives the environment where the embodied intelligent terminal is located during the model training process; the initial ontology cognition layer is the network layer that recognizes the embodied intelligent terminal during the model training process; the initial motion planning layer is the network layer that plans the motion of the embodied intelligent terminal during the model training process; the fusion layer is the network layer that fuses and aligns the above network layers, i.e., the initial natural language understanding layer, the initial in-vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer, during the model training process.

[0106] Among them, the embodied intelligent terminal is a terminal connected to the embodied intelligent server and used to execute the movement planned by the embodied intelligent terminal.

[0107] Step S102: Obtain a training set for the model.

[0108] Specifically, a training set of the model may be obtained. The present application does not limit the process of obtaining the training set of the model. Optionally, a pre-stored training set of the model may be extracted from the embodied intelligent server, or a pre-stored training set of the model may be extracted from other servers, or a training set of the model input by a user may be received.

[0109] The training set of the model includes at least one training sample. The present application does not limit the number of training samples included in the model training set. Optionally, the training samples included in the model training set correspond to the type of the embodied intelligent service terminal to which the embodied intelligent server is connected. The model training set may include at least one training sample corresponding to the type of each embodied intelligent service terminal.

[0110] The training samples include natural language subsamples, in vitro perception subsamples, ontology cognition subsamples, and motion planning subsamples. The natural language subsamples, in vitro perception subsamples, ontology cognition subsamples, and motion planning subsamples in each training sample correspond to each other. That is, each training sample is a sample generated based on an instance of motion planning for an embodied intelligent terminal at a certain location according to a user instruction. The natural language subsamples, in vitro perception subsamples, ontology cognition subsamples, and motion planning subsamples in each training sample correspond to the same instance. The subsamples included in the training sample respectively represent the input and / or output of the network layer of the embodied intelligence initial model.

[0111] Step S103: train the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model.

[0112] Specifically, according to the training set of the model obtained in step S102, the embodied intelligence initial model obtained in step S101 is trained to obtain an embodied intelligence model. The embodied intelligence model is a model for motion planning of the embodied intelligent terminal. The present application does not limit the embodied intelligence model, and any model for motion planning of the embodied intelligent terminal obtained by training the embodied intelligence initial model according to the training set of the model can be used as the embodied intelligence model provided by the present application.

[0113] Among them, the present application does not limit the process of training the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model. Optionally, a training sample can be extracted from the training set of the model, and based on the extracted training sample and the fusion layer in the embodied intelligence initial model, the initial natural language understanding layer, the initial in vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model are trained to obtain the trained embodied intelligence initial model. Among them, if each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions, the embodied intelligence model is obtained. Otherwise, the above training process is repeated until each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions.

[0114] The training method of the embodied intelligence model for motion planning provided in the embodiment of the present application obtains an embodied intelligence initial model, obtains a training set of the model, and trains the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model, wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in-vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer. Adding the ontology cognition layer to the embodied intelligence initial model can improve the ontology cognition of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model. In the training process of the embodied intelligence model, the alignment function of the fusion layer is combined to improve the accuracy of the trained embodied intelligence model, thereby improving the accuracy of motion control based on the trained embodied intelligence model.

[0115] In a possible embodiment, the process of training the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model may include:

[0116] First, a training sample is extracted from the training set of the model, and based on the extracted training sample and the fusion layer in the embodied intelligence initial model, the initial natural language understanding layer, initial in-vitro perception layer, initial ontology cognition layer, and initial motion planning layer in the embodied intelligence initial model are trained to obtain the trained embodied intelligence initial model.

[0117] For the description of the model training set and training samples, please refer to the description in step S102, which will not be repeated here. The present application does not limit the process of extracting a training sample from the model training set. Optionally, a training sample can be randomly extracted from the model training set.

[0118] Among them, the present application does not limit the process of training the initial natural language understanding layer, the initial in-vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model according to the extracted training samples and the fusion layer in the embodied intelligence initial model to obtain the trained embodied intelligence initial model. Optionally, the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value can be obtained according to the extracted training samples and the embodied intelligence initial model; wherein the first loss value is the loss value corresponding to the initial natural language understanding layer; the second loss value is the loss value corresponding to the initial in-vitro perception layer. The third loss value is the loss value corresponding to the initial ontology recognition layer; the fourth loss value is the loss value corresponding to the initial motion planning layer; the fifth loss value is the loss value obtained according to the fusion layer; then according to the first loss value and the fifth loss value, the parameters of the initial natural language understanding layer are adjusted, according to the second loss value and the fifth loss value, the parameters of the initial in-vitro perception layer are adjusted, according to the third loss value and the fifth loss value, the parameters of the initial ontology recognition layer are adjusted, according to the fourth loss value and the fifth loss value, the parameters of the initial motion planning layer are adjusted to obtain the trained initial model of embodied intelligence.

[0119] Secondly, if each trained layer in the trained embodied intelligence initial model meets the preset convergence condition, the embodied intelligence model is obtained. Otherwise, the above training process is repeated until each trained layer in the trained embodied intelligence initial model meets the preset convergence condition.

[0120] Among them, the present application does not limit the preset convergence conditions, wherein the preset convergence conditions correspond to the specific process of training the initial natural language understanding layer, initial in-vitro perception layer, initial ontology cognition layer, and initial motion planning layer in the embodied intelligence initial model according to the extracted training samples and the fusion layer in the embodied intelligence initial model. Optionally, if the specific process of training the initial natural language understanding layer, initial in-vitro perception layer, initial ontology cognition layer, and initial motion planning layer in the embodied intelligence initial model according to the extracted training samples and the fusion layer in the embodied intelligence initial model is as shown above, then four loss value thresholds can be set, corresponding to the first loss value and the fifth loss value, the second loss value and the fifth loss value, the third loss value and the fifth loss value, and the fourth loss value and the fifth loss value described above, respectively. Among them, if each loss value obtained according to the trained embodied intelligence initial model meets the preset loss value threshold, it is determined that each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions.

[0121] Among them, in the process of training the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model, the network layers in the embodied intelligence initial model are trained according to the extracted training samples and the fusion layer in the embodied intelligence initial model, so that the network layers in the trained embodied intelligence model can be aligned, thereby improving the accuracy of motion control based on the trained embodied intelligence model.

[0122] In a possible embodiment, the training method of the embodied intelligent model for motion planning also includes: if the types of embodied intelligent terminals increase, the training set of the model is updated, and the embodied intelligent model is updated and trained based on the updated model training set.

[0123] Among them, the description of the model training set can refer to the description in step S102, which will not be repeated here.

[0124] Specifically, if the types of embodied intelligent terminals increase, that is, the types of embodied intelligent terminals connected to the embodied intelligent server increase, the training set of the model can be updated, wherein the present application does not limit the process of updating the training set of the model. Optionally, according to the types of the newly added embodied intelligent terminals, pre-stored training samples corresponding to the newly added embodied intelligent terminals can be extracted from the embodied intelligent server to update the training set of the model; optionally, according to the types of the newly added embodied intelligent terminals, pre-stored training samples corresponding to the newly added embodied intelligent terminals can be extracted from other servers to update the training set of the model; optionally, the training samples corresponding to the newly added embodied intelligent terminals input by the user can also be received to update the training set of the model. For example, according to the types of the newly added embodied intelligent terminals, corresponding training sample input prompts can be generated and presented, so that the user can input the training samples corresponding to the newly added embodied intelligent terminals according to the presented training sample input prompts to update the training set of the model.

[0125] Specifically, after the training set of the model is updated, the embodied intelligence model can be updated and trained according to the updated model training set. The present application does not limit the process of updating and training the embodied intelligence model according to the updated model training set. Optionally, the description of the process of training the embodied intelligence initial model according to the model training set to obtain the embodied intelligence model can be referred to in step S103, which will not be repeated here.

[0126] Among them, if the types of embodied intelligent terminals increase, the accuracy of the training set can be improved by updating the training set of the model, and the accuracy of the trained embodied intelligent model can be further improved, thereby improving the accuracy of motion control based on the trained embodied intelligent model.

[0127] Figure 2 Schematic diagram of the training method of the embodied intelligent model for motion planning provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, the process of training the initial natural language understanding layer, the initial in vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model according to the extracted training samples and the fusion layer in the embodied intelligence initial model to obtain the trained embodied intelligence initial model is described in detail. The method includes:

[0128] Step S201: According to the extracted training samples and the embodied intelligence initial model, a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value are obtained.

[0129] Specifically, according to the extracted training samples and the embodied intelligence initial model, a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value can be obtained. Among them, the first loss value is the loss value corresponding to the initial natural language understanding layer, which is used to train the initial natural language understanding layer. The second loss value is the loss value corresponding to the initial in vitro perception layer, which is used to train the initial in vitro perception layer. The third loss value is the loss value corresponding to the initial ontology cognition layer, which is used to train the initial ontology cognition layer. The fourth loss value is the loss value corresponding to the initial motion planning layer, which is used to train the initial motion planning layer. The fifth loss value is the loss value obtained according to the fusion layer, which is used to train the initial natural language understanding layer, the initial in vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer.

[0130] Among them, the description of the natural language sub-sample, in-vitro perception sub-sample, ontology cognition sub-sample, and motion planning sub-sample included in the training sample can refer to the description in step S102. The present application does not limit the natural language sub-sample, in-vitro perception sub-sample, ontology cognition sub-sample, and motion planning sub-sample included in the training sample. Any sub-sample corresponding to the same instance that can represent the input and / or output of the network layer of the initial model of embodied intelligence can be used as the natural language sub-sample, in-vitro perception sub-sample, ontology cognition sub-sample, and motion planning sub-sample included in the training sample provided by the present application.

[0131] Optionally, the natural language subsample includes user instructions, and embodied intelligent action sequences corresponding to the user instructions. Specifically, the user instructions are instructions for instructing the movement of the embodied intelligent terminal, wherein the present application does not display the form of the user instructions, and optionally, may include but is not limited to at least one of the forms such as text form and voice form. Specifically, the embodied intelligent action sequence is an action sequence executable by the embodied intelligent terminal, wherein the embodied intelligent action sequence includes at least one executable action, wherein the present application does not limit the number of executable actions included in the embodied intelligent action sequence. For example, if the user instruction is "put the apple on the table on the chair", the corresponding embodied intelligent action sequence may be "grab, move, place".

[0132] Optionally, the in vitro perception subsample includes environmental information, and the item category and item posture corresponding to the environmental information. Specifically, the environmental information is the information of the environment in which the embodied intelligent terminal is located, wherein the present application does not limit the type of environmental information, and optionally, it can be the visual information of the environment in which the embodied intelligent terminal is located, wherein the visual information generally refers to the external environment data perceived and acquired by the robot or intelligent body through its visual sensor (such as a camera, a depth camera, etc.). Among them, the present application does not limit the visual information, and optionally, the visual information includes but is not limited to at least one of the images such as RGB images and depth images. Specifically, the item category is the type of item in the environment in which the embodied intelligent terminal is located, wherein the present application does not limit the item category, and any type of item in the environment in which the embodied intelligent terminal is located can be used as the item category provided by the present application. For example, if the environment in which the embodied intelligent terminal is located includes tables, chairs, and apples, the item type may include furniture type and fruit type. Specifically, the object posture is the position and orientation of the object in the three-dimensional space in the environment where the embodied intelligent terminal is located. The present application does not limit the object posture. Optionally, it can be a six-degree-of-freedom posture (Six Degrees of Freedom, referred to as 6-DoF), which refers to the six degrees of freedom of an object in three-dimensional space, including three translational degrees of freedom and three rotational degrees of freedom. The translational degree of freedom includes movement along the x, y, and z axes, while the rotational degree of freedom includes rotation around the x, y, and z axes. Optionally, it can also be a seven-degree-of-freedom posture 7-DoF, which adds an additional degree of freedom on the basis of 6-DoF, and is usually used for more complex motion control, such as considering the tilt or roll of the object. Optionally, it can also be a three-degree-of-freedom posture 3-DoF, which only includes three translational or rotational degrees of freedom, and is suitable for simple motion control.

[0133] Optionally, the ontology recognition subsample includes ontology parameter information and the terminal target posture corresponding to the ontology parameter information. Specifically, the ontology parameter information is the parameter information of the embodied intelligent terminal, wherein the present application does not limit the ontology parameter information, optionally, it may include ontology state parameters and ontology basic parameters, optionally, ontology state parameters include but are not limited to: angle parameters, speed parameters, acceleration parameters, torque parameters and other parameters, optionally, ontology basic parameters, that is, ontology static parameters include but are not limited to ontology structure parameters, ontology size parameters and other parameters, wherein the present application does not limit the process of obtaining ontology state parameters, optionally, it can be obtained according to the sensors set in the embodied intelligent terminal, wherein the present application does not limit the type of sensors set in the embodied intelligent terminal, optionally, it may include but is not limited to joint sensors, torque sensors, space posture sensors and other sensors. Among them, the terminal target posture is the target posture of the embodied intelligent terminal at the end point, wherein the end point is the point corresponding to the end point of the movement of the embodied intelligent terminal, wherein this application does not limit the end point of the movement of the embodied intelligent terminal, optionally, if the user instruction is "put the apple on the table on the chair", then the corresponding end point of the movement may include the position on the table where the apple is obtained and the position on the chair where the apple will be placed, wherein the target posture is the posture of each joint of the embodied intelligent terminal at the end point, wherein the description of the posture can refer to the above description and will not be repeated here. Among them, if the type of the embodied intelligent terminal is different, for example, the model of the robotic arm is different, the corresponding body parameter information and the terminal target posture are also different.

[0134] Optionally, the motion planning subsample includes a joint angle planning sequence. Specifically, the joint angle planning sequence is a sequence of joint angles of the embodied intelligent terminal changing over time. If the type of the embodied intelligent terminal is different, for example, the model of the robotic arm is different, the corresponding joint angle planning sequence is also different.

[0135] Among them, the present application does not limit the process of obtaining the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value according to the extracted training sample and the embodied intelligence initial model. Optionally, if the natural language sub-sample, the in vitro perception sub-sample, the ontology cognition sub-sample, and the motion planning sub-sample included in the training sample are as described above, then according to the extracted training sample and the embodied intelligence initial model, the process of obtaining the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value may include:

[0136] First, based on the user instructions, environmental information, ontology parameter information, and embodied intelligence initial model included in the extracted training samples, the predicted embodied intelligence action sequence, predicted object category, predicted object posture, predicted terminal target posture, predicted joint angle planning sequence, and instruction feature vectors, environmental feature vectors, ontology feature vectors, and posture feature vectors can be obtained.

[0137] Among them, the prediction of embodied intelligence action sequence is the predicted value of the embodied intelligence action sequence obtained according to the user instructions and the initial natural language understanding layer in the embodied intelligence initial model; the prediction of object category and the prediction of object posture are the predicted values ​​of object category and object posture obtained according to the environmental information and the initial in-vitro perception layer in the embodied intelligence initial model; the prediction of terminal target posture is the predicted value of terminal target posture obtained according to the ontology parameter information and the initial ontology cognition layer in the embodied intelligence initial model; the prediction of joint angle planning sequence is the predicted value of joint angle planning sequence obtained according to the predicted value of terminal target posture and the initial motion planning layer in the embodied intelligence initial model.

[0138] Among them, the instruction feature vector , is the feature vector obtained according to the user instruction; the environmental feature vector , is the feature vector obtained based on the environmental information; the ontology feature vector , is the feature vector obtained based on the ontology parameter information; the pose feature vector , is the feature vector obtained based on the predicted terminal target pose.

[0139] Among them, the present application does not limit the process of obtaining the predicted embodied intelligent action sequence, predicted item category, predicted item posture, predicted terminal target posture, predicted joint angle planning sequence, and instruction feature vector, environment feature vector, ontology feature vector, and posture feature vector according to the user instructions, environment information, ontology parameter information, and embodied intelligence initial model included in the extracted training samples. Optionally, the instruction feature vector of the user instruction and the predicted embodied intelligent action sequence corresponding to the user instruction can be determined according to the user instruction and the initial natural language understanding layer. According to the environment information and the initial in-vitro perception layer, the environment feature vector of the environment information, the predicted item type and predicted item posture corresponding to the environment information are determined. According to the ontology parameter information, the embodied intelligent action sequence, the item type, the item posture, and the initial ontology cognitive layer, the ontology feature vector of the ontology parameter information and the predicted terminal target posture corresponding to the ontology parameter information are determined. According to the predicted terminal target posture, user instructions, environment information, and the initial motion planning layer, the posture feature vector of the predicted terminal target posture and the predicted joint angle planning sequence of the embodied intelligent terminal are determined.

[0140] Secondly, the first loss value can be determined according to the predicted embodied intelligent action sequence and the embodied intelligent action sequence. The present application does not limit the first loss value determined according to the predicted embodied intelligent action sequence and the embodied intelligent action sequence. Optionally, the first loss value can be a loss determined by a cross entropy algorithm, for example, a loss value calculated according to the category of each predicted action in the predicted embodied intelligent action sequence and the category of each action in the embodied intelligent action sequence, wherein the calculation formula of the first loss value is as follows:

[0141]

[0142] in, is the first loss value, is the category of the i-th predicted action, is the category of the i-th action.

[0143] The second loss value may be determined based on the predicted item category, the predicted item posture, and the item category and item posture. The present application does not limit the second loss value determined based on the predicted item category, the predicted item posture, and the item category and item posture. Optionally, the second loss value may include an item category loss value and an item position loss value, wherein the calculation formulas for the item category loss value and the item posture loss value are as follows:

[0144]

[0145]

[0146] in, is the item category loss value, is the object posture loss value, is the i-th predicted item category, is the i-th item category, is the i-th predicted object pose, is the i-th object pose, is the dimension of the object pose.

[0147] The third loss value may be determined based on the predicted terminal target posture and the terminal target posture. The present application does not limit the third loss value determined based on the predicted terminal target posture and the terminal target posture, wherein the calculation formula of the third loss value is as follows:

[0148]

[0149] in, is the third loss value, To predict the terminal target pose, is the terminal target pose, and i is the dimension of the terminal target pose.

[0150] The fourth loss value can be determined based on the predicted joint angle planning sequence and the joint angle planning sequence. Among them, the present application does not limit the fourth loss value determined based on the predicted joint angle planning sequence and the joint angle planning sequence. Optionally, the fourth loss value may include: planning consistency loss value, collision cost loss value, joint angle mutation loss value and other loss values. Among them, the present application does not limit the collision cost loss value. Optionally, the collision cost can be measured by detecting the collision point in the planned trajectory and using the collision avoidance loss. Among them, the present application does not limit the joint angle mutation loss value. Optionally, the smoothness loss can be used to ensure the smoothness of the trajectory and reduce the mutation of the joint angle. Among them, the calculation formulas for the planning consistency loss value, the collision cost loss value, and the joint angle mutation loss value are as follows:

[0151]

[0152]

[0153]

[0154] in, is the planning consistency loss value, is the collision cost loss value, is the joint angle mutation loss value, is the predicted joint angle planning at the i-th moment, is the joint angle planning at the i-th moment, is the joint angle planning at the i-1th moment.

[0155] Then, the fifth loss value is determined according to the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector.

[0156] Among them, the present application does not limit the process of determining the fifth loss value according to the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector. Optionally, the process of determining the fifth loss value according to the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector may include: first inputting the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector into the fusion layer, and outputting the fused feature vector. Then, the fifth loss value is determined according to the fused feature vector.

[0157] The present application does not limit the fifth loss value determined, and may optionally be the alignment loss of each feature vector, wherein the calculation formula of the fifth loss value is as follows:

[0158]

[0159] in, is the fifth loss value, is the fused feature vector, that is, the fusion layer instruction feature vector , environmental feature vector , ontology feature vector , pose feature vector , the multimodal feature vector is obtained after fusion processing.

[0160] Step S202: According to the first loss value and the fifth loss value, adjust the parameters of the initial natural language understanding layer; according to the second loss value and the fifth loss value, adjust the parameters of the initial in-vitro perception layer; according to the third loss value and the fifth loss value, adjust the parameters of the initial ontology cognition layer; according to the fourth loss value and the fifth loss value, adjust the parameters of the initial motion planning layer to obtain the trained embodied intelligence initial model.

[0161] Specifically, after obtaining the first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value based on the extracted training samples and the embodied intelligence initial model, the parameters of the initial natural language understanding layer can be adjusted according to the first loss value and the fifth loss value, the parameters of the initial in-vitro perception layer can be adjusted according to the second loss value and the fifth loss value, the parameters of the initial ontology cognition layer can be adjusted according to the third loss value and the fifth loss value, and the parameters of the initial motion planning layer can be adjusted according to the fourth loss value and the fifth loss value to obtain the trained embodied intelligence initial model.

[0162] Among them, the present application does not limit the process of adjusting the parameters of the initial natural language understanding layer according to the first loss value and the fifth loss value. Optionally, the first comprehensive loss value can be first determined based on the first loss value and the fifth loss value and a preset weight ratio, and then the global parameters of the initial natural language understanding layer can be adjusted based on the first comprehensive loss value, so that the first comprehensive loss value of the initial natural language understanding layer after training is more in line with the requirements of the corresponding loss threshold among the four loss value thresholds described in the above embodiments.

[0163] Among them, the present application does not limit the process of adjusting the parameters of the initial in vitro perception layer according to the second loss value and the fifth loss value. Optionally, the second comprehensive loss value can be first determined based on the second loss value and the fifth loss value and a preset weight ratio, and then the global parameters of the initial in vitro perception layer can be adjusted based on the second comprehensive loss value, so that the second comprehensive loss value of the trained initial in vitro perception layer is more in line with the requirements of the corresponding loss threshold among the four loss value thresholds described in the above embodiments.

[0164] Among them, the present application does not limit the process of adjusting the parameters of the initial ontology cognitive layer according to the third loss value and the fifth loss value. Optionally, the third comprehensive loss value can be first determined based on the third loss value and the fifth loss value and a preset weight ratio, and then the global parameters of the initial ontology cognitive layer can be adjusted based on the third comprehensive loss value, so that the third comprehensive loss value of the trained initial ontology cognitive layer is more in line with the requirements of the corresponding loss threshold among the four loss value thresholds described in the above embodiments.

[0165] Among them, the present application does not limit the process of adjusting the parameters of the initial motion planning layer according to the fourth loss value and the fifth loss value. Optionally, the fourth comprehensive loss value can be first determined according to the fourth loss value and the fifth loss value and a preset weight ratio, and then the global parameters of the initial motion planning layer can be adjusted according to the fourth comprehensive loss value, so that the fourth comprehensive loss value of the initial motion planning layer after training is more in line with the requirements of the corresponding loss threshold among the four loss value thresholds described in the above embodiments.

[0166] Specifically, based on the parameter adjustment process described above, a trained embodied intelligence initial model can be obtained.

[0167] The embodiment of the present application provides a process for training the initial natural language understanding layer, the initial external perception layer, the initial ontology cognition layer, and the initial motion planning layer in the embodied intelligence initial model according to the extracted training samples and the fusion layer in the embodied intelligence initial model to obtain the trained embodied intelligence initial model. The first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value are obtained according to the extracted training samples and the embodied intelligence initial model. The parameters of the initial natural language understanding layer are adjusted according to the first loss value and the fifth loss value, the parameters of the initial external perception layer are adjusted according to the second loss value and the fifth loss value, the parameters of the initial ontology cognition layer are adjusted according to the third loss value and the fifth loss value, and the parameters of the initial motion planning layer are adjusted according to the fourth loss value and the fifth loss value to obtain the trained embodied intelligence initial model. In the process of model training, when adjusting the parameters of each network layer, in addition to the loss value corresponding to the network layer, the parameters of each network layer are also adjusted in combination with the loss value corresponding to the fusion layer, so as to improve the accuracy of the trained embodied intelligence model and further improve the accuracy of motion control based on the trained embodied intelligence model.

[0168] Optionally, if each of the first comprehensive loss value of the trained initial natural language understanding layer, the second comprehensive loss value of the trained initial in-vitro perception layer, the third comprehensive loss value of the trained initial ontology cognition layer, and the fourth comprehensive loss value of the trained initial motion planning layer meets the requirements of the corresponding loss threshold, then the training is determined to be completed, that is, the trained embodied intelligence initial model is an embodied intelligence model.

[0169] Optionally, if at least one of the first comprehensive loss value of the trained initial natural language understanding layer, the second comprehensive loss value of the trained initial in vitro perception layer, the third comprehensive loss value of the trained initial ontology cognition layer, and the fourth comprehensive loss value of the trained initial motion planning layer meets the requirements of the corresponding loss threshold, it is determined that the training is incomplete.

[0170] Optionally, if it is determined that the training is not completed, in the process of repeating the above training process until each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions, in the process of adjusting the parameters of the corresponding network layer according to the comprehensive loss value and loss value threshold calculated again, the parameter adjustment method using local parameter adjustment is selected. Among them, this application does not limit the network layer parameters corresponding to the global parameter adjustment and local parameter adjustment in each network layer.

[0171] In a possible embodiment, according to the user instructions, environment information, entity parameter information, and embodied intelligence initial model included in the extracted training samples, the process of obtaining the predicted embodied intelligence action sequence, the predicted object category, the predicted object posture, the predicted terminal target posture, the predicted joint angle planning sequence, and the instruction feature vector, the environment feature vector, the entity feature vector, and the posture feature vector may include:

[0172] Firstly, according to the user instructions and the initial natural language understanding layer, the instruction feature vector of the user instruction and the predicted embodied intelligent action sequence corresponding to the user instruction are determined.

[0173] Among them, this application does not limit the process of determining the instruction feature vector of the user instruction and the process of predicting the embodied intelligent action sequence corresponding to the user instruction according to the user instruction and the initial natural language understanding layer. Optionally, the pre-trained Llama3.2 model can be used as the user instruction encoder to obtain the instruction feature vector. , optionally, a multi-layer Transformer decoder can be used, the instruction feature vector , generating a predicted embodied intelligent action sequence corresponding to the user's instructions.

[0174] Secondly, according to the environmental information and the initial in vitro perception layer, the environmental feature vector of the environmental information, the predicted object type and the predicted object posture corresponding to the environmental information are determined.

[0175] Among them, this application does not limit the process of determining the environmental feature vector of the environmental information according to the environmental information and the initial external body perception layer, and predicting the object type and the object posture corresponding to the environmental information. Optionally, a pre-trained visual Transformer can be used as an external body data encoder to obtain the environmental feature vector , optionally, a multimodal visual language model and an environmental feature vector , detect objects in the external scene, and generate predicted object types and predicted object poses corresponding to the environmental information.

[0176] Then, according to the ontology parameter information, the embodied intelligent action sequence, the object type, the object posture, and the initial ontology cognitive layer, the ontology feature vector of the ontology parameter information and the predicted terminal target posture corresponding to the ontology parameter information are determined.

[0177] Among them, this application does not limit the process of determining the ontology feature vector of the ontology parameter information and predicting the terminal target posture corresponding to the ontology parameter information according to the ontology parameter information, the embodied intelligent action sequence, the object type, the object posture, and the initial ontology cognitive layer. Optionally, the pre-trained Llama3.2 model can be used as the ontology state signal encoder to obtain the ontology feature vector ,Optionally, multimodal information can be combined, for example, embodied intelligent action sequence, item type, and item pose, to generate the predicted terminal target pose corresponding to the ontological parameter information.

[0178] Finally, according to the predicted terminal target posture, user instructions, environmental information, and the initial motion planning layer, the posture feature vector of the predicted terminal target posture and the predicted joint angle planning sequence of the embodied intelligent terminal are determined.

[0179] Among them, this application does not limit the process of determining the posture feature vector of the predicted terminal target posture and the predicted joint angle planning sequence of the embodied intelligent terminal according to the predicted terminal target posture, user instructions, environmental information, and the initial motion planning layer. Optionally, a multi-layer Transformer network can be used as a target posture encoder to obtain the posture feature vector. Optionally, a three-dimensional scene map in external body perception can be generated according to environmental information as a spatial constraint for motion planning, for example, so that obstacles in the three-dimensional space can be avoided during motion planning, and a motion planning model based on multimodal deep learning, for example, an autoregressive trajectory generation network based on Transformer, predicts the terminal target pose, and generates a predicted joint angle planning sequence for the embodied intelligent terminal using user instructions.

[0180] Among them, the three-dimensional scene mapping in the out-of-body perception is the three-dimensional spatial structure of the environment where the embodied intelligent terminal is located. Among them, this application does not limit the process of obtaining the three-dimensional scene mapping in the out-of-body perception. Optionally, a pre-built intelligent obstacle avoidance neural network model can be used to process the information of the environment where the embodied intelligent terminal is located. For example, the voxel data of the physical environment at different angles around the embodied intelligent terminal is processed to obtain the three-dimensional spatial structure of the environment where the embodied intelligent terminal is located. Optionally, the intelligent obstacle avoidance neural network model is a pre-trained intelligent model for dynamic three-dimensional reconstruction based on voxel data.

[0181] Among them, in the process of obtaining predicted embodied intelligence action sequences, predicted object categories, predicted object postures, predicted terminal target postures, predicted joint angle planning sequences, and instruction feature vectors, environmental feature vectors, ontology feature vectors, and posture feature vectors based on user instructions, environmental information, ontology parameter information, and embodied intelligence initial models included in the extracted training samples, the accuracy of the obtained prediction data can be improved through the combination of sample data and each network layer, thereby improving the accuracy of model training and further improving the accuracy of motion control based on the trained embodied intelligence model.

[0182] Figure 3 This is a flow chart of the motion planning method based on the embodied intelligence model provided in this application. The execution subject is the embodied intelligence server. Generally speaking, it can be implemented by software, hardware, or a combination of software and hardware. Figure 3 As shown, the method includes:

[0183] Step S301: In response to a current user instruction sent by the embodied intelligent terminal, current environment information and current entity parameter information of the embodied intelligent terminal are obtained.

[0184] Specifically, in response to the current user instruction sent by the embodied intelligent terminal, the current environment information and current entity parameter information of the embodied intelligent terminal can be obtained. Among them, the description of the current user instruction can refer to the description of the user instruction in step S201, which will not be repeated here. Among them, the description of the current environment information can refer to the description of the environment information in step S201, which will not be repeated here. Among them, the description of the current entity parameter information can refer to the description of the entity parameter information in step S201, which will not be repeated here.

[0185] Among them, the present application does not limit the process of obtaining the current environment information and current entity parameter information of the embodied intelligent terminal in response to the current user instructions sent by the embodied intelligent terminal. Optionally, in response to the current user instructions sent by the embodied intelligent terminal, an environment information acquisition instruction and an entity parameter information acquisition instruction can be generated and sent to the corresponding embodied intelligent terminal, so that the embodied intelligent terminal responds to the received environment information acquisition instruction and entity parameter information acquisition instruction, obtains and sends the current environment information and current entity parameter information to the embodied intelligent server.

[0186] Step S302: determine and output motion planning instructions based on current user instructions, current environmental information, current ontology parameter information, and the natural language understanding layer, in-vitro perception layer, ontology cognition layer, and motion planning layer included in the embodied intelligent model.

[0187] Specifically, based on the current user instructions, current environmental information, current ontology parameter information obtained in step S301, and the natural language understanding layer, in-vitro perception layer, ontology cognition layer, and motion planning layer included in the embodied intelligent model, motion planning instructions can be determined and output.

[0188] Among them, the embodied intelligent model is a model trained by the training method of the embodied intelligent model for motion planning described in any of the above embodiments.

[0189] Among them, the description of the natural language understanding layer, the in-vitro perception layer, the ontology cognition layer, and the motion planning layer can refer to the description of the initial natural language understanding layer, the initial in-vitro perception layer, the initial ontology cognition layer, and the initial motion planning layer in step S101, which will not be repeated here.

[0190] Among them, the motion planning instructions are instructions for controlling the motion of the embodied intelligent terminal.

[0191] The process of the flow chart of the method for motion planning based on the embodied intelligent model provided in the embodiment of the present application obtains the current environment information and the current entity parameter information of the embodied intelligent terminal in response to the current user instructions sent by the embodied intelligent terminal, and determines and outputs the motion planning instructions according to the current user instructions, the current environment information, the current entity parameter information, and the natural language understanding layer, the in-vitro perception layer, the entity cognition layer, and the motion planning layer included in the embodied intelligent model. The embodied intelligent model used to determine and output the motion planning instructions is a model trained based on the training process described in the above embodiment. Therefore, the motion planning instructions can be improved, thereby improving the accuracy of motion control based on the motion planning instructions.

[0192] In a possible embodiment, the process of determining and outputting motion planning instructions according to the current user instructions, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model may include:

[0193] According to the current user instructions and the natural language understanding layer, the current embodied intelligent action sequence corresponding to the current user instructions is determined; according to the current environmental information and the external perception layer, the current object type and the current object posture corresponding to the current environmental information are determined; according to the current ontological parameter information, the current embodied intelligent action sequence, the current object type, the current object posture, and the ontological cognition layer, the current terminal target posture corresponding to the current ontological parameter information is determined; according to the current terminal target posture, the current user instructions, the current environmental information, and the motion planning layer, the current joint angle planning sequence of the embodied intelligent terminal is determined.

[0194] Among them, the present application describes the process of determining the current embodied intelligent action sequence corresponding to the current user instruction according to the current user instruction and the natural language understanding layer; determining the current object type and the current object posture corresponding to the current environmental information according to the current environmental information and the external perception layer; determining the current terminal target posture corresponding to the current ontology parameter information according to the current ontology parameter information, the current embodied intelligent action sequence, the current object type, the current object posture, and the ontology cognition layer; and determining the current joint angle planning sequence of the embodied intelligent terminal according to the current terminal target posture, the current user instruction, the current environmental information, and the motion planning layer. The above-described embodiments can be referred to for the process of determining the current joint angle planning sequence according to the user instruction and the initial natural language understanding. layer, determine the instruction feature vector of the user instruction, and the predicted embodied intelligent action sequence corresponding to the user instruction; determine the environmental feature vector of the environmental information, and the predicted object type and predicted object posture corresponding to the environmental information according to the environmental information and the initial external perception layer; determine the ontology feature vector of the ontology parameter information, the embodied intelligent action sequence, the object type, the object posture, and the initial ontology cognitive layer, and the predicted terminal target posture corresponding to the ontology parameter information; determine the posture feature vector of the predicted terminal target posture, and the predicted joint angle planning sequence of the embodied intelligent terminal according to the predicted terminal target posture, user instructions, environmental information, and the initial motion planning layer. The description of the process will not be repeated here.

[0195] Specifically, after obtaining the current joint angle planning sequence of the embodied intelligent terminal, the motion planning instruction can be determined and output according to the current joint angle planning sequence of the embodied intelligent terminal to control the embodied intelligent terminal to move according to the motion planning instruction.

[0196] Figure 4 A schematic diagram of the structure of the training device for the embodied intelligent model for motion planning provided in this application, such as Figure 4 As shown, the training device 40 for the embodied intelligent model of motion planning provided in this embodiment includes:

[0197] The first acquisition module 401 is used to acquire an embodied intelligence initial model; wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial external perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer;

[0198] The second acquisition module 402 is used to acquire a training set of the model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language sub-sample, an in-vitro perception sub-sample, an ontology cognition sub-sample, and a motion planning sub-sample; wherein the natural language sub-sample, the in-vitro perception sub-sample, the ontology cognition sub-sample, and the motion planning sub-sample in each training sample correspond to each other;

[0199] The training module 403 is used to train the embodied intelligence initial model according to the training set of the model to obtain the embodied intelligence model.

[0200] The training device for the embodied intelligent model for motion planning provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and this embodiment will not be described in detail here.

[0201] Figure 5 A schematic diagram of the structure of the device for model-based motion planning provided in this application, such as Figure 5 As shown, the model-based motion planning device 50 provided in this embodiment includes:

[0202] The third acquisition module 501 is used to obtain the current environment information and current entity parameter information of the embodied intelligent terminal in response to the current user instruction sent by the embodied intelligent terminal;

[0203] The planning module 502 is used to determine and output the motion planning instruction according to the current user instruction, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model;

[0204] Among them, the embodied intelligent model is a model trained by the training method of the embodied intelligent model for motion planning described in any of the above embodiments.

[0205] The model-based motion planning device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0206] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 6As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0207] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0208] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0209] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0210] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0211] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0212] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0213] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0214] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0215] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0216] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0217] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0218] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0219] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0220] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0221] Sending environment information acquisition instructions and entity information acquisition instructions to the corresponding embodied intelligent terminal;

[0222] receiving the environment information and the entity parameter information sent by the embodied intelligent terminal,

[0223] Figure 7 Schematic diagram of the structure of the embodied intelligent system provided for this application Figure 1 ,like Figure 7 As shown, the system includes: an embodied intelligent server 1 and at least one embodied intelligent terminal 2. Each of the at least one embodied intelligent terminal 2 is connected to the embodied intelligent server 1.

[0224] Specifically, the present application does not limit the number of embodied intelligent terminals 2 included in the embodied intelligent system 100, wherein the number of embodied intelligent terminals 2 included in the embodied intelligent system 100 corresponds to the embodied intelligent scenario applied by the embodied intelligent system 100. Among them, the present application does not limit the embodied intelligent scenario applied by the embodied intelligent system 100, and any embodied intelligent scenario in any industry can be used as the embodied intelligent scenario applied by the embodied intelligent system 100 provided by the present application. For example, one or more of the scenarios such as the research and development design scenario, the production and manufacturing scenario, the test and testing scenario, the business management scenario, and the service operation scenario in the manufacturing industry. Among them, the number of embodied intelligent terminals 2 included in the embodied intelligent system 100 corresponds to the number of embodied intelligent terminals 2 required in the embodied intelligent scenario applied by the embodied intelligent system 100. For example: if the embodied intelligent scenario applied by the embodied intelligent system 100 is a research and development design scenario in the manufacturing industry, and one embodied intelligent terminal 2 is required in the research and development design scenario, then the number of embodied intelligent terminals 2 included in the embodied intelligent system 100 is one; for example: if the embodied intelligent scenario applied by the embodied intelligent system 100 is a research and development design scenario, a manufacturing scenario, a trial and test scenario, a business management scenario, and a service operation scenario, and one embodied intelligent terminal 2 is required in the research and development design scenario, five embodied intelligent terminals 2 are required in the manufacturing scenario, one embodied intelligent terminal 2 is required in the trial and test scenario, three embodied intelligent terminals 2 are required in the business management scenario, and two embodied intelligent terminals 2 are required in the service operation scenario, then the number of embodied intelligent terminals 2 included in the embodied intelligent system 100 is twelve.

[0225] The embodied intelligence server 1 is used to store and train the embodied intelligence model.

[0226] Among them, the embodied intelligent model is an intelligent model that enables the embodied intelligent terminal 2 to realize the function corresponding to the user instruction. Among them, the embodied intelligent model is obtained by training each embodied intelligent terminal 2 in at least one embodied intelligent terminal 2. Among them, this application does not limit the embodied intelligent model, and any intelligent model that is obtained by training each embodied intelligent terminal 2 in at least one embodied intelligent terminal 2, which enables the embodied intelligent terminal 2 to realize the function corresponding to the user instruction, can be used as the embodied intelligent model provided by this application. Optionally, it can be one of the models for realizing perception function, memory function, cognitive function, reasoning function, decision-making function, planning function, control function, evolution function, etc.

[0227] Specifically, the model that realizes the perception function is used to analyze the multimodal information collected in the embodied intelligence scene, such as visual information, language information, pressure information, temperature information and other information; the model that realizes the memory function is used to store and memorize multimodal information related to production, wherein different tasks can be completed in the entire production process by compressing the information identified from the data pattern; the model that realizes the cognitive function is used to integrate and associate the multimodal information of perception and memory, so as to obtain a comprehensive understanding of the production environment, wherein the model that realizes the cognitive function enables the embodied intelligence to better simulate the human cognitive process and provide support in complex and dynamic industrial environments; the model that realizes the reasoning function is used to reason about the cognitive results and deduce the entire production process and each production The logical relationship between the steps ensures that the production process proceeds in sequence; the model that realizes the decision-making function is specifically used to automatically determine the current production stage and make the next production decision accordingly; the model that realizes the planning function is specifically used to further plan and select appropriate methods to promote the production process based on the comprehensive results of reasoning and decision-making, so as to ensure the subsequent smooth control; the model that realizes the control function is specifically used for the embodied intelligence to accurately control the embodied intelligent body through actuators, controllers and communication equipment after planning, so as to ensure that the production activities are carried out according to the predetermined procedures and methods; the model that realizes the evolutionary function is specifically used in the process of executing the task, not only to continuously perceive, reason and make decisions to ensure the smooth progress of the production task, but also to continuously expand its database through the data collected throughout the production process, thereby gradually evolving its learning ability.

[0228] The embodied intelligent terminal 2 is used to respond to user instructions and call the embodied intelligent model stored in the embodied intelligent server 1 to implement the function corresponding to the user instruction.

[0229] The present application does not limit the user instructions, and any instructions that can call the embodied intelligence model stored in the embodied intelligence server 1 to implement the function corresponding to the user instruction can be used as the user instruction provided by the present application. The present application does not limit the form of the user instruction, and the user instruction can be in one or more forms such as natural language form, document form, and sub-instruction set form.

[0230] Among them, the present application does not limit the type of embodied intelligent terminal 2. Any terminal that can respond to user instructions and call the embodied intelligent model stored in the embodied intelligent server 1 to implement the function corresponding to the user instruction can be used as the embodied intelligent terminal 2 provided by the present application. Optionally, it can be a robot used in various embodied intelligent scenarios, for example: a robot used in research and development design scenarios, such as a metamorphic robot, wherein a metamorphic robot is a robot that can change and evolve in body structure, pattern and function, reflecting the autonomous evolution capability of embodied intelligence. They can autonomously change structure, form and pattern according to the environment and evolve in real time. The goal of evolution can be the structure and material of the embodied entity. For example, in the design process of the robot's dexterous hand, the metamorphic robot collects data on hand movements and object grasping, analyzes the optimal grasping strategy, and then adjusts the shape or structure of the hand to improve grasping efficiency and adaptability. For example, for a quadruped robot working in an outdoor environment, the robot's motion efficiency on different terrains is analyzed by monitoring joint load and motion data, and the stiffness and damping of the joints are adjusted; applied to production and manufacturing Robots in scenarios, such as special robots, where in certain special environments, special robots replace workers to complete production tasks autonomously, which can ensure the safety of workers and improve operational efficiency. For example, in tasks with higher risks such as welding, special robots are designed to be able to complete welding tasks autonomously, thereby avoiding workers from being directly exposed to high temperatures and radiation. For example, in the mining industry, special robots can autonomously perform tasks such as underground excavation, transportation and exploration, avoiding miners from working in narrow, dark and dangerous environments. For example, in underwater construction, salvage or marine resource exploration, special robots can perform precise underwater welding, cutting or equipment installation tasks. For example, in the aerospace field, assembly robots can autonomously identify, disassemble or assemble components of satellites or space stations in space; robots used in experimental test scenarios, such as self-test robots, where during the test process, the self-test robot not only collects and monitors its own real-time parameters, but also conducts in-depth analysis of the test environment and adjusts the test conditions to ensure the smooth progress of the test. In addition, the self-test robot can autonomously identify potential problems and analyze the root causes of the problems, thereby optimizing the entire test process.Based on the accumulated knowledge and experience, the self-testing robot provides analysis results and targeted optimization suggestions, which helps to identify potential defects and make self-improvements; robots used in business management scenarios, such as sales robots, where the sales robots autonomously provide customers with intelligent sales services through their multimodal interaction capabilities. The sales robots interact with customers through voice, touch screen or gesture recognition, provide product information, answer questions, and even handle return and exchange inquiries, guide customers to visit, and based on the customer's purchase history or real-time interaction data, the sales robot can accurately recommend related products, thereby increasing sales opportunities; robots used in service operation scenarios, such as self-maintenance robots, where the self-maintenance robot is an embodied intelligent robot with powerful autonomous maintenance capabilities. The self-maintenance robot can perform predictive maintenance based on real-time data parameters, and adjust its own operating mode in time through the AI ​​cerebellum to minimize damage, thereby extending the service life of the equipment and improving production efficiency under optimal conditions. In addition, the self-maintenance robot can automatically perform maintenance tasks such as lubrication, cleaning and replacement of worn parts, reducing the need for human intervention.

[0231] Among them, the embodied intelligent terminal 2 described above can be divided into two types of embodied intelligent terminals 2: assistant-type embodied intelligent terminal 2 and autonomous-type embodied intelligent terminal 2. The assistant-type embodied intelligent terminal 2 focuses more on assisting humans in product production, with strong human-computer interaction capabilities, strong equipment production performance and safety performance. The autonomous-type embodied intelligent terminal 2 can independently carry out production work without humans, with strong autonomy and high production efficiency.

[0232] The embodied intelligent system provided by the embodiment of the present application includes an embodied intelligent server and at least one embodied intelligent terminal. Among them, each embodied intelligent terminal in the at least one embodied intelligent terminal is connected to the embodied intelligent server, and the embodied intelligent server is used to store and train the embodied intelligent model, wherein the embodied intelligent model is obtained by training each embodied intelligent terminal in the at least one embodied intelligent terminal, and the embodied intelligent terminal is used to respond to the user instruction and call the embodied intelligent model stored in the embodied intelligent server to implement the function corresponding to the user instruction, wherein, in the embodied intelligent system, when each embodied intelligent terminal implements the function corresponding to the user instruction, it is implemented by the embodied intelligent model stored in the embodied intelligent server based on the training of each embodied intelligent terminal in the at least one embodied intelligent terminal, so as to solve the problem of poor coordination between the embodied intelligent terminals in the embodied intelligent system, thereby improving the coordination and accuracy of the embodied intelligent system.

[0233] In a possible embodiment, the embodied intelligent terminal 2 is used to respond to the user instruction and call the embodied intelligent model stored in the embodied intelligent server 1 to implement the function corresponding to the user instruction. The process may include:

[0234] Optionally, first, the embodied intelligent terminal 2 is used to send user instructions to the embodied intelligent server 1, so that the embodied intelligent server 1 generates and sends information collection instructions to the embodied intelligent terminal 2 based on the stored embodied intelligent model.

[0235] For the description of user instructions and embodied intelligence models, please refer to Figure 1 The description in the illustrated embodiment will not be repeated here. Specifically, after receiving the user instruction, the embodied intelligent terminal 2 can send the user instruction to the embodied intelligent server 1, so that the embodied intelligent server 1 generates and sends the information collection instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model. Among them, the present application does not limit the process of the embodied intelligent server 1 generating and sending the information collection instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model. Optionally, at least one embodied intelligent model corresponding to the received user instruction can be determined from the stored embodied intelligent models, and based on the corresponding at least one embodied intelligent model, an information collection instruction is generated and sent to the embodied intelligent terminal 2.

[0236] The information collection instruction is an instruction to instruct the corresponding embodied intelligent terminal 2 to collect the corresponding terminal information. The present application does not limit the information collection instruction. Any instruction that can instruct the corresponding embodied intelligent terminal 2 to collect the corresponding terminal information can be used as the information collection instruction provided by the present application. The terminal information is information related to the corresponding embodied intelligent terminal 2. The present application does not limit the terminal information. Any information related to the corresponding embodied intelligent terminal 2 can be used as the terminal information provided by the present application. Optionally, the terminal information can include body information and external body information.

[0237] Optionally, secondly, the embodied intelligent terminal 2 is used to respond to the information collection instruction, obtain and send the corresponding terminal information to the embodied intelligent server 1, so that the embodied intelligent server 1 generates and sends the corresponding execution instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model.

[0238] Specifically, the embodied intelligent terminal 2 can respond to the received information collection instruction and obtain and send the corresponding terminal information to the embodied intelligent server 1. The present application does not limit the process of obtaining and sending the corresponding terminal information to the embodied intelligent server 1 in response to the received information collection instruction, wherein the process of obtaining and sending the corresponding terminal information to the embodied intelligent server 1 in response to the received information collection instruction is related to the information collection device of the embodied intelligent terminal 2, the information collection instruction, and the terminal information collected as instructed by the information collection instruction.

[0239] Specifically, after the embodied intelligent terminal 2 is used to respond to the information collection instruction, obtain and send the corresponding terminal information to the embodied intelligent server 1, the embodied intelligent server 1 can generate and send the corresponding execution instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model according to the received corresponding terminal information. Among them, the present application does not limit the process of generating and sending the corresponding execution instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model according to the received corresponding terminal information. Optionally, at least one corresponding embodied intelligent model can be determined from the stored embodied intelligent models according to the user instruction and the corresponding information received, and based on the determined at least one embodied intelligent model, a corresponding execution instruction is generated and sent to the embodied intelligent terminal 2.

[0240] Among them, the execution instruction is an instruction that instructs the corresponding embodied intelligent terminal 2 to perform a corresponding action. The present application does not limit the execution instruction, and any instruction that can instruct the corresponding embodied intelligent terminal 2 to perform a corresponding action can be used as the execution instruction provided by the present application.

[0241] Optionally, then, the embodied intelligent terminal 2 is used to respond to the execution instruction and implement the function corresponding to the execution instruction.

[0242] Specifically, the embodied intelligent terminal 2 can respond to the execution instruction described above to implement the function corresponding to the execution instruction, that is, to execute the action corresponding to the execution instruction.

[0243] Figure 8 Schematic diagram of the structure of the embodied intelligent system provided for this application Figure 2 ,like Figure 8 As shown, in this embodiment Figure 2 Based on the embodiment, another embodied intelligent system 100 is provided, which includes: an embodied intelligent server 1 and at least one embodied intelligent terminal 2. Each of the at least one embodied intelligent terminal 2 is connected to the embodied intelligent server 1. The embodied intelligent server 1 is used to store and train the embodied intelligent model. The embodied intelligent model is obtained by training each of the at least one embodied intelligent terminal 2. The embodied intelligent terminal 2 is used to call the embodied intelligent model stored in the embodied intelligent server 1 in response to a user instruction to implement the function corresponding to the user instruction.

[0244] For the description of the embodied intelligent server 1 and the embodied intelligent terminal 2 described above, please refer to Figure 1 The description of the illustrated embodiment will not be repeated here.

[0245] Optionally, the embodied intelligent terminal includes a terminal communication module 3 and a sensor module 4. The terminal communication module 3 is connected to the sensor module 4 and the embodied intelligent server 1 respectively. The information collection instruction includes a main body information collection instruction and an external body information collection instruction. The terminal information includes the main body information and the external body information.

[0246] Specifically, the terminal communication module 3 is a module in the embodied intelligent terminal 2 that processes information. The present application does not limit the process of information processing corresponding to the terminal communication module 3. Optionally, it can be a process of transmitting information or a process of performing other processing in response to received information.

[0247] Specifically, the sensor module 4 is a module in the embodied intelligent terminal 2 that senses and processes information. The present application does not limit the process of sensing and processing information corresponding to the sensor module 4. The process of sensing and processing information corresponding to the sensor module 4 corresponds to the information collection instruction.

[0248] Specifically, the entity information collection instruction is an instruction to instruct the corresponding embodied intelligent terminal 2 to collect the entity information included in the corresponding terminal information, wherein the entity information is information related to the entity of the corresponding embodied intelligent terminal 2. The present application does not limit the entity information, and any information related to the entity of the corresponding embodied intelligent terminal 2 can be used as the entity information provided by the present application. For example, the entity information includes but is not limited to information such as the position of the joints of the robotic arm itself, the motor operating speed, and the size of the mechanical structure.

[0249] Specifically, the external body information collection instruction is an instruction to instruct the corresponding embodied intelligent terminal 2 to collect the external body information included in the corresponding terminal information, wherein the external body information is information related to the external body of the corresponding embodied intelligent terminal 2, wherein the present application does not limit the external body information, and any information related to the external body of the corresponding embodied intelligent terminal 2 can be used as the external body information provided by the present application, for example: the external body information includes but is not limited to environmental information, operation object information, operation process information and other information, wherein the environmental information includes but is not limited to: background information, temperature information, humidity information, lighting information and other information, the operation object information includes but is not limited to: component information, assembly information and other information, the operation process information includes but is not limited to: RGB-D image information in three-dimensional scenes, infrared signal information, force sensing information and other information.

[0250] Optionally, the process in which the embodied intelligent terminal 2 is used to obtain and send corresponding terminal information to the embodied intelligent server 1 in response to the information collection instruction, so that the embodied intelligent server 1 generates and sends corresponding execution instructions to the embodied intelligent terminal 2 based on the stored embodied intelligent model, may include:

[0251] Optionally, first, the terminal communication module 3 is used to obtain and send the body information to the embodied intelligent server 1 in response to the received body information collection instruction, and send the received external body information collection instruction to the sensor module 4.

[0252] Specifically, after receiving the entity information collection instruction in the information collection instruction, the terminal communication module 3 of the embodied intelligent terminal 2 can respond to the received entity information collection instruction, obtain the entity information corresponding to the entity information collection instruction, and send the obtained entity information to the embodied intelligent server 1.

[0253] Specifically, after receiving the external body information collection instruction in the information collection instruction, the terminal communication module 3 of the embodied intelligent terminal 2 can send the received external body information collection instruction to the sensor module 4.

[0254] Optionally, after receiving the user instruction, the terminal communication module 3 of the embodied intelligent terminal 2 can also send the received user instruction to the embodied intelligent server 1, so that the embodied intelligent server 1 generates and sends information collection instructions to the embodied intelligent terminal 2 based on the stored embodied intelligent model.

[0255] Optionally, secondly, the sensor module 4 is used to respond to the external body information collection instruction received through the terminal communication module 3, obtain and send the external body information to the embodied intelligent server 1 through the terminal communication module 3.

[0256] Specifically, after receiving the external body information collection instruction sent by the terminal communication module 3 of the embodied intelligent terminal 2, the sensor module 4 of the embodied intelligent terminal 2 can respond to the external body information collection instruction received through the terminal communication module 3, obtain the external body information corresponding to the external body information collection instruction, and then send the obtained external body information collection instruction to the terminal communication module 3 of the embodied intelligent terminal 2, so that the terminal communication module 3 of the embodied intelligent terminal 2 sends the external body information to the embodied intelligent server 1.

[0257] Optionally, then, the embodied intelligent server 1 is used to generate and send corresponding execution instructions to the embodied intelligent terminal 2 based on the stored embodied intelligent model according to the body information and the external body information.

[0258] Specifically, after receiving the body information and external body information sent by the terminal communication module 3 of the embodied intelligent terminal 2, the embodied intelligent server 1 can generate and send corresponding execution instructions to the embodied intelligent terminal 2 based on the body information and external body information and based on the stored embodied intelligent model. Optionally, in the process of generating and sending corresponding execution instructions to the embodied intelligent terminal 2 based on the body information and external body information and based on the stored embodied intelligent model, the embodied intelligent terminal 2 can also be self-detected and self-diagnosed, and early warning processing can be performed based on the results of the self-detection and self-diagnosis. Among them, this application does not limit the process of self-detection and self-diagnosis of the embodied intelligent terminal 2. Any process that can self-detect and self-diagnose the embodied intelligent terminal 2 can be used as the process of self-detection and self-diagnosis of the embodied intelligent terminal 2 provided by this application. Optionally, early warning can be given for conditions that affect subsequent motion planning, such as motor failure and mechanical structure damage.

[0259] Optionally, if the embodied intelligent terminal 2 includes a terminal sensor module 4 and a sensor module 4, the sensor module includes at least one sensor unit 9. The external body information collection instruction includes at least one external body sub-information collection instruction. The external body information includes at least one external body sub-information. Among them, the sensor unit 9 corresponds to the external body sub-information collection instruction and the external body sub-information one by one.

[0260] Specifically, the external body sub-information collection instruction is an instruction to instruct the sensor unit 9 in the corresponding embodied intelligent terminal 2 to collect the external body sub-information in the external body information included in the corresponding terminal information. Specifically, the sensor unit 9 is a module in the embodied intelligent terminal 2 that senses and processes the external body sub-information, wherein the present application does not limit the process of the sensor unit 9 sensing and processing the external body sub-information, wherein the process of the sensor unit 9 sensing and processing the external body sub-information corresponds to the external body sub-information collection instruction. Among them, the present application does not limit the number of sensor units 9 included in the sensor module 4, wherein the number of sensor units 9 included in the sensor module 4 is related to the type of the embodied intelligent terminal 2.

[0261] Optionally, the process of the sensing module 4, which is used to respond to the external body information collection instruction received through the terminal communication module 3, acquire and send the external body information to the embodied intelligent server 1 through the terminal communication module 3, may include: the sensing unit 9, which is used to respond to the corresponding external body sub-information collection instruction received through the terminal communication module 3, acquire and send the corresponding external body sub-information to the embodied intelligent server 1 through the terminal communication module 3.

[0262] Specifically, the sensing unit 9 can receive the external body sub-information collection instruction sent by the communication terminal module, and in response to the received external body sub-information collection instruction, can obtain the external body sub-information corresponding to the external body sub-information collection instruction, and send the obtained external body sub-information to the terminal communication module 3, so that the terminal communication module 3 sends the received external body sub-information to the embodied intelligent server 1.

[0263] Among them, by collecting multimodal data through multiple sensor units 9 in the sensor module 4, the accuracy of the embodied intelligent system 100 can be improved.

[0264] Optionally, the embodied intelligent terminal further includes an execution module 5 . The execution module 5 is connected to the terminal communication module 3 .

[0265] Specifically, the execution module 5 is a module that responds to the execution instruction and executes the action corresponding to the instruction. Among them, this application does not limit the execution module 5 included in the embodied intelligence, and any module that can respond to the execution instruction and execute the action corresponding to the instruction can be used as the execution module 5 provided by this application.

[0266] Optionally, the process of the embodied intelligent terminal 2 for implementing the function corresponding to the execution instruction in response to the execution instruction may include: a terminal communication module 3 for sending the received execution instruction to the execution module 5. The execution module 5 is used to implement the function corresponding to the execution instruction in response to the execution instruction.

[0267] Optionally, if the embodied intelligent server 1 includes an execution module 5, the execution module includes at least one execution unit 10. The execution instruction includes at least one sub-execution instruction. Among them, the execution unit 10 corresponds to the sub-execution instruction one by one. The external body information collection instruction includes at least one external body sub-information collection instruction.

[0268] Specifically, the sub-execution instruction is an instruction that instructs the execution unit 10 in the corresponding embodied intelligent terminal 2 to execute the corresponding action. Specifically, the sub-execution unit 10 is a unit in the embodied intelligent terminal 2 that responds to the corresponding sub-execution instruction and executes the action corresponding to the sub-execution instruction. The present application does not limit the number of execution units 10 included in the execution module 5, and the number of execution units 10 included in the execution module 5 is related to the type of the embodied intelligent module.

[0269] Optionally, the process in which the terminal communication module 3 is used to send the received execution instruction to the execution module 5 may include: the terminal communication module 3 is used to send the received sub-execution instruction to the corresponding execution unit 10.

[0270] Optionally, the execution module 5, which is used to implement the function corresponding to the execution instruction in response to the execution instruction, may include: an execution unit 10, which is used to implement the function corresponding to the corresponding sub-execution instruction in response to the corresponding sub-execution instruction.

[0271] Fig. 9 This is a schematic diagram of the structure of the embodied intelligent terminal provided by this application. Fig. 9 As shown, the embodied intelligent terminal 2 includes a terminal communication module 3, a sensor module 4 and an execution module 5, wherein the terminal communication module 3 is connected to the embodied intelligent server 1, the sensor module 4, and the execution module 5, respectively, wherein the sensor module 4 includes at least one sensor unit 9, and the execution module 5 includes multiple execution units 10.

[0272] Optionally, the embodied intelligence server includes a functional module 6. Specifically, the functional module 6 is a module in the embodied intelligence server 1 for storing and training the embodied intelligence model, wherein the description of the embodied intelligence model can refer to Figure 1 The description in the illustrated embodiment will not be repeated here. Optionally, the embodied intelligence server 1, for storing and training the embodied intelligence model, may include: a functional module 6, for storing and training the embodied intelligence model.

[0273] Optionally, the functional module 6 includes at least one functional unit 13, wherein the number of the functional units 13 corresponds to the number of embodied intelligence models stored in the functional module 6, and the functional unit 13 is used to store and train the corresponding embodied intelligence model. Optionally, the embodied intelligence model stored and trained in the functional unit 13 may be an embodied intelligence model corresponding to the perception function, or an embodied intelligence model corresponding to the memory function, or an embodied intelligence model corresponding to the cognitive function, or an embodied intelligence model corresponding to the reasoning function, or an embodied intelligence model corresponding to the decision-making function, or an embodied intelligence model corresponding to the planning function, or an embodied intelligence model corresponding to the control function, or an embodied intelligence model corresponding to the evolution function.

[0274] Optionally, the embodied intelligent server further includes a server communication module 7. The server communication module 7 is connected to the embodied intelligent terminal 2 and the functional module 6 respectively.

[0275] Specifically, the server communication module 7 is a module in the embodied intelligent server 1 that processes information, wherein the present application does not limit the process of processing information corresponding to the server communication module 7, and optionally, it may be a process of transmitting information.

[0276] Optionally, the process of the embodied intelligent terminal 2 for sending the user instruction to the embodied intelligent terminal 2 so that the embodied intelligent terminal 2 generates and sends the information collection instruction to the embodied intelligent terminal 2 based on the stored embodied intelligent model may include:

[0277] Optionally, first, the embodied intelligent terminal 2 is used to send the user instruction to the server communication module 7. Optionally, if the embodied intelligent terminal 2 includes a terminal communication module 3, the process of the embodied intelligent terminal 2 sending the user instruction to the server communication module 7 may include: the terminal communication module 3 of the embodied intelligent terminal 2 may also send the received user instruction to the server communication module 7 in the embodied intelligent server 1 after receiving the user instruction.

[0278] Optionally, secondly, the server communication module 7 is used to send the received user instruction to the functional module 6. Specifically, after receiving the user instruction, the server communication module 7 in the embodied intelligent server 1 can send the received user instruction to the functional module 6 in the embodied intelligent server 1.

[0279] Optionally, then, the function module 6 is used to determine the corresponding embodied intelligent model in response to the received user instruction, and generate an information collection instruction based on the corresponding embodied intelligent module, and send it to the embodied intelligent terminal 2 through the server communication module 7.

[0280] Specifically, after receiving the user instruction sent by the server communication module 7 of the embodied intelligent server 1, the functional module 6 of the embodied intelligent server 1 can determine the corresponding embodied intelligent model from the stored embodied intelligent models in response to the received user instruction, and generate an information collection instruction based on the corresponding embodied intelligent module. Specifically, the functional module 6 of the embodied intelligent server 1 can send the generated information collection instruction to the server communication module 7 of the embodied intelligent server 1, and the server communication module 7 of the embodied intelligent server 1 can send the generated information collection instruction to the terminal communication module 3 of the embodied intelligent terminal 2 after receiving the information collection instruction.

[0281] Optionally, if the embodied intelligent server 1 includes a server communication module 7 and a functional module 6, the embodied intelligent terminal is used to respond to the information collection instruction, obtain and send corresponding terminal information to the embodied intelligent server 1, so that the embodied intelligent server 1 generates and sends corresponding execution instructions to the embodied intelligent terminal 2 based on the stored embodied intelligent model. The process may include:

[0282] Optionally, first, the embodied intelligent terminal 2 is used to respond to the information collection instruction, obtain and send the corresponding terminal information to the server communication module 7. The process of the embodied intelligent terminal 2 responding to the information collection instruction, obtaining and sending the corresponding terminal information to the server communication module 7 can refer to the above description and will not be repeated here.

[0283] Optionally, secondly, the server communication module 7 is used to send the received terminal information to the function module 6.

[0284] Specifically, the server communication module 7 of the embodied intelligent server 1 can receive the terminal information sent by the terminal communication module 3 of the embodied intelligent terminal 2, and send the received terminal information to the functional module 6 of the embodied intelligent server 1.

[0285] Optionally, then, the functional module 6 is used to determine the corresponding embodied intelligent model in response to the received terminal information, and generate an execution instruction based on the corresponding embodied intelligent module, and send it to the embodied intelligent terminal 2 through the server communication module 7.

[0286] Specifically, after receiving the terminal information sent by the server communication module 7 of the embodied intelligent server 1, the functional module 6 of the embodied intelligent server 1 can determine at least one corresponding embodied intelligent model from the stored embodied intelligent models, and generate an execution instruction based on the corresponding at least one embodied intelligent module. Specifically, after generating the execution instruction, the functional module 6 of the embodied intelligent server 1 can send it to the server communication module 7 of the embodied intelligent server 1, and the server communication module 7 of the embodied intelligent server 1 can send the received execution instruction to the terminal communication module 3 of the embodied intelligent terminal 2.

[0287] Optionally, the embodied intelligent server further includes a storage module 8. The storage module 8 is connected to the functional module 6 and the communication module respectively.

[0288] Specifically, the storage module 8 is a module for storing data in the embodied intelligent server 1. The present application does not limit the data stored in the storage module 8. Optionally, the storage module 8 is used to store terminal data transmitted by the server communication module 7. The terminal data is the data of the embodied intelligent terminal 2 during operation.

[0289] Specifically, the terminal data includes but is not limited to instruction data and information data. For example, the instruction data may include the user instructions, collection instructions, execution instructions, internal body information collection instructions, external body information collection instructions, external body sub-information collection instructions, and sub-execution instructions described above, and the information data may include the terminal information, internal body information, external body information, and external body sub-information described above. Optionally, the terminal data may also include but is not limited to the execution results of the execution instructions and the execution results of the sub-execution instructions.

[0290] Optionally, if the embodied intelligent server 1 includes a storage module 8, the storage module includes at least one first storage unit 11. The terminal data includes at least one terminal sub-data. The terminal sub-data is the terminal data corresponding to the embodied intelligent terminal 2 of each embodied intelligent scene. The first storage unit 11 corresponds one-to-one to the terminal sub-data and the embodied intelligent scene.

[0291] Specifically, the first storage unit 11 is a unit included in the storage module 8 for storing terminal data corresponding to the embodied intelligent terminal 2 of an embodied intelligent scene. Figure 1 The description in the illustrated embodiment will not be repeated here. For example: if the embodied intelligent scenario applied by the embodied intelligent system 100 is a research and development design scenario in the manufacturing industry, and the number of embodied intelligent terminals 2 included in the research and development design scenario is one, then the terminal data includes a terminal sub-data, and the terminal sub-data is the terminal data corresponding to an embodied intelligent terminal 2 included in the research and development design scenario; for example: if the embodied intelligent scenario applied by the embodied intelligent system 100 is a research and development design scenario, a manufacturing scenario, an experimental testing scenario, a business management scenario, and a service operation scenario, and the number of embodied intelligent terminals 2 included in the research and development design scenario is one, the number of embodied intelligent terminals 2 included in the manufacturing scenario is five, and the number of embodied intelligent terminals 2 included in the experimental testing scenario is one. , the number of embodied intelligent terminals 2 included in the business management scenario is three, and the number of embodied intelligent terminals 2 included in the service operation scenario is two, then the terminal data includes five terminal sub-data, the terminal sub-data corresponding to the R&D and design scenario is the terminal data corresponding to one embodied intelligent terminal 2 included, the terminal sub-data corresponding to the production and manufacturing scenario is the terminal data corresponding to five embodied intelligent terminals 2 included, the terminal sub-data corresponding to the experimental testing scenario is the terminal data corresponding to one embodied intelligent terminal 2 included, the terminal sub-data corresponding to the business management scenario is the terminal data corresponding to three embodied intelligent terminals 2 included, and the terminal sub-data corresponding to the service operation scenario is the terminal data corresponding to two embodied intelligent terminals 2 included.

[0292] Optionally, the storage module 8 is used to store the terminal data transmitted by the server communication module 7, and may include: a first storage unit 11, which is used to store the corresponding terminal sub-data transmitted by the server communication module 7. If the embodied intelligence scenario is a research and development design scenario, the corresponding terminal sub-data may be design data; if the embodied intelligence scenario is a production and manufacturing scenario, the corresponding terminal sub-data may be manufacturing data; if the embodied intelligence scenario is a test scenario, the corresponding terminal sub-data may be test data; if the embodied intelligence scenario is a business management scenario, the corresponding terminal sub-data may be business data; if the embodied intelligence scenario is a service operation scenario, the corresponding terminal sub-data may be maintenance data.

[0293] Optionally, the storage module is also used to store embodied intelligent knowledge data. The embodied intelligent knowledge data is knowledge data related to the embodied intelligent scene. The present application does not limit the embodied intelligent knowledge data, and any knowledge data related to the embodied intelligent scene can be used as the embodied intelligent knowledge data provided by the present application. The embodied intelligent knowledge data includes at least one embodied intelligent knowledge sub-data. The present application does not limit the number of embodied intelligent knowledge sub-data included in the embodied intelligent knowledge data. The number of embodied intelligent knowledge sub-data included in the embodied intelligent knowledge data is associated with the embodied intelligent scene corresponding to the embodied intelligent system 100. Specifically, if the embodied intelligent system 100 corresponds to an embodied intelligent scene, the number of embodied intelligent knowledge sub-data included in the embodied intelligent knowledge data is associated with the embodied intelligent scene. If the embodied intelligent system 100 corresponds to multiple embodied intelligent scenes, the number of embodied intelligent knowledge sub-data included in the embodied intelligent knowledge data is associated with each embodied intelligent scene in the multiple embodied intelligent scenes. Among them, the embodied intelligence knowledge sub-data is industrial knowledge data or digital model data. Among them, the industrial knowledge data is the relevant knowledge of the industrial field corresponding to all embodied intelligence scenarios. The present application does not limit the industrial knowledge data. Optionally, it can be specific industrial knowledge, cross-domain knowledge, and multidisciplinary knowledge related to industrial operations, such as computer-aided design knowledge CAD, product lifecycle management knowledge PLM, Internet of Things knowledge IoT, enterprise resource planning knowledge ERP, software configuration management knowledge SCM and other knowledge. Digital model data are the industrial fields corresponding to all embodied intelligence scenarios, and the relevant models of the basic industrial fields. The present application does not limit the digital model data. Optionally, it can be the industrial basic model and the model in the artificial intelligence model library of the industrial field.

[0294] Optionally, the storage module further includes at least one second storage unit 12. The second storage unit 12 corresponds to the embodied intelligence knowledge sub-data one by one. Optionally, the second storage unit 12 is used to store the embodied intelligence knowledge sub-data.

[0295] Optionally, the storage module is also used to store the corresponding historical terminal data transmitted by the server communication module 7. The historical terminal data is the data of the embodied intelligent terminal 2 during the historical operation process.

[0296] Specifically, the historical terminal data includes multiple historical terminal sub-data, wherein the historical terminal sub-data is the historical terminal data corresponding to the embodied intelligent terminal 2 of each embodied intelligent scene; optionally, the first storage unit 11 is also used to store the corresponding historical terminal sub-data transmitted by the server communication module 7.

[0297] Fig.10 This is a schematic diagram of the structure of the embodied intelligent server provided in this application. Fig.10 As shown, the embodied intelligent server 1 includes a server communication module 7, a functional module 6 and a storage module 8, wherein the server communication module 7 is connected to the embodied intelligent terminal 2, the functional module 6, and the storage module 8, respectively, and the functional module 6 is connected to the storage module 8, wherein the functional module 6 includes at least one functional unit 13, and the storage module 8 includes at least one first storage unit 11 and at least one second storage unit 12.

[0298] Optionally, the process of training the embodied intelligence model includes:

[0299] Functional module 6 is used to obtain terminal data, embodied intelligence knowledge data, and historical terminal data from storage module 8 in real time, and train the stored embodied intelligence model in real time based on the obtained terminal data, embodied intelligence knowledge data, and historical terminal data.

[0300] Specifically, as the functions of the embodied intelligent terminals 2 are realized, the data stored in the storage module 8 is updated in real time, and the function module 6 performs real-time training on the stored embodied intelligent model based on the real-time updated data.

[0301] The embodied intelligent system provided in the embodiment of the present application is Figure 1On the basis of the illustrated embodiment, the embodied intelligent terminal includes a terminal communication module, a sensing module, and an execution module, and the embodied intelligent server includes a functional module, a server communication module, and a storage module, wherein the terminal communication module and the sensing module collect internal and external body information, thereby improving the accuracy of the embodied intelligent system, wherein the embodied intelligent model stored in the functional module and obtained through the training of each embodied intelligent terminal in at least one embodied intelligent terminal can improve the collaboration of the embodied intelligent system, wherein the embodied intelligent model is trained by the data stored in the memory, thereby improving the accuracy of environmental perception and the accuracy of function execution, thereby improving the accuracy, performance, efficiency and innovation of the embodied intelligent system, wherein the stored embodied intelligent model is trained in real time by the functional module, thereby improving the generalization, performance and efficiency of the embodied intelligent system.

[0302] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for training an embodied intelligence model for motion planning, characterized in that: Applied to an embodied intelligent server, the method comprises: Acquire an embodied intelligence initial model; wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial external perception layer, an initial ontology recognition layer, an initial motion planning layer, and a fusion layer; the fusion layer is used to fuse and align the remaining layers during the model training process; the initial ontology recognition layer is used to recognize different types of embodied intelligent terminals during the model training process; Acquire a training set of a model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language subsample, an in vitro perception subsample, an ontology recognition subsample, and a motion planning subsample; wherein the natural language subsample, the in vitro perception subsample, the ontology recognition subsample, and the motion planning subsample in each training sample correspond to each other; Extracting a training sample from the training set, and obtaining loss values ​​corresponding to each layer in the embodied intelligence initial model according to the training sample and the embodied intelligence initial model; For each layer of the embodied intelligence initial model except the fusion layer, the comprehensive loss value corresponding to the layer is determined according to the loss value corresponding to the layer, the loss value corresponding to the fusion layer and the preset weight ratio, and the global parameters of the layer are adjusted according to the comprehensive loss value corresponding to the layer to obtain the trained embodied intelligence initial model.

2. The method according to claim 1, characterized in that The method further comprises: If each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions, the embodied intelligence model is obtained; otherwise, the above training process is repeated until each trained layer in the trained embodied intelligence initial model meets the preset convergence conditions.

3. The method according to claim 2, characterized in that The step of obtaining the loss value corresponding to each layer in the embodied intelligence initial model according to the training sample and the embodied intelligence initial model includes: According to the extracted training samples and the embodied intelligence initial model, a first loss value, a second loss value, a third loss value, a fourth loss value, and a fifth loss value are obtained; Among them, the first loss value is the loss value corresponding to the initial natural language understanding layer; the second loss value is the loss value corresponding to the initial external perception layer; the third loss value is the loss value corresponding to the initial ontology cognition layer; the fourth loss value is the loss value corresponding to the initial motion planning layer; the fifth loss value is the loss value obtained according to the fusion layer; For each layer of the embodied intelligence initial model except the fusion layer, determining the comprehensive loss value corresponding to the layer according to the loss value corresponding to the layer, the loss value corresponding to the fusion layer and a preset weight ratio, and adjusting the global parameters of the layer according to the comprehensive loss value corresponding to the layer, including: Determine a first comprehensive loss value according to the first loss value, the fifth loss value and a preset weight ratio; and perform global parameter adjustment on the initial natural language understanding layer according to the first comprehensive loss value; Determine a second comprehensive loss value according to the second loss value, the fifth loss value and a preset weight ratio; perform global parameter adjustment on the initial in vitro perception layer according to the second comprehensive loss value; Determine a third comprehensive loss value according to the third loss value, the fifth loss value and a preset weight ratio; perform global parameter adjustment on the initial ontology cognitive layer according to the third comprehensive loss value, A fourth comprehensive loss value is determined according to the fourth loss value, the fifth loss value and a preset weight ratio; and a global parameter adjustment is performed on the initial motion planning layer according to the fourth comprehensive loss value.

4. The method according to claim 3, characterized in that The natural language sub-sample includes user instructions and the embodied intelligent action sequence corresponding to the user instructions; the extracorporeal perception sub-sample includes environmental information and the object category and object posture corresponding to the environmental information; the ontology cognition sub-sample includes ontology parameter information and the terminal target posture corresponding to the ontology parameter information; the motion planning sub-sample includes a joint angle planning sequence.

5. The method according to claim 4, characterized in that The first loss value, the second loss value, the third loss value, the fourth loss value, and the fifth loss value are obtained according to the extracted training samples and the embodied intelligence initial model, including: According to the user instructions, the environment information, the entity parameter information, and the embodied intelligence initial model included in the extracted training samples, a predicted embodied intelligence action sequence, a predicted object category, a predicted object posture, a predicted terminal target posture, a predicted joint angle planning sequence, and an instruction feature vector, an environment feature vector, an entity feature vector, and a posture feature vector are obtained; determining the first loss value according to the predicted embodied intelligent action sequence and the embodied intelligent action sequence; Determining the second loss value according to the predicted item category, the predicted item posture, the item category, and the item posture; Determining the third loss value according to the predicted terminal target posture and the terminal target posture; Determining the fourth loss value according to the predicted joint angle planning sequence and the joint angle planning sequence; The fifth loss value is determined according to the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector.

6. The method according to claim 5, characterized in that The method obtains predicted embodied intelligence action sequences, predicted object categories, predicted object postures, predicted terminal target postures, predicted joint angle planning sequences, and instruction feature vectors, environment feature vectors, entity feature vectors, and posture feature vectors according to the user instructions, environment information, entity parameter information, and embodied intelligence initial model included in the extracted training samples, including: Determine, according to the user instruction and the initial natural language understanding layer, an instruction feature vector of the user instruction and a predicted embodied intelligent action sequence corresponding to the user instruction; Determine, according to the environmental information and the initial in-vitro perception layer, an environmental feature vector of the environmental information, and a predicted object type and a predicted object posture corresponding to the environmental information; Determine, according to the ontology parameter information, the embodied intelligent action sequence, the object type, the object posture, and the initial ontology cognitive layer, an ontology feature vector of the ontology parameter information and a predicted terminal target posture corresponding to the ontology parameter information; According to the predicted terminal target posture, the user instruction, the environmental information, and the initial motion planning layer, a posture feature vector of the predicted terminal target posture and a predicted joint angle planning sequence of the embodied intelligent terminal are determined.

7. The method according to claim 5, characterized in that The step of determining the fifth loss value according to the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector comprises: Input the instruction feature vector, the environment feature vector, the body feature vector, and the posture feature vector into the fusion layer, and output a fused feature vector; The fifth loss value is determined according to the fused feature vector.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: If the types of the embodied intelligent terminals increase, the training set of the model is updated, and the embodied intelligent model is updated and trained according to the updated model training set.

9. A method for motion planning based on an embodied intelligence model, characterized in that: Applied to an embodied intelligent server, the method comprises: In response to a current user instruction sent by the embodied intelligent terminal, obtaining current environment information and current entity parameter information of the embodied intelligent terminal; Determine and output motion planning instructions according to the current user instructions, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model; Among them, the embodied intelligent model is a model trained based on the training method of the embodied intelligent model for motion planning described in any one of claims 1-8.

10. The method according to claim 9, characterized in that According to the current user instruction, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model, the motion planning instruction is determined and output, including: Determine, according to the current user instruction and the natural language understanding layer, a current embodied intelligent action sequence corresponding to the current user instruction; Determine, according to the current environment information and the in-vitro perception layer, a current object type and a current object posture corresponding to the current environment information; Determine a current terminal target posture corresponding to the current ontology parameter information according to the current ontology parameter information, the current embodied intelligent action sequence, the current object type, the current object posture, and the ontology cognitive layer; Determining a current joint angle planning sequence of the embodied intelligent terminal according to the current terminal target posture, the current user instruction, the current environment information, and the motion planning layer; According to the current joint angle planning sequence of the embodied intelligent terminal, a motion planning instruction is determined and output.

11. A training device for an embodied intelligent model for motion planning, characterized in that: include: The first acquisition module is used to acquire the embodied intelligence initial model; wherein the embodied intelligence initial model includes: an initial natural language understanding layer, an initial in vitro perception layer, an initial ontology cognition layer, an initial motion planning layer, and a fusion layer; the fusion layer is used to fuse and align the remaining layers during the model training process; the initial ontology cognition layer is used to recognize different types of embodied intelligent terminals during the model training process; the second acquisition module is used to acquire the training set of the model; wherein the training set of the model includes at least one training sample; the training sample includes a natural language sub-sample, an in vitro perception sub-sample, an ontology cognition sub-sample, and a motion planning sub-sample; wherein the natural language sub-sample, the in vitro perception sub-sample, the ontology cognition sub-sample, and the motion planning sub-sample in each of the training samples correspond to each other one by one; A training module is used to extract a training sample from the training set, and obtain the loss value corresponding to each layer in the embodied intelligence initial model according to the training sample and the embodied intelligence initial model; for each layer of the embodied intelligence initial model except the fusion layer, determine the comprehensive loss value corresponding to the layer according to the loss value corresponding to the layer, the loss value corresponding to the fusion layer and a preset weight ratio, and adjust the global parameters of the layer according to the comprehensive loss value corresponding to the layer to obtain the trained embodied intelligence initial model.

12. A device for model-based motion planning, characterized in that: include: A third acquisition module, configured to acquire current environment information and current entity parameter information of the embodied intelligent terminal in response to a current user instruction sent by the embodied intelligent terminal; A planning module, configured to determine and output a motion planning instruction according to the current user instruction, the current environment information, the current ontology parameter information, and the natural language understanding layer, the external perception layer, the ontology cognition layer, and the motion planning layer included in the embodied intelligence model; Among them, the embodied intelligent model is a model trained based on the training method of the embodied intelligent model for motion planning described in any one of claims 1-8.

13. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8 or claims 9 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 or claims 9 to 10 when executed by a processor.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8 or claims 9 to 10.

16. An embodied intelligent system, characterized in that: include: An embodied intelligent server and at least one embodied intelligent terminal; wherein each of the at least one embodied intelligent terminal is connected to the embodied intelligent server; The embodied intelligent server is used to store and train the embodied intelligent model; wherein the embodied intelligent model is obtained by training based on each of the at least one embodied intelligent terminal; the embodied intelligent model is a model trained based on the training method for the embodied intelligent model for motion planning according to any one of claims 1 to 8; The embodied intelligent terminal is used to call the embodied intelligent model stored in the embodied intelligent server in response to user instructions to implement the function corresponding to the user instructions.

Images