Robot program synthesis method, device and equipment and storage medium

Abstract

The invention provides a robot program synthesis method and device, equipment and a storage medium. The robot program synthesis method comprises the steps that robot candidate programs are acquired; each identifier in the robot candidate program is processed, and a program diagram is constructed; recording environment feedback data in the process of executing the robot candidate program, and constructing an environment context; key environment features are extracted from local perception through a convolutional neural network to form environment context features; aggregating the adjacent grammar symbol information of each identifier based on the program diagram to obtain a grammar context; establishing a relation between the code symbols and the environment context, and aggregating to obtain grammar features; inputting the environment context and the grammar context into a preset model for iterative training to obtain each possible program identifier probability; according to the environment context features and the grammar features, program identifiers are selected one by one, and a repaired robot program is obtained. Therefore, an efficient and accurate robot program can be generated.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a robot program synthesis method, device, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence (AI) technology, AI has profoundly changed the way people live and has had a far-reaching impact on all walks of life. In this process, robots, as an objective existence that can interact with the real world, naturally become an extension of human exploration of the world and a tool to assist and liberate humans. The combination with AI can further enhance the capabilities of robots and promote the application of intelligent robots in various fields.

[0003] For intelligent robots, programs are the key to achieving automation and fine control. An ideal robot program often has strict and comprehensive control logic, which enables the robot to work in complex and changing environmental conditions, and can complete the set goals even if the scene changes, and has good generalization ability.

[0004] However, for robot programs, the environment is an important factor that cannot be ignored. In actual situations, robots usually perceive the surrounding environment through sensors to collect data, and this data will become the basis for each decision. This leads to the robot's perception level of the environment directly affecting the synthesis results of the program. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a robot program synthesis method, device, equipment and storage medium.

[0006] In a first aspect, an embodiment of the present application provides a robot program synthesis method, comprising:

[0007] Obtain candidate robot programs to be executed;

[0008] Processing each identifier in the candidate robot program to construct a program graph, wherein each node in the program graph corresponds to each identifier in the candidate robot program, and an undirected edge in the program graph connects each node to represent a semantic connection in the program;

[0009] Recording corresponding environmental feedback data during the execution of the robot candidate program to construct an environmental context;

[0010] The potential environmental context representation is learned through convolutional neural networks, and key environmental features are extracted from local perception to form environmental context features;

[0011] Aggregating adjacent grammatical symbol information of each identifier based on the program graph to obtain a grammatical context;

[0012] Establish the connection between code symbols and environmental context, and aggregate them to obtain grammatical features;

[0013] Inputting the environmental context and the grammatical context into a preset model for iterative training to obtain each possible program identifier probability;

[0014] According to the probability of each possible program identifier, as well as the environmental context features and grammatical features, program identifiers are selected one by one to obtain the repaired robot program.

[0015] Optionally, processing each identifier in the robot candidate program to construct a program graph includes:

[0016] Processing the identifiers in the robot candidate program one by one, and creating a graph node for each identifier;

[0017] If the identifier corresponding to the graph node is a conditional part in a control flow statement, the left pointer of the previous node points to the graph node;

[0018] If the identifier corresponding to the graph node is not a conditional part in a control flow statement, the right pointer of the previous node is used to point to the graph node.

[0019] Optionally, recording corresponding environmental feedback data during the execution of the robot candidate program to construct an environmental context includes:

[0020] Based on the program graph, the robot candidate programs are divided into executable minimum code segments in sequence;

[0021] Starting from the initial state before the robot candidate program is executed, executing the segmented code fragments one by one to obtain a post-state;

[0022] Using the post-state as the pre-state of the next code snippet until all code snippets are executed;

[0023] The preceding state and the following state corresponding to each code snippet are respectively combined to obtain the environment context corresponding to each identifier in the robot candidate program.

[0024] Optionally, adjacent grammatical symbol information of each identifier is aggregated based on the program graph to obtain a grammatical context, including:

[0025] Get the vector representation of each node in the program graph by graph embedding;

[0026] The attention scores of neighbor nodes are determined through the graph attention mechanism and normalized to obtain the aggregated grammatical context.

[0027] In a second aspect, an embodiment of the present application provides a robot program synthesis device, including:

[0028] An acquisition module, used to acquire a candidate robot program to be executed;

[0029] A program graph construction module, used to process each identifier in the robot candidate program to construct a program graph; wherein each node in the program graph corresponds to each identifier in the robot candidate program, and the undirected edges in the program graph connect each node to represent the semantic connection in the program;

[0030] An environment context building module, used to record corresponding environment feedback data during the execution of the robot candidate program and build an environment context;

[0031] The environment context feature construction module is used to learn the potential environment context representation through convolutional neural network, extract key environment features from local perception, and construct environment context features;

[0032] A grammatical context acquisition module, used for aggregating adjacent grammatical symbol information of each identifier based on the program graph to obtain a grammatical context;

[0033] The grammatical feature acquisition module is used to establish the connection between code symbols and environmental contexts and aggregate the grammatical features;

[0034] A prediction module, used for inputting the environmental context and the grammatical context into a preset model for iterative training to obtain the probability of each possible program identifier;

[0035] The repair module is used to select program identifiers one by one according to each possible program identifier probability, environmental context features and grammatical features to obtain a repaired robot program.

[0036] Optionally, the program graph construction module is specifically used to:

[0037] Processing the identifiers in the robot candidate program one by one, and creating a graph node for each identifier;

[0038] If the identifier corresponding to the graph node is a conditional part in a control flow statement, the left pointer of the previous node points to the graph node;

[0039] If the identifier corresponding to the graph node is not a conditional part in a control flow statement, the right pointer of the previous node is used to point to the graph node.

[0040] Optionally, the environment context building module is specifically used to:

[0041] Based on the program graph, the robot candidate programs are divided into executable minimum code segments in sequence;

[0042] Starting from the initial state before the robot candidate program is executed, executing the segmented code fragments one by one to obtain a post-state;

[0043] Using the post-state as the pre-state of the next code snippet until all code snippets are executed;

[0044] The preceding state and the following state corresponding to each code snippet are respectively combined to obtain the environment context corresponding to each identifier in the robot candidate program.

[0045] Optionally, the grammatical context acquisition module is specifically used to:

[0046] Get the vector representation of each node in the program graph by graph embedding;

[0047] The attention scores of neighbor nodes are determined through the graph attention mechanism and normalized to obtain the aggregated grammatical context.

[0048] In a third aspect, an embodiment of the present application provides a robot program synthesis device, comprising: a processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to:

[0049] Execute the steps of the robot program synthesis method as described in any one of the first aspects.

[0050] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the robot program synthesis method as described in any one of the first aspects.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] In this application, a robot candidate program to be executed is obtained; each identifier in the candidate robot program is processed one by one to construct a program graph; the robot candidate program is executed, and the corresponding environmental feedback data is recorded during the robot execution process, and the environmental data is processed by a convolutional neural network to form an environmental context feature; based on the program graph, the aggregation of the adjacent grammatical symbol information of each identifier in the program is realized by a graph neural network to form a grammatical context; the environmental context and the grammatical context are used as model inputs into the iterative training model, and finally a robot program repair model based on the environmental context is obtained to generate an efficient and accurate robot program. Through the cyclic process of "try-evaluation-repair", valuable guidance is provided for the evolution of the program, thereby improving the quality and generalization ability of the robot program. A candidate program is obtained by an existing program synthesis method, and then the environmental observation information (environmental context) before and after each action can be obtained by executing the candidate program. In addition, the context of the candidate program grammatical level can also be modeled, and the environmental context and the grammatical context can be linked, so as to implicitly evaluate the impact of each program code segment, and the semantic conflicts in the code can be continuously corrected in the iteration, and the wrong program code segments can be corrected to obtain the correct program after repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0054] Figure 1 A schematic diagram of the principle of the robot program synthesis method provided in the embodiment of the present application;

[0055] Figure 2 A flowchart of a robot program synthesis method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of the robot program synthesis model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there can be a central component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there can be a central component at the same time.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0062] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0063] For example, Figure 1 A schematic diagram of the principle of the robot program synthesis method provided in the embodiment of the present application, such as Figure 1As shown in the figure, firstly, the candidate program is obtained, then the program graph is constructed, and code snippets are generated based on the program graph. When the robot executes the candidate program, local environment feedback is collected, and the environment context is constructed based on the generated code snippets and local environment feedback. The grammatical context and environment context constructed based on the program graph are input into the model together, and the model generates a repair program.

[0064] In this embodiment, for the robot program, the environment in which it is located is an important factor that cannot be ignored. In actual situations, the robot usually collects data through sensors to perceive the surrounding environment, and this data will become the basis for each decision. This causes the robot's perception level of the environment to directly affect the synthesis result of the program. By simultaneously modeling and aligning the local environmental perception information during the execution of the robot program with the context of the corresponding grammatical code segment, the model can be helped to better capture the semantics and grammatical information in the program. Therefore, the potential impact of each code segment on the robot and environmental state can be evaluated in a local environment, and the cognition of the environmental state can be updated for the model. Through the aggregation of environmental and grammatical information flows by neural networks, smooth guidance can be provided for the model to correct the erroneous semantics in the program, and finally a program that can complete the task and has stronger generalization ability is obtained under continuous iteration, reducing the difficulty of robot program application.

[0065] For example, Figure 2 A flowchart of a robot program synthesis method provided in an embodiment of the present application, such as Figure 2 As shown, the method in this embodiment may include the following steps:

[0066] Step S201: Obtain a candidate robot program to be executed.

[0067] In this embodiment, the robot candidate program is a robot program used to control the robot's movement process and control logic in the physical world so that the robot can make preliminary attempts and explorations according to the specified task description. The robot program is mainly composed of execution actions, environmental perception, and control flow statements. The robot candidate program can be obtained by an existing program synthesis method, or it can be obtained by a large language model based on the natural language description of the task. Then, taking the robot candidate program as the starting point, the corresponding local environment data is collected during the execution of the robot candidate program, and the program is repaired according to the model. Exemplarily, the repair operation may include modifying, deleting or adding robot actions, adjusting the conditions for executing actions, etc. The repaired program can be executed and repaired again, and this process can be repeated until the generated program can achieve the expected task objectives.

[0068] Step S202: Process each identifier in the robot candidate program to construct a program graph.

[0069] Each node in the program graph corresponds to each identifier in the candidate robot program, and the undirected edge in the program graph connects each node to represent the semantic connection in the program.

[0070] Exemplarily, the identifiers in the robot candidate program are processed one by one, and a graph node is created for each identifier; if the identifier corresponding to the graph node is a conditional part in a control flow statement, the left pointer of the previous node points to the graph node; if the identifier corresponding to the graph node is not a conditional part in a control flow statement, the right pointer of the previous node points to the graph node.

[0071] In this embodiment, each identifier in the candidate program of the robot is processed one by one, with the purpose of constructing a graph, in which each node corresponds to each identifier in the candidate program, and each node is connected with an undirected edge to represent the semantic connection in the program. In the process of constructing the graph, the text of the program is traversed and the identifiers therein are identified and then added to the graph as a node one by one, so as to realize the conversion of the program text into a program graph; in the process of repairing the candidate program of the robot, each identifier in the program needs to be processed one by one. In order to better represent the semantic connection between program codes and facilitate subsequent processing, the program text needs to be converted into a graph to express it. This conversion process can be achieved by constructing a graph, in which the nodes of the graph represent the identifiers in the program, and the edges represent the semantic connection between the identifiers. Among them, the left child of the node will point to the condition, and the right child will point to the action or other node. In the process of constructing the graph, the text of the program is traversed, and the identifiers therein are identified and then added to the graph as a node.

[0072] Step S203: Record the corresponding environmental feedback data during the execution of the robot candidate program to build an environmental context.

[0073] Exemplarily, based on the program graph, the robot candidate program is divided into the smallest executable code fragments in sequence; starting from the initial state before the execution of the robot candidate program, the divided code fragments are executed one by one to obtain the post-state; the post-state is used as the pre-state of the next code fragment until all code fragments are executed; the pre-state and post-state corresponding to each code fragment are combined respectively to obtain the environmental context corresponding to each identifier in the robot candidate program.

[0074] In this embodiment, in the robot program, only the identifiers representing the robot's actions (such as forward or right turn) can change the environment and state, while other identifiers are components of specific statements (such as if and while). That is, non-action identifiers cannot exist independently. Only when they appear together with action identifiers can the code block form a complete semantics. Therefore, when processing the robot program, this embodiment repairs a minimum segment with complete semantics composed of action identifiers and non-action identifiers as a whole. This embodiment defines a code segment as a triple S=<cond,token,body> Among them, cond represents the conditional code fragment when there is judgment logic, body represents the relevant robot action code or nested sub-code segment, and token represents the current identifier. When simply encountering an action identifier, the segment degenerates into S=<null,action,null> , where action represents the corresponding robot action identifier. By recursively parsing the abstract syntax tree corresponding to the program, we can finally get a set of T program identification segments, denoted as S T By executing code segment S t ∈S T , we can get the postcondition after the code segment is executed, that is, Combine it with the precondition to obtain fragment S t The environmental context representation Represents the state of the environment observed by the robot before the program is executed. Represents the state of the environment observed by the robot after the program is executed.

[0075] Step S204: learning the potential environmental context representation through a convolutional neural network, extracting key environmental features from local perception to form environmental context features.

[0076] In this embodiment, when the robot executes the program, each time the robot executes a step, it will more or less change its own state or the local state of the current environment, and after each step, it will observe the local environment (denoted as ) to obtain the local environment sequence of the execution program Among them, ac i represents the i-th action executed, n represents the total number of action steps executed by the robot, represents the local environment state obtained by the initial observation. The subsequent learning of the environment context representation is as follows Figure 3 As shown in the dotted box in (a), in order to utilize the environmental information contained in these images, this embodiment implicitly learns the environmental feature representation by applying a convolutional network layer, where the embedding vector of the lth layer can be expressed as:

[0077]

[0078] Among them: K is the convolution kernel in the convolutional neural network, g [l] is the activation function of the lth layer, Represents the matrix element at row x and column y in the environment matrix of layer l-1 in the convolutional network, Represents the vector length of the convolution kernel of the lth layer in the convolutional network in dimension H, Represents the vector length of the convolution kernel of the lth layer in the convolutional network in dimension W, K represents the vector length of the convolution kernel of the lth layer in the convolutional network in dimension C, h,w,c Represents the elements corresponding to the index subscripts h, w, and c in the three-dimensional matrix of the convolution kernel. Represents the element corresponding to the index subscript x+h-1,y+w-1,c in the environment matrix of the l-1 layer in the convolutional network. This formula describes the calculation process of the convolution operation, which uses the convolution kernel K to perceive the input image of the robot's local environment. Perform convolution operation to get the embedding vector of layer l During the calculation process, the convolution kernel slides on the input image and multiplies the pixel value at each position by the convolution kernel weight at the corresponding position, then sums and applies the activation function g [l] By applying convolutional network layers, it is possible to extract feature representations of the current local environment from the image, which can be used in the subsequent restoration process of the model. This implicit learning method allows the robot to automatically learn important features of the environment from the image without manually designing a feature extractor.

[0079] Step S205: Aggregate adjacent grammatical symbol information of each identifier based on the program graph to obtain a grammatical context.

[0080] Exemplarily, a vector representation of each node in a program graph is obtained by graph embedding; the attention scores of neighboring nodes are determined by a graph attention mechanism, and after normalization, an aggregated grammatical context is obtained.

[0081] In this embodiment, the robot program itself also has the properties of the program. Relying only on local environment observation information is not enough for the model to understand the grammatical structure of the program code fragment itself and the long-range association of the code. By considering the grammatical context, the semantic and grammatical associations between different code segments are captured during the model's repair of the program. At the same time, the feature of pruning according to grammatical correctness during the repair process is realized. Based on the previously generated candidate program graph, the relationship between the structure and identifiers in the program can be well displayed.

[0082] Step S206: Establish a connection between the code symbol and the environmental context, and aggregate to obtain grammatical features.

[0083] In this embodiment, when constructing the grammatical context, the node representation of each node i in the candidate program graph is first obtained by graph embedding: represents a real vector of length d. Then according to a scoring function represents a real number, which is used to calculate the attention score of its neighboring node j to reflect the importance of the neighboring node. Then, the softmax function is used to calculate the attention score of all neighboring nodes. The attention scores are normalized. Represents the set of neighbor nodes connected to node i in the graph; that is:

[0084]

[0085] Where A and W are learnable weight matrices, and ∥ represents a vector concatenation operation. Figure 3 As shown in the dashed box in (b), at this time, node i obtains a new representation by calculating the weighted average of the modified features of its neighboring nodes:

[0086]

[0087] Among them, h′ i represents the weighted vector representation of node i, α i,i Indicates that by α(h i ,h i ) The calculated weight of node i, α i,j Indicates that by α(h i ,h j ) The calculated weight of node j with respect to node i, h j represents the vector representation of node j in the graph, h k Represents the vector representation of node k in the graph, and LeakyReLU is the activation function, which is defined as:

[0088]

[0089] Then, after the previous steps After obtaining the corresponding environmental context representation of node i, the environmental context is concatenated with the grammatical context To achieve the purpose of establishing a connection between code symbols and environmental context.

[0090] In this embodiment, the feature extraction of each identifier node is performed by using the graph attention mechanism for each identifier to capture the corresponding context features. Similarly, each node in the program graph has a left pointer pointing to the node representing the condition and a right pointer indicating the successor node. The type of the identifier is stored in the value of the node, and each node also stores the relationship between the identifier and its corresponding code segment. This establishes the connection between the code symbol and the environmental context, and further aggregates the semantic and syntactic features.

[0091] Step S207: input the environmental context and the grammatical context into a preset model for iterative training to obtain the probability of each possible program identifier.

[0092] Step S208: According to each possible program identifier probability, as well as environmental context features and grammatical features, program identifiers are selected one by one to obtain a repaired robot program.

[0093] In this embodiment, the program repair process can be regarded as an iterative optimization process. The model will continuously modify the candidate program fragments to make the final generated program meet expectations. As input, the program identifier t is semantically repaired one by one, and the forward calculation method is:

[0094] i t =Sigmoid(W ix ·x t +W is ·s t-1 +b i ),

[0095] f t =Sigmoid(W fx ·x t +W fs ·s t―1 +b f ),

[0096]

[0097] o t =Sigmoid(W ox ·x t +W os ·s t―1 +b o ),

[0098] s t =o t ⊙Tanh(C t ),

[0099] Where: x t represents the vector representation of the t-th time step data in the input sequence, h′ t represents the weighted vector representation of the program identifier t, Represents the eigenvector obtained by convolution of the environment state matrix corresponding to the program identifier t, i t represents the input gate vector calculated at the tth time step, W ix represents the model weight corresponding to the input gate i with respect to the input sequence vector x, W is represents the model weight corresponding to the input gate i with respect to the hidden state vector s, s t―1 represents the hidden state vector at the t-1th time step, b i represents the bias weight of the input gate, f t represents the forget gate vector calculated at the tth time step, W fx represents the model weight of the forget gate f corresponding to the input sequence vector x, b f represents the bias weight of the forget gate f, Indicates update C t The intermediate vector, W cx represents the model weight of the cell state C corresponding to the input sequence vector x, b c represents the bias weight of the cell state C, C t represents the cell state calculated at the tth time step, f t Same definition as above, C t―1 represents the cell state calculated at the t-1th time step, i t Same definition as above, o t represents the output gate vector calculated at the tth time step, W ox represents the model weight of the output gate o corresponding to the input sequence vector x, b o represents the bias weight of the output gate o, s t represents the hidden state vector at the tth time step, and ⊙ represents element-wise multiplication.

[0100] The hidden state of LSTM corresponding to the current identifier t, C tis the corresponding cell state, and W and b are the model parameters that can be updated. The backward calculation method is similar. Later, in the process of repairing the program, the model uses multi-layer perceptron and Softmax to predict the appropriate program identifier. These identifiers will become part of the repaired program generated by the model. The multi-layer perceptron consists of multiple neuron layers. Each neuron layer is connected to the neurons in the previous and next layers, and information is transmitted and calculated through weights and biases. Softmax converts the output of the neural network into a probability distribution. In the process of repairing the program, the model uses the Softmax function to convert the output of the multi-layer perceptron into the probability of each possible program identifier. The specific training method of the model is shown in Algorithm 3.

[0101] In this embodiment, the potential impact of each code snippet in each program on the environment is implicitly evaluated by learning the embedded representation of the environment context. By using a convolutional neural network to model the environment, a series of hidden layer representations can be learned. These representations capture the key features and contextual information in the environment. For example, in a navigation task, the environment embedding can learn information such as the robot's current position, the position of obstacles, and the target position. These environmental embedding representations can implicitly evaluate the impact of each program identifier on the current environment, helping the model to better understand the semantic meaning and behavior of program fragments in different environments.

[0102] In this embodiment, the grammatical information flow in the code snippet is aggregated through a graph structure to capture the dependencies and semantic associations between program identifiers, and to constrain the correctness of the program at the grammatical level. Each program identifier is represented as a node in the graph, and adjacent tags are connected by edges. Subsequently, different weights are assigned according to the importance and relevance between the identifiers through a graph attention mechanism. By calculating the attention weights between each identifier and its neighboring nodes, the model can obtain a representation of the grammatical context, i.e., the grammatical context information of each identifier, so that the model can better understand and process complex program tasks.

[0103] Exemplarily, the present application also provides a robot program synthesis device, including: an acquisition module, used to acquire a robot candidate program to be executed; a program graph construction module, used to process each identifier in the robot candidate program and construct a program graph; wherein each node in the program graph corresponds to each identifier in the robot candidate program, and the undirected edges in the program graph connect each node to represent the semantic connection in the program; an environment context construction module, used to record the corresponding environment feedback data during the execution of the robot candidate program and construct the environment context; an environment context feature construction module, used to represent the potential environment context through a convolutional neural network The invention discloses a robot program which comprises a robot program which is a robot which is provided with a robot body and a robot which is provided with a robot body. The robot program comprises a robot body which is provided with a robot body and a robot body ... which is provided with a robot body. The robot program comprises a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body. The robot program comprises a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body. The robot program comprises a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which is provided with a robot body which

[0104] Exemplarily, the program graph construction module is specifically used to: process the identifiers in the robot candidate program one by one, and create a graph node for each identifier; if the identifier corresponding to the graph node is the conditional part in the control flow statement, the left pointer of the previous node points to the graph node; if the identifier corresponding to the graph node is not the conditional part in the control flow statement, the right pointer of the previous node is used to point to the graph node.

[0105] Exemplarily, the environmental context construction module is specifically used to: based on the program graph, sequentially divide the robot candidate program into the smallest executable code fragments; starting from the initial state before the execution of the robot candidate program, execute the divided code fragments one by one to obtain the post-state; use the post-state as the pre-state of the next code fragment until all code fragments are executed; and respectively combine the pre-state and post-state corresponding to each code fragment to obtain the environmental context corresponding to each identifier in the robot candidate program.

[0106] Exemplarily, the grammatical context acquisition module is specifically used to: obtain the vector representation of each node in the program graph in a graph embedding manner; determine the attention scores of neighbor nodes through a graph attention mechanism, and obtain the aggregated grammatical context after normalization.

[0107] In addition, the present application also provides a robot program synthesis device, including: a processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to: execute the steps in the above method embodiment.

[0108] It should be noted that those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "platform" herein.

[0109] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods. Among them, the computer-readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also exist in a communication device as discrete components.

[0110] The present application also provides a program product, which includes a computer program. The computer program is stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server implements any method of the above-mentioned embodiments of the present invention.

[0111] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0112] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A robot program synthesis method, characterized in that: include: Obtain candidate robot programs to be executed; Processing each identifier in the candidate robot program to construct a program graph, wherein each node in the program graph corresponds to each identifier in the candidate robot program, and an undirected edge in the program graph connects each node to represent a semantic connection in the program; Recording corresponding environmental feedback data during the execution of the robot candidate program to construct an environmental context; The potential environmental context representation is learned through convolutional neural networks, and key environmental features are extracted from local perception to form environmental context features; Aggregating adjacent grammatical symbol information of each identifier based on the program graph to obtain a grammatical context; Establish the connection between code symbols and environmental context, and aggregate them to obtain grammatical features; Inputting the environmental context and the grammatical context into a preset model for iterative training to obtain each possible program identifier probability; According to the probability of each possible program identifier, as well as the environmental context features and grammatical features, program identifiers are selected one by one to obtain the repaired robot program.

2. The robot program synthesis method according to claim 1, characterized in that: Each identifier in the robot candidate program is processed to construct a program graph, including: Processing the identifiers in the robot candidate program one by one, and creating a graph node for each identifier; If the identifier corresponding to the graph node is a conditional part in a control flow statement, the left pointer of the previous node points to the graph node; If the identifier corresponding to the graph node is not a conditional part in a control flow statement, the right pointer of the previous node is used to point to the graph node.

3. The robot program synthesis method according to claim 1, characterized in that: Record the corresponding environmental feedback data during the execution of the robot candidate program and build the environmental context, including: Based on the program graph, the robot candidate programs are divided into executable minimum code segments in sequence; Starting from the initial state before the robot candidate program is executed, executing the segmented code fragments one by one to obtain a post-state; Using the post-state as the pre-state of the next code snippet until all code snippets are executed; The preceding state and the following state corresponding to each code snippet are respectively combined to obtain the environment context corresponding to each identifier in the robot candidate program.

4. The robot program synthesis method according to claim 1, characterized in that: Based on the program graph, adjacent grammatical symbol information of each identifier is aggregated to obtain a grammatical context, including: Get the vector representation of each node in the program graph by graph embedding; The attention scores of neighbor nodes are determined through the graph attention mechanism and normalized to obtain the aggregated grammatical context.

5. A robot program synthesis device, characterized in that: include: An acquisition module, used to acquire a candidate robot program to be executed; A program graph construction module, used to process each identifier in the robot candidate program to construct a program graph; wherein each node in the program graph corresponds to each identifier in the robot candidate program, and the undirected edges in the program graph connect each node to represent the semantic connection in the program; An environment context building module, used to record corresponding environment feedback data during the execution of the robot candidate program and build an environment context; The environment context feature construction module is used to learn the potential environment context representation through convolutional neural network, extract key environment features from local perception, and construct environment context features; A grammatical context acquisition module, used for aggregating adjacent grammatical symbol information of each identifier based on the program graph to obtain a grammatical context; The grammatical feature acquisition module is used to establish the connection between code symbols and environmental contexts and aggregate the grammatical features; A prediction module, used for inputting the environmental context and the grammatical context into a preset model for iterative training to obtain the probability of each possible program identifier; The repair module is used to select program identifiers one by one according to each possible program identifier probability, environmental context features and grammatical features to obtain a repaired robot program.

6. The robot program synthesis device according to claim 5, characterized in that: The program graph construction module is specifically used for: Processing the identifiers in the robot candidate program one by one, and creating a graph node for each identifier; If the identifier corresponding to the graph node is a conditional part in a control flow statement, the left pointer of the previous node points to the graph node; If the identifier corresponding to the graph node is not a conditional part in a control flow statement, the right pointer of the previous node is used to point to the graph node.

7. The robot program synthesis device according to claim 5, characterized in that: The environment context building module is specifically used for: Based on the program graph, the robot candidate programs are divided into executable minimum code segments in sequence; Starting from the initial state before the robot candidate program is executed, executing the segmented code fragments one by one to obtain a post-state; Using the post-state as the pre-state of the next code snippet until all code snippets are executed; The preceding state and the following state corresponding to each code snippet are respectively combined to obtain the environment context corresponding to each identifier in the robot candidate program.

8. The robot program synthesis device according to claim 5, characterized in that: The grammatical context acquisition module is specifically used for: Get the vector representation of each node in the program graph by graph embedding; The attention scores of neighbor nodes are determined through the graph attention mechanism and normalized to obtain the aggregated grammatical context.

9. A robot program synthesis device, characterized in that: include: A processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to: Execute the steps of the robot program synthesis method described in any one of claims 1 to 4.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the robot program synthesis method according to any one of claims 1 to 4 are implemented.

Images