Live-line robot natural language instruction analysis and operation mapping method, electronic device and storage medium

Abstract

The application discloses a live-line work robot natural language instruction analysis and operation mapping method, an electronic device and a storage medium, the method comprises a model training step and a model using step, the model training step comprises the following steps: collecting natural language instruction data, and performing BIO slot labeling and intent labeling; a first mapping model is established, the first mapping model comprises a BERT module, a first BiLSTM module, a first Softmax layer and a first CRF module; the first CRF module performs slot labeling on time sequence capture data output by the first BiLSTM module; the first Softmax layer performs intent recognition on global semantic features output by the BERT module; and the first mapping module is trained. Through the use of the trained mapping model, the natural language instruction can be accurately and efficiently converted into an operation instruction executable by the robot, so that the autonomy and operation reliability of the live-line work robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and robot control technology, and particularly relates to a live-line work robot natural language instruction analysis and operation mapping method, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, robots have been widely applied to various complex work scenarios, especially in live-line work environments, robots not only need to perform highly precise operation tasks, but also must accurately analyze and respond to complex natural language instructions. However, the diversity and complexity of natural language make it difficult for traditional analysis methods based on simple word matching to meet the high precision and safety requirements in live-line work.

[0003] Therefore, there is an urgent need for a system that combines natural language processing and operation instruction mapping technology to accurately and efficiently convert natural language instructions into robot executable operation instruction codes, thereby improving the autonomy and operation reliability of live-line work robots. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provide a live-line work robot natural language instruction analysis and operation mapping method, an electronic device and a storage medium.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] In a first aspect of the present application, a live-line work robot natural language instruction analysis and operation mapping method is provided, including a model training step and a model using step, the model training step comprising:

[0007] Collect natural language instruction data and perform BIO slot labeling and intent labeling;

[0008] A first mapping model is established, the first mapping model comprising a BERT module, a first BiLSTM module, a first Softmax layer and a first CRF module; wherein the BERT module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the first BiLSTM module performs timing capture on the global semantic features output by the BERT module, and the first CRF module performs slot labeling on the timing capture data output by the first BiLSTM module; the first Softmax layer performs intent recognition on the global semantic features output by the BERT module;

[0009] The first mapping module is trained, including calculating slot loss using the slot labeling output by the first CRF module, calculating intent loss using the intent recognition output by the first Softmax layer, and training the first mapping module using the slot loss and the intent loss.

[0010] The model using step comprises:

[0011] Using the trained mapping model, an intent and a slot of an input natural language instruction are identified;

[0012] According to a mapping rule, the identified intent and slot are mapped into a specific operation instruction;

[0013] The operation instruction is forwarded to the live-line work robot.

[0014] Further, the collection of the natural language instruction data comprises: collecting natural language instruction data, including an operation scenario performed by the live-line work robot; and generating natural language instruction data in different expressions by using an AI model;

[0015] The BIO slot labeling is word-by-word labeling, and the intent labeling is whole sentence labeling.

[0016] Further, the BERT module and the Softmax layer comprise:

[0017] Action word feature extraction: a part-of-speech tagging tool is used to label action words in the training data to generate a position information vector v of the action words, the position information of the action words being represented by the vector v=[v1,v2,……v n ], wherein v i =1 indicates that the word is an action word, and otherwise, 0;

[0018] Self-attention mechanism enhancement: an action word attention weight formula is designed:

[0019]

[0020] e i is a self-attention score for modeling context semantics; v i is a saliency vector of the action word; and λ is a hyperparameter for adjusting the weight saliency of the action word.

[0021] Local and global information fusion: the enhanced action word feature is fused with a global semantic vector representation output by the BERT module to generate a final feature vector for intent classification:

[0022] H final =W concat ·[H global ;H verb ]

[0023] wherein W concat is a learnable weight matrix; and the global semantic feature H=[h1,h2,……h n ] of the natural language instruction is output by the BERT module, wherein hi is the context vector representation of the i-th word;

[0024] using the enhanced feature vector H final perform intent classification:

[0025] p intent = Softmax(W intent · H final + b intent )

[0026] where W intent is the weight matrix of intent classification, b intent is the bias vector of intent classification, used to adjust the mapped result.

[0027] Further, the first BiLSTM module and the first CRF module further comprise a context enhancement module and a semantic consistency constraint module;

[0028] The context enhancement module comprises:

[0029] dynamically extract the context features of the slot position keyword, and assign a weight a i to each word by introducing an attention mechanism:

[0030] a i = Softmax(W a · h i + b a )

[0031] where h i is the output of the first BiLSTM module, W a and b a are learnable parameters;

[0032] reweight the context features using the attention weight a i to generate an enhanced feature:

[0033] h′ i = h i + a i · Context(h)

[0034] where Context(h) is the weighted aggregation of the context features, which can strengthen the semantic expression of the keyword;

[0035] The semantic consistency constraint module comprises:

[0036] Define a slot embedding matrix: construct a slot embedding matrix E solt ∈ R N×d , where N is the number of slot labels, and d is the embedding dimension; each slot label j corresponds to an embedding vector Esolt [j], represents the semantic features of the slot type;

[0037] Fusing context and slot semantic information: the input vector of the first CRF module is obtained by adding the context feature h' i and the slot embedding E solt

[0038] x i =h′ i +E solt [j]

[0039] where j represents the index of the current slot label.

[0040] Further, the model training step further comprises:

[0041] establishing a second mapping model, the second mapping model comprising a second BiLSTM module, a third BiLSTM module, a second Softmax layer and a second CRF module; wherein the second BiLSTM module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the third BiLSTM module performs time sequence capture on the global semantic features output by the second BiLSTM module, and the second CRF module performs slot labeling on the time sequence capture data output by the third BiLSTM module; the second Softmax layer performs intent recognition on the global semantic features output by the second BiLSTM module;

[0042] training the second mapping model using the first mapping model.

[0043] Further, the training of the second mapping model using the first mapping model comprises:

[0044] using a cross-entropy loss function to calculate hard label loss for the slot labeling and intent recognition output by the second mapping module:

[0045]

[0046] where y is the real label, is the model prediction value, and w(j) is the class weight to balance the imbalance of different classes in the data;

[0047] using mean square error to calculate soft label loss for the probability distribution output by the first mapping model and the probability distribution output by the second mapping model:

[0048] L distill =||z (t) -z (S ||

[0049] where z (t) ​is a probability distribution output by the first mapping model, i.e., including a slot labeling distribution and an intent recognition distribution output by the first mapping model, z (S) is a probability distribution output by the second mapping model, i.e., including a slot labeling distribution and an intent recognition distribution output by the second mapping model;

[0050] The second mapping model is trained by using the hard label loss and the soft label loss.

[0051] Further, in the process of calculating the soft label loss, a temperature parameter T is introduced to adjust the output distribution of the Softmax layer:

[0052]

[0053] When T>1, the output distribution is smoother, and the information between categories is richer.

[0054] Further, the training of the second mapping model by using the first mapping model comprises:

[0055] The global semantic features output by the BERT module of the first mapping model and the global semantic features output by the second BiLSTM module of the second mapping model are calculated by using the mean square error for low-level loss calculation.

[0056] The slot distributions output by the first CRF module of the first mapping model and the slot distributions output by the second CRF module of the second mapping model are calculated by using the mean square error for middle-level loss calculation.

[0057] The first mapping model further comprises a third Softmax layer located after the first CRF module, and the first mapping model further comprises a fourth Softmax layer located after the second CRF module; the intent distributions output by the first Softmax layer of the first mapping model and the intent distributions output by the second Softmax layer of the second mapping model, the slot distributions output by the third Softmax layer of the first mapping model and the slot distributions output by the fourth Softmax layer of the second mapping model are calculated by using the KL divergence loss function for high-level loss calculation.

[0058] The second mapping model is trained by using the low-level loss calculation, the middle-level loss calculation and the high-level loss calculation.

[0059] In a second aspect, the present application provides an electronic device comprising a storage unit and a processing unit, wherein the storage unit stores computer instructions executable on the processing unit, and the processing unit executes the computer instructions to perform the steps of the method according to the first aspect.

[0060] In a third aspect, the present application provides a storage medium storing computer instructions, wherein the computer instructions are executable to perform the steps of the method according to the first aspect.

[0061] The beneficial effects of the present invention are:

[0062] In an exemplary embodiment of the present invention, a method is provided that can combine natural language processing and operation instruction mapping technology, by using a trained mapping model to accurately and efficiently convert natural language instructions into operation instructions executable by the robot, thereby improving the autonomy and operational reliability of the live working robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of a method for natural language instruction parsing and operation mapping for a live working robot provided in an exemplary embodiment of the present invention;

[0064] Figure 2 is a schematic diagram of a first mapping model provided in an exemplary embodiment of the present invention;

[0065] Figure 3 Schematic diagram of an action word attention module provided in an exemplary embodiment of the present invention;

[0066] Figure 4 A schematic diagram of a context enhancement module and a semantic consistency constraint module provided in an exemplary embodiment of the present invention;

[0067] Figure 5 A flowchart of a model training step provided in an exemplary embodiment of the present invention;

[0068] Figure 6 is a schematic diagram of a second mapping model provided in an exemplary embodiment of the present invention;

[0069] Figure 7 A schematic diagram of using a first mapping model to train a second mapping model provided in an exemplary embodiment of the present invention;

[0070] Figure 8 A schematic diagram of using a first mapping model to train a second mapping model provided in yet another exemplary embodiment of the present invention;

[0071] Figure 9 A schematic diagram of an operation confirmation and user feedback process provided in yet another exemplary embodiment of the present invention; DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described clearly and completely below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0073] In the description of the present application, it should be noted that the directions or positional relationships belonging to "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like are the directions or positional relationships described based on the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, "first", "second" are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0074] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, "mounting", "connection", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0075] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict.

[0076] Reference Figure 1 , Figure 1 The method for analyzing and mapping natural language instructions of a live working robot provided in an exemplary embodiment of the present application is shown, which includes a model training step and a model using step, and the model training step includes:

[0077] Collect natural language instruction data and perform BIO slot labeling and intent labeling;

[0078] Establish a first mapping model, as shown in Figure 2 The first mapping model includes a BERT module, a first BiLSTM module, a first Softmax layer and a first CRF module; wherein the BERT module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the first BiLSTM module performs timing capture on the global semantic features output by the BERT module, and the first CRF module performs slot labeling on the timing capture data output by the first BiLSTM module; the first Softmax layer performs intent recognition on the global semantic features output by the BERT module;

[0079] Training the first mapping module includes: calculating slot loss using the slot labeling output by the first CRF module, calculating intent loss using the intent recognition output by the first Softmax layer, and training the first mapping module using the slot loss and the intent loss.

[0080] The model using step comprises:

[0081] Using the trained mapping model, the intent and slot of the input natural language instruction are identified;

[0082] According to the mapping rule, the identified intent and slot are mapped into specific operation instructions;

[0083] The operation instructions are forwarded to the live-line work robot.

[0084] Specifically, in the present exemplary embodiment, a method capable of combining natural language processing and operation instruction mapping technology is provided, which accurately and efficiently converts natural language instructions into operation instructions executable by a robot by using a trained mapping model, thereby improving the autonomy and operation reliability of the live-line work robot:

[0085] For the first mapping model:

[0086] The BERT module is used for word embedding of each word in the natural language instruction to generate context-related vector representation. Specifically, the BERT module can capture the semantic information of each word in the instruction through its multi-layer bidirectional Transformer structure, ensuring that the embedding vector fully expresses the meaning of the word in the instruction context. These word vectors provide a semantic basis for subsequent temporal feature extraction, enabling the system to have higher accuracy in parsing natural language.

[0087] After obtaining the word embedding, the BiLSTM model is used to capture the temporal relationship and dependency between words. BiLSTM can process the sequence features in the natural language instruction, capturing the information of the preceding and following text through its bidirectional structure, thereby improving the understanding ability of actions, objects, locations and other elements in natural language.

[0088] The CRF model follows the BiLSTM model and is used for slot labeling and decoding. The CRF model optimizes the labeling of slots by analyzing global information in the output layer, ensuring that slot information (such as tool name, wire type, location, etc.) can be accurately extracted in complex instructions.

[0089] This combination ensures the consistency and accuracy of slot labeling, and even in the case of long instructions and multiple intents, it can achieve high-precision parsing results.

[0090] The input natural language instruction is first embedded by the BERT module, and then sent to the Softmax layer and the BiLSTM module + CRF module, respectively. The Softmax layer is used for intent recognition, and the BiLSTM + CRF module is used for slot filling. Finally, the total loss is calculated through the intent loss and slot loss to achieve the goal of multi-task learning.

[0091] Intention and slot mapping rules: the system ensures that natural language commands can be accurately mapped to operation instruction codes by defining intention recognition and slot filling mechanisms. Each intention corresponds to a specific operation behavior, while slots are used to fill in object attributes (such as tools, wire types, locations, etc.) in the instruction, enabling seamless integration of natural language commands and robot operation instructions.

[0092] Through this data labeling and model training method, the system can better understand the timing and semantic features of complex natural language commands, providing a foundation for subsequent operation instruction generation and mapping.

[0093] The following will describe the preferred exemplary embodiments:

[0094] More preferably, in an exemplary embodiment, the collection of natural language instruction data includes: collecting natural language instruction data, including the operation scenarios performed by the hot-line work robot; using an AI model to generate natural language instruction data in different expressions;

[0095] The BIO slot labeling is word-by-word labeling, and the intention labeling is sentence-by-sentence labeling.

[0096] Specifically, in the present exemplary embodiment, the specific steps of collecting natural language instruction data are first disclosed: a large amount of natural language instruction data is first collected, covering various operation scenarios that the hot-line work robot can perform, such as "grabbing a spiral clamp wire" and "stripping a main wire". To enhance the system's ability to understand diversity, the present exemplary embodiment further applies an AI model (such as a GPT model) to generate natural language instruction samples in different expressions, expanding the data set and ensuring that the system can handle various instruction expression forms and accurately analyze user intentions.

[0097] (1) Natural language instruction data set preparation

[0098] The collected natural language instruction data covers various operation behaviors and corresponding parameter configurations that the hot-line work robot may involve, such as common operation steps such as grabbing, cutting, and installing. Each instruction contains specific operations and their objects, locations, tool types, etc., to ensure that the system can identify the specific needs of different instructions in hot-line work.

[0099] (2) Slot BIO labeling and intention labeling

[0100] The system uses a BIO tagging mechanism to finely tag key operation words in natural language instructions. The BIO tagging mechanism is used to structure the instruction text according to the intent and slot attributes to ensure the accuracy and consistency of the parsing. During the tagging process, the intent is tagged separately as a whole sentence, and the slot information is represented by the BIO tag word by word. Table 1 shows common tagging labels and their corresponding slot types, each label corresponds to specific tools, wire types, positions, etc. required by the live-line robot.

[0101] Label class BIO label Example text Tool slot - wire stripper B-Tool-WireStripper Wire stripper Tool slot - spiral clamp tool B-Tool-SpiralClampTool Spiral clamp tool Tool slot - wire cutter B-Tool-WireCutter Wire cutter Tool slot - splice clamp B-Tool-SpliceClamp Splice clamp Tool slot - J clamp B-Tool-JClamp J clamp Tool slot - grounding ring tool B-Tool-GroundingRingTool Grounding ring tool Wire type slot - main B-WireType-Main Main Wire type slot - branch B-WireType-Branch Branch Position slot - near phase B-Position-NearPhase Near phase Position slot - mid phase B-Position-MidPhase Mid phase Position slot - far phase B-Position-FarPhase Far phase Clip type slot - splice clamp B-ClipType-SpliceClamp Splice clamp Clip type slot - J clamp B-ClipType-JClamp J clamp Force slot - max force B-Force-Max Max force Diameter slot B-Diameter 5.5mm Length slot B-Length 12cm

[0102] Table 1 BIO tagging mechanism example

[0103] Tagging example:

[0104] In the actual parsing process of natural language instructions, the present exemplary embodiment identifies the slot attributes of each word through BIO tagging to ensure the accuracy and consistency of the parsing process. For example, for the instruction "please grab the wire stripper", the BIO tagging result is as follows:

[0105] Instruction text: please grab the wire stripper

[0106] Intent label: grab_tool

[0107] BIO slot tagging: O O O B-Tool-WireStripper I-Tool-WireStripper I-Tool-WireStripper

[0108] More preferably, in an exemplary embodiment, as shown in Figure 3 , the BERT module and the Softmax layer include:

[0109] Action word feature extraction: use part-of-speech tagging tools to tag action words in training data to generate position information vector v of action words. The position information of the action word is represented by the vector v = [v1, v2, … v n ], where v i = 1 indicates that the word is an action word, otherwise 0;

[0110] Self-attention mechanism enhancement: design an action word attention weight formula:

[0111]

[0112] e i is the self-attention score, which is used to model the context semantics; v i is the saliency vector of the action word; λ is a hyperparameter that adjusts the weight saliency of the action word;

[0113] Local and global information fusion: the enhanced action word features are fused with the global semantic vector representation output by the BERT module to generate the final feature vector for intent classification:

[0114] H final =W concat ·[H global ; H verb ]

[0115] where W concat is a learnable weight matrix; the global semantic features H = [h1, h2, … h n ] output by the BERT module for natural language instructions, where h i is the contextual vector representation of the i-th word;

[0116] Use the enhanced feature vector H final to classify the intent:

[0117] p intent = Softmax(W intent · H final + b intent )

[0118] where W intent is the weight matrix for intent classification, and b intent is the bias vector for intent classification, used to adjust the mapped results.

[0119] Specifically, for the intent recognition module corresponding to the first Softmax layer, the existing framework has the following defects: (1) Insufficient sensitivity to action words: In the original framework, intent classification mainly relies on global semantic information (overall sentence features extracted by the BERT module), and there is a lack of significant marking and focus on key action words (such as "pick up" "put back" etc.). (2) High dependence on noise words: Noise words such as "please" "help me" "take a little" can interfere with intent classification, making the model rely more on the overall pattern of the sentence rather than the core action information. This interference can lead to ambiguous semantics or classification errors, especially when similar intents need to be strictly distinguished (such as "pick up tools" and "put back tools"). (3) Insufficient combination of local and global information: The original framework mainly uses global information, lacks specialized modeling of local key features (such as action words), and thus lacks robustness for diverse natural language instructions.

[0120] In the improved scheme in the present exemplary embodiment, an "action word attention module" is introduced. Goal: Add a verb feature extraction and enhancement module to the existing framework to enhance the model's attention to key action words while retaining the ability to model global semantic information. The specific implementation is as shown in the above content.

[0121] The adoption of this approach has the following advantages: (1) modeling of the saliency of key action words: traditional methods rely on global semantic information, while the present exemplary embodiment assigns the model stronger perception of action words by introducing a verb attention module. (2) dual modeling of global and local information: the verb attention module provides explicit action word enhancement for intent classification, while retaining the global semantic features of the BERT module, achieving complementarity between the two through feature fusion. (3) dual modeling of global and local information: the verb feature enhancement module can effectively ignore the interference of noise words such as "please" and "help me", making the model more focused on capturing core semantics and improving the generalization ability for natural language instructions.

[0122] In a specific exemplary embodiment, the part-of-speech tagging tool is used to label the action words in the training data in the action word feature extraction, and an action word labeling table (Table 2) can be used:

[0123] Intention label Action word grab_tool grab, take, get, pick up, take return_tool put back, return grab_wire grab, take, get, pick up, take strip_wire strip, peel, strip, peel off arrange_wire smooth, straighten, arrange, comb, organize cut_wire cut, trim, sever, cut off install_clamp install, fix, mount, install... clamp, place... clamp tighten_clamp tighten, secure, tighten, tighten remove_clamp remove, take off, remove, take off, take away adjust_clamp adjust, correct, modify thread_wire thread, insert, insert, lead, feed, lead hang_wire hang, mount, arrange grab_tool grab, take, get, pick up, take return_tool put back, return grab_wire grab, take, get, pick up, take

[0124] Table 2 Action word labeling table

[0125] More preferably, in an exemplary embodiment, as shown in Figure 4 , the first BiLSTM module and the first CRF module further include a context enhancement module and a semantic consistency constraint module;

[0126] The context enhancement module includes:

[0127] Dynamically extract the context features of the slot keywords, and assign a weight α i to each word by introducing an attention mechanism:

[0128] α i = Softmax(W a ·h i +b a )

[0129] where h i is the output of the first BiLSTM module, W a and b a are learnable parameters;

[0130] The context features are reweighted using the attention weight α i to generate enhanced features:

[0131] h′ i = h i + α i ·Context(h)

[0132] wherein Context(h) is a weighted aggregation of the context features, which can strengthen the semantic expression of the keyword;

[0133] The semantic consistency constraint module comprises:

[0134] Define the slot embedding matrix: construct a slot embedding matrix E solt ∈R N×d wherein N is the number of slot labels, and d is the embedding dimension; each slot label j corresponds to an embedding vector E solt [j], which represents the semantic features of the slot type.

[0135] Fusion of context and slot semantic information: the input vector of the first CRF module is obtained by adding the context feature h i and the slot embedding E solt

[0136] x i =h′ i +E solt [j]

[0137] wherein j represents the index of the current slot label.

[0138] Specifically, in the present exemplary embodiment, for the first BiLSTM module and the first CRF module, the existing framework has the following defects: (1) insufficient local feature extraction of the slot recognition module: the current BiLSTM+CRF mainly captures the context information through sequence modeling, but for words such as "wire stripper" and "fruit stripper" that are easy to confuse, the semantic distinction is insufficient, and the model cannot effectively associate the keyword and the slot type. (2) unable to introduce explicit modeling of the keyword: the explicit importance of the keyword (such as "wire stripper") is ignored in the slot recognition process, which leads to the model being easily disturbed by noise words when processing diversified natural language expressions.

[0139] In the present exemplary embodiment, a dynamic context enhancement module is introduced, and the module position is: before the output of the first BiLSTM module to the first CRF module, a context enhancement module is added; the semantic consistency constraint is introduced: a fixed semantic embedding vector E solt is introduced for each slot label (such as B-Tool-WireStripper, B-Tool-SpiralClampTool), which enhances the model's ability to distinguish slot types and boundaries by fusing the semantic embedding with the context feature h i . The specific implementation manner is as described above.

[0140] ​(1)Enhancing the ability to identify slot boundaries: By introducing a dynamic context enhancement module, the attention mechanism is introduced into the slot identification process, which realizes the dynamic weighting of the keyword context features. Compared with the traditional BiLSTM model which only relies on sequence modeling, this module can explicitly focus on keywords (such as "wire stripper"), strengthen the ability to distinguish easily confused words, and effectively reduce the interference of noise words. This mechanism is particularly suitable for complex slot identification needs in diversified natural language expressions, improving the accuracy and robustness of slot boundary.

[0141] (2)Deep integration of slot semantics and context features: By introducing a fixed semantic embedding vector E solt for each slot label, combined with the context enhancement feature h' i , the fusion of slot label prior information and dynamic context semantics is realized. Compared with slot identification models that simply rely on context, this method introduces semantic features of slot labels into the model, significantly improving the model's ability to distinguish slot types and boundaries, especially in the recognition of similar slot labels (such as B-Tool-WireStripper and B-Tool-WireCutter). This semantic consistency constraint further ensures the semantic reasonableness and accuracy of the model output.

[0142] More preferably, in an exemplary embodiment, as shown in Figure 5 , the model training step further comprises:

[0143] establishing a second mapping model, as shown in Figure 6 , the second mapping model comprises a second BiLSTM module, a third BiLSTM module, a second Softmax layer and a second CRF module; wherein the second BiLSTM module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the third BiLSTM module performs time sequence capture on the global semantic features output by the second BiLSTM module, and the second CRF module performs slot labeling on the time sequence capture data output by the third BiLSTM module; the second Softmax layer performs intent recognition on the global semantic features output by the second BiLSTM module;

[0144] training the second mapping model using the first mapping model.

[0145] Specifically, although existing deep learning models can improve the parsing effect, their high computational resource requirements limit their application in low computing power environments. Therefore, in the present exemplary embodiment, a method combining natural language processing and operation instruction mapping is provided, which can efficiently and accurately convert complex natural language instructions into operation instruction codes executable by robots through multi-task knowledge distillation technology while saving computing resources, thereby improving the autonomy and operation reliability of live-line work robots.

[0146] In the present exemplary embodiment, the first mapping model is based on the framework of BERT combined with BiLSTM and CRF. Word embedding is performed through the BERT model, the temporal relationship in natural language is captured by combining BiLSTM, and slot filling is performed by using the CRF layer, so as to accurately parse the intent and slot in the natural language instruction. The second mapping model is a lightweight BiLSTM-CRF architecture, which is responsible for receiving the distilled knowledge from the first mapping model, while reducing the model parameters and maintaining high accuracy in identifying the intent and slot.

[0147] By guiding the lightweight second mapping model (BiLSTM+BiLSTM+CRF) to learn through the first mapping model (BERT+BiLSTM+CRF), the second mapping model can achieve high precision close to the teacher model in intent recognition and slot filling tasks while reducing the consumption of computing resources.

[0148] In addition, it should be noted that, in order to make the learning process of the second mapping model more accurate, the second mapping model can use the same preferred exemplary embodiments as the first mapping model (such as the action word attention module, the dynamic context enhancement module, and the semantic consistency constraint module).

[0149] More preferably, in an exemplary embodiment, as shown in Figure 7 training the second mapping model using the first mapping model includes:

[0150] Using the cross-entropy loss function, the hard label loss calculation is performed on the slot label and intent recognition output by the second mapping module:

[0151]

[0152] where y is the true label, is the model prediction value, and w(j) is the class weight to balance the imbalance of different classes in the data;

[0153] Using the mean square error, the soft label loss calculation is performed on the probability distribution output by the first mapping model and the probability distribution output by the second mapping model:

[0154] L distill= ||z (t) -z (S )|

[0155] wherein z (t) is the probability distribution output by the first mapping model, i.e., including the slot labeling distribution and the intent recognition distribution output by the first mapping model, z (S) is the output probability distribution of the second mapping model, i.e., the slot labeling distribution and the intent recognition distribution output by the second mapping model;

[0156] The second mapping model is trained using the hard label loss and the soft label loss.

[0157] Specifically, in the present exemplary embodiment, during the distillation training process, soft labels and hard labels are generated from the first mapping model. The soft labels are the probability distribution output by the first mapping model, containing rich information between classes; while the hard labels are specific class labels. This dual labeling information provides comprehensive learning guidance for the second mapping model, enabling it to retain the resolving ability of the first mapping model while adapting to resource-constrained environments. Specifically, the soft labels are obtained through the Softmax output distribution of the first mapping model, containing relevant information between classes, which is missing in traditional one-hot labels. By minimizing the soft target loss, the second mapping model can learn the "softening knowledge" in the first mapping model, thereby performing more accurately and flexibly in the process of resolving natural language instructions.

[0158] Specifically, using a multi-task distillation strategy, the soft labels (soft targets) and hard labels (hard targets) generated by the first mapping model are combined to guide the learning of the second mapping model. Through the cross-entropy loss function and the mean square error loss function, the distillation loss of intent recognition and slot filling is calculated, so that the output distribution of the second mapping model approximates the distribution of the first mapping model as much as possible. In the distillation training, the weight parameters are balanced to ensure that the second mapping model still has high resolving accuracy in a multi-task environment.

[0159] More preferably, in an exemplary embodiment, during the calculation of the soft label loss, a temperature parameter T is introduced to adjust the output distribution of the Softmax layer:

[0160]

[0161] When T>1, the output distribution is smoother and the information between classes is richer.

[0162] Specifically, in the present exemplary embodiment, to enhance the influence of soft targets in training, a temperature parameter T is introduced to adjust the Softmax output distribution. When T>1, the output distribution is smoother and the information between classes is richer, which is conducive to the second mapping model extracting knowledge from the first mapping model during training.

[0163] More preferably, in an exemplary embodiment, as shown in Figure 8 The training of the second mapping model using the first mapping model (including the preferred schemes of Figure 3 and Figure 4 may be selected according to actual conditions), includes:

[0164] using mean square error, calculating low-level loss of global semantic features output by the BERT module of the first mapping model and global semantic features output by the second BiLSTM module of the second mapping model;

[0165] using mean square error, calculating middle-level loss of slot distribution output by the first CRF module of the first mapping model and slot distribution output by the second CRF module of the second mapping model;

[0166] The first mapping model further includes a third Softmax layer located after the first CRF module, and the first mapping model further includes a fourth Softmax layer located after the second CRF module; using KL divergence loss function, calculating high-level loss of intent distribution output by the first Softmax layer of the first mapping model and intent distribution output by the second Softmax layer of the second mapping model, slot distribution output by the third Softmax layer of the first mapping model and slot distribution output by the fourth Softmax layer of the second mapping model;

[0167] training the second mapping model using the low-level loss calculation, the middle-level loss calculation and the high-level loss calculation.

[0168] Specifically, Figure 7The training of the second mapping model using the first mapping model also has certain framework defects: (1) Limitations of single-layer knowledge transfer: traditional distillation methods directly transfer the overall knowledge of the first mapping model to the student model, but ignore the hierarchical nature of knowledge (such as contextual dynamic representation, task distribution, slot semantics, etc.). In the natural language instruction parsing scene of the live-line working robot, it may cause the second mapping model to have a decline in accuracy when parsing complex instructions. (2) Insufficient adaptability to complex instructions: the live-line working robot needs to parse complex and diverse natural language instructions, and existing methods lack robustness in parsing multi-slot and long instructions, especially in slot filling tasks, with limited boundary recognition performance. (3) Insufficient extraction of domain-specific knowledge: the first mapping model generates feature information with implicit industry rules in a specific domain (such as the live-line working scene), but existing methods do not make full use of these specific knowledge, making it difficult for the second mapping model to quickly adapt to professional scenes.

[0169] Therefore, in the present exemplary embodiment, for the natural language instruction parsing task of the live-line working robot, the low, medium, and high-level features of the teacher model are extracted layer by layer to guide the learning of the student model, and the adaptability of the student model to complex tasks and domain-specific scenes is strengthened. Specifically:

[0170] (1) Low-level features: contextual dynamic representation

[0171] The contextual dynamic features generated by the BiLSTM module of the first mapping model contain the temporal relationships of actions, positions, tools, and other vocabulary in natural language instructions. By distilling these low-level features, the second mapping model is guided to more accurately capture the basic semantic associations in the instructions. For example, the contextual dynamics of "grabbing a spiral wire clip" can help the model distinguish between "grabbing" and "peeling open".

[0172] L low =MSE(H teacher ,H student )

[0173] where H teacher and H student are the contextual dynamic feature representations of the first mapping model and the second mapping model, respectively.

[0174] (2) Medium-level features: semantic-slot distribution

[0175] The slot distribution information generated by the CRF layer of the first mapping model contains explicit associations between slots and actions in natural language instructions (such as "wire stripper" and tool slot, "main line" and wire slot). By learning these features, the second mapping model can significantly improve the accuracy of parsing slots in live-line working instructions. Introduce the semantic embedding matrix E to calculate the consistency loss of slot distribution:

[0176] L mid =MSE(E teacher ,E student )

[0177] (3) High-level features: intent-slot collaborative distribution

[0178] The softmax layer of the first mapping model outputs a co-distribution of intent and slot information, encompassing the overall task intent of the natural language instruction (e.g., "grab the tool" and "strip the wire"). By learning these distributions, the second mapping model demonstrates greater robustness when parsing multiple intents or long instructions.

[0179]

[0180] Where P( i ) is the probability distribution of the i-th task, α i is the task weight.

[0181] (4) Total loss

[0182] L total =w1L low +w2L mid +w3L high

[0183] In summary, the advantages of this exemplary embodiment are: (1) Targeted improvement of hierarchical knowledge distillation: Contribution to natural language instructions for live working: The timing relationship, slot association and task distribution in the natural language instructions are extracted layer by layer, so that the second mapping model can accurately parse complex instructions and avoid the performance degradation caused by simplifying the model structure. (2) Improved adaptability of domain-specific tasks: Help for live working scenarios: Through the slot distribution learning at the middle level and the intention collaborative distribution modeling at the high level, the second mapping model's ability to parse domain-specific slots and instruction combinations is enhanced. For example, when processing "installing the trench clamp to the mid-phase branch line position", the operating parameters (tools, position, wire type, etc.) can be matched more accurately. (3) Balance between lightweight and precise parsing: Applicability to low-computing power devices: The distillation process effectively compresses the model scale, so that the second mapping model can also be efficiently deployed on embedded devices while maintaining high-precision instruction parsing capabilities.

[0184] In addition, it should be noted that when using the mean square error to calculate the low-level loss of the global semantic features output by the BERT module of the first mapping model and the global semantic features output by the second BiLSTM module of the second mapping model, since the feature dimension output by the BERT module of the first mapping model is higher, and the feature dimension output by the second BiLSTM module of the second mapping model is lower, before calculating the loss, the output features of the two can be preferably aligned in dimension by linear transformation, so that they have the same dimension, and effective loss calculation can be ensured.

[0185] More preferably, in an exemplary embodiment, the model using step (natural language parsing and operation instruction generation) can specifically include:

[0186] (1) Parsing process

[0187] The user inputs a natural language instruction through voice or text. For voice input, the system first converts the voice into text using a voice recognition tool (existing tool), and then parses the instruction text word by word through the first mapping model (BERT+BiLSTM+CRF) or preferably through the lightweight second mapping model (BiLSTM+BiLSTM+CRF) to identify the intent and slot of the instruction.

[0188] (2) Intent and slot mapping rules

[0189] After parsing the intent and slot information, the system maps the natural language instruction to a specific operation instruction code according to the pre-set intent and slot mapping rules. Table 3 shows the mapping rules of different intents and corresponding slots, covering common operation types, tools, wire types, positions, and other parameters. The mapping rules ensure that each parsed intent and slot can be accurately matched to the parameter configuration of the operation instruction code.

[0190]

[0191]

[0192]

[0193] Table 3 Intent and slot mapping rules

[0194] (3) Operation instruction generation

[0195] After the intent and slot mapping, the system converts the mapping results into operation instruction codes executable by the live-line working robot, ensuring that the instructions can directly control the execution behavior of the robot. Table 4 shows the operation instruction code format of the live-line working robot, including commonly used operations such as grabbing, stripping, and installing, etc. Each operation instruction code consists of a function name and specific parameters, and is suitable for the robot to perform various live-line working tasks.

[0196]

[0197]

[0198] Table 4: Operation instruction code table of live-line working robot

[0199] More preferably, in an exemplary embodiment, for the model using step, it further includes operation confirmation and user feedback:

[0200] To ensure the accuracy of the instructions and the safety of the operation, the present exemplary embodiment provides a user feedback confirmation and real-time monitoring mechanism before generating the operation instruction code and sending it to the robot for execution. Figure 9 The operation confirmation and user feedback process of the present exemplary embodiment is shown, which ensures the accuracy and safety of the operation through user confirmation and real-time monitoring mechanism during instruction execution. The specific steps are as follows:

[0201] (1) Feedback confirmation

[0202] Before generating the operation instruction code, the system feeds back the parsed instruction content to the user through voice or text, ensuring that the parsing result meets the user's intent. For example, when the user inputs "install and install the line clamp at the middle phase branch line position", the system will generate feedback information: "confirm installation and installation of line clamp at middle phase branch line position?" and wait for user confirmation. If the user confirms that it is correct, the system will continue to perform the subsequent steps; if the user has objections, the system will allow the user to modify the instructions.

[0203] (2) Execution operation and real-time monitoring

[0204] After the user confirms, the system generates the corresponding operation instruction code and sends it to the live-line working robot to start execution. During the entire execution process, the system monitors the progress of the operation in real time through sensors, ensuring that each step of the operation is performed under safe and accurate conditions. If an abnormal situation is detected during the operation, the system will immediately pause the operation and prompt the user to reconfirm or correct the instructions to ensure the safety of the operation.

[0205] That is, before performing the operation, the user confirms the parsed instruction through voice or text feedback, ensuring the accuracy of each instruction for the user. If the user confirms the accuracy, the system generates an operation instruction code and sends it to the robot for execution. During the execution of the instruction, the system performs real-time monitoring and confirms the success or failure of each operation step according to the sensor feedback. If an abnormal situation occurs, the system will promptly notify the user and make adjustments according to the situation to ensure the safety and accuracy of the operation in the live-line working environment. If the user inputs an instruction unrelated to live-line work (such as weather query), the system will prompt the user to input a valid operation instruction through voice or text feedback, thereby avoiding misoperation and ensuring the safety and reliability of the system in operation.

[0206] With the same inventive concept as the above exemplary embodiments, another exemplary embodiment of the present application provides an electronic device comprising a storage unit and a processing unit, the storage unit having stored thereon computer instructions executable on the processing unit, the processing unit executing the computer instructions to perform the steps of the method according to the first aspect.

[0207] The electronic device is in the form of a general-purpose computing device. The components of the electronic device can include, but are not limited to, the at least one processing unit described above, the at least one storage unit described above, and a bus connecting different system components, including the storage unit and the processing unit.

[0208] The storage unit stores program code that can be executed by the processing unit, so that the processing unit performs the steps described in the above "Exemplary Method" section of the present description according to various exemplary embodiments of the present application. For example, the processing unit can execute the method as shown in Figure 1

[0209] The storage unit can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0210] The storage unit can also include programs / utilities with a set of (at least one) program modules, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0211] The bus can be one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0212] ​The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. The electronic device can communicate with one or more devices that enable a user to interact with it through the I / O interface. The electronic device can also communicate with one or more devices or networks (e.g., a local area network (LAN), a wide area network (WAN), or the Internet) through a network adapter. The network adapter can communicate with the other modules of the electronic device through the bus. It should be appreciated that other hardware and / or software modules can be used in conjunction with the electronic device. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0213] Through the above description, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the example embodiments can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the example embodiments.

[0214] According to the same inventive concept as the above example embodiments, another example embodiment of the present application provides a storage medium having stored thereon computer instructions, which, when executed, perform the steps of the method according to the first aspect.

[0215] Based on this understanding, the technical solution of the present embodiment, or the part of the technical solution that essentially contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product (program product). The computer software product is stored in a storage medium and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application.

[0216] The program product can employ any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0217] The computer-readable signal medium can include a computer-readable storage medium that is configured to store and deliver a computer-readable program code. The computer-readable program code can be propagated as a computer-readable signal medium.

[0218] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0219] The program code can be executed by one or more programmable processors, which can be implemented in one or more computer devices including any combination of hardware, software, firmware, and / or virtual implementation.

[0220] It should be apparent that the foregoing examples are merely illustrative of the embodiments and should not be considered limiting in any way. Numerous modifications and variations reflecting combinations of features from the above-described examples can be made by those skilled in the art without departing from the scope of the present application. The embodiments are not limited to the specific examples described in this document, but include any alternatives falling within the scope of the appended claims.

Claims

1. A method for parsing natural language instructions and mapping operations for a live working robot, characterized by: It includes model training steps and model usage steps. The model training steps include: Collect natural language instruction data and perform BIO slot labeling and intent labeling; Establish a first mapping model, which includes a BERT module, a first BiLSTM module, a first Softmax layer, and a first CRF module; wherein the BERT module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the first BiLSTM module performs time series capture on the global semantic features output by the BERT module, and the first CRF module performs slot labeling on the time series capture data output by the first BiLSTM module; and the first Softmax layer performs intent recognition on the global semantic features output by the BERT module; Training the first mapping module, including: calculating slot loss using slot labels output by the first CRF module, calculating intent loss using intent recognition output by the first Softmax layer, and training the first mapping module using the slot loss and intent loss; The steps to use the model include: Use the trained mapping model to identify the intent and slot of the input natural language command; Map the identified intent and slot into specific operation instructions according to the mapping rules; Forwarding operation instructions to the live working robot; The collecting of natural language instruction data includes: collecting natural language instruction data, including operation scenarios performed by the live working robot; generating natural language instruction data in different expressions using an AI model; The BIO slot marking is word-by-word marking, and the intention marking is sentence-by-sentence marking; The BERT module and the Softmax layer include: Action word feature extraction: Use part-of-speech tagging tools to tag action words in the training data and generate position information vectors of action words , the position information of the action word is represented by vector ,in Indicates that the word is an action word, otherwise it is 0; Self-attention mechanism enhancement: Design an action word attention weight formula: ； is the self-attention score, used to model contextual semantics; is the saliency vector of the action word; is a hyperparameter that adjusts the weight significance of action words; Fusion of local and global information: The enhanced action word features are fused with the global semantic vector representation output by the BERT module to generate the final feature vector for intent classification: ； in, Is a learnable weight matrix; natural language instructions are global semantic features output by the BERT module ,in It is Context vector representation of each word; Using the enhanced feature vector Perform intent classification: ； in, is the weight matrix for intent classification, It is the bias vector of intent classification, which is used to adjust the mapping result; A context enhancement module and a semantic consistency constraint module are also included between the first BiLSTM module and the first CRF module; The context enhancement module includes: Dynamically extract the contextual features of slot keywords and assign weights to each word by introducing an attention mechanism : ； in, is the output of the first BiLSTM module, and is a learnable parameter; Using attention weights Reweight the context features to generate enhanced features: ； in, It is a weighted aggregation of contextual features that can enhance the semantic expression of keywords; The semantic consistency constraint module includes: Define slot embedding matrix: Construct a slot embedding matrix ,in is the number of slot labels, is the embedding dimension; each slot label Corresponding to an embedding vector , represents the semantic features of the slot type; Fusion of context and slot semantic information: The input vector of the first CRF module is composed of context features and slot embedding Adding them together we get: ； in Indicates the index of the current slot label.

2. The method for natural language instruction parsing and operation mapping of a live working robot according to claim 1, characterized in that: The model training step further includes: Establish a second mapping model, the second mapping model including a second BiLSTM module, a third BiLSTM module, a second Softmax layer, and a second CRF module; wherein the second BiLSTM module performs word embedding on each word of the input natural language instruction to generate context-related global semantic features; the third BiLSTM module performs time series capture on the global semantic features output by the second BiLSTM module, and the second CRF module performs slot labeling on the time series capture data output by the third BiLSTM module; and the second Softmax layer performs intent recognition on the global semantic features output by the second BiLSTM module; The second mapping model is trained using the first mapping model.

3. The method for natural language instruction parsing and operation mapping of a live working robot according to claim 2, characterized in that: The method of training the second mapping model by using the first mapping model includes: Using the cross entropy loss function, the hard label loss is calculated for the slot annotation and intent recognition output by the second mapping module: ； in, is the true label, is the model prediction value, is the class weight to balance the imbalance of different classes in the data; Using mean square error, the probability distribution output by the first mapping model and the probability distribution output by the second mapping model are calculated. Soft label loss calculation: ； in, is the probability distribution output by the first mapping model, including the slot labeling distribution and intent recognition distribution output by the first mapping model. is the output probability distribution of the second mapping model, the slot labeling distribution and intent recognition distribution output by the second mapping model; The second mapping model is trained using hard label loss and soft label loss.

4. The method for natural language instruction parsing and operation mapping of a live working robot according to claim 3, characterized in that: In the process of calculating the soft label loss, the temperature parameter is introduced To adjust the output distribution of the Softmax layer: ； when When , the output distribution is smoother and the information between categories is richer.

5. The method for natural language instruction parsing and operation mapping of a live working robot according to claim 2, characterized in that: The method of training the second mapping model by using the first mapping model includes: Use mean square error to calculate the low-level loss of the global semantic features output by the BERT module of the first mapping model and the global semantic features output by the second BiLSTM module of the second mapping model; Using mean square error, calculate the mid-level loss for the slot distribution output by the first CRF module of the first mapping model and the slot distribution output by the second CRF module of the second mapping model; The first mapping model further includes a third Softmax layer located after the first CRF module, and the first mapping model further includes a fourth Softmax layer located after the second CRF module; using a KL divergence loss function, high-level loss calculation is performed on the intent distribution output by the first Softmax layer of the first mapping model and the intent distribution output by the second Softmax layer of the second mapping model, the slot distribution output by the third Softmax layer of the first mapping model, and the slot distribution output by the fourth Softmax layer of the second mapping model; The second mapping model is trained using the low-layer loss calculation, the middle-layer loss calculation, and the high-layer loss calculation.

6. An electronic device comprising a storage unit and a processing unit, wherein the storage unit stores computer instructions that can be executed on the processing unit, wherein: When the processing unit runs the computer instructions, the steps of the method according to any one of claims 1 to 5 are performed.

7. A storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed, the steps of the method according to any one of claims 1 to 5 are executed.

Images