Humanoid robot instruction understanding method and related device

Abstract

The application discloses a kind of humanoid robot instruction understanding method and related equipment, belong to humanoid robot technical field, including: receiving industrial operation instruction information and detecting whether it belongs to industrial scene;Input to industrial scene semantic understanding model, obtain the target operation parameter result of output;Safety verification is carried out based on industrial safety rule base, and control instruction is generated using target operation parameter result to drive humanoid robot;Wherein, industrial scene semantic understanding model is constructed based on large language model;Target operation parameter result includes operation object identification, action type, execution accuracy and safety boundary value.The application improves the correlation of scene by detecting whether industrial operation instruction information conforms to industrial scene, and improves the analysis accuracy of professional terms and operation parameters using industrial scene semantic understanding model, reduces the risk of over-threshold operation by introducing industrial safety rule base for safety verification, improves safety, thereby effectively improves the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of humanoid robots, and in particular to a humanoid robot instruction understanding method and related equipment. BACKGROUND

[0002] Currently, humanoid robots are increasingly widely used in industrial scenarios such as manufacturing workshops, warehouse logistics, and power maintenance. Operators often issue assembly, handling, maintenance, and operation industrial instructions to humanoid robots through natural language. Therefore, accurate understanding of industrial instructions by robots is the core of ensuring operation efficiency and safety.

[0003] In related technologies, a general semantic model is usually used to implement instruction understanding of humanoid robots. However, in actual applications, it is found that the industrial term recognition accuracy of the general semantic model is low, which can easily cause safety accidents.

[0004] In summary, the technical problems existing in related technologies need to be improved. SUMMARY

[0005] The embodiments of the present application provide a humanoid robot instruction understanding method and related equipment, which can improve scene relevance, improve the analysis accuracy of professional terms and operation parameters, reduce the risk of over-threshold operation, and improve safety, thereby effectively improving the user experience.

[0006] In one aspect, the embodiments of the present application provide a humanoid robot instruction understanding method, which includes the following steps:

[0007] In response to an instruction issuing request, receiving industrial operation instruction information;

[0008] Detecting whether the industrial operation instruction information belongs to a preset industrial scene;

[0009] In the case where the industrial operation instruction information belongs to a preset industrial scene, inputting the industrial operation instruction information into an industrial scene semantic understanding model to obtain a target operation parameter result output by the industrial scene semantic understanding model;

[0010] Based on a preset industrial safety rule library, performing safety verification on the target operation parameter result;

[0011] In the case where the target operation parameter result passes the safety verification, generating a control instruction using the target operation parameter result and driving a humanoid robot to execute the control instruction;

[0012] The industrial scene semantic understanding model is constructed based on a large language model; and the target operation parameter result includes an operation object identifier, an action type, an execution accuracy, and a safety boundary value.

[0013] Optionally, the detection of whether the industrial operation instruction information belongs to a preset industrial scene comprises:

[0014] The industrial operation instruction information is subjected to word segmentation processing, key words are extracted, and a key word set is formed;

[0015] According to the key word set and a preset industrial scene word library, a scene matching degree of the industrial operation instruction information is calculated;

[0016] In the case where the scene matching degree is greater than or equal to a preset threshold, it is determined that the industrial operation instruction information belongs to a preset industrial scene;

[0017] In the case where the scene matching degree is less than the preset threshold, it is determined that the industrial operation instruction information does not belong to a preset industrial scene.

[0018] Optionally, the safety verification of the target operation parameter result based on the preset industrial safety rule library comprises:

[0019] Based on the operation object identifier and the action type, a safety standard value interval is matched and extracted from the industrial safety rule library;

[0020] Based on the safety boundary value and the execution accuracy, the safety standard value interval is compared to determine the safety verification result of the target operation parameter result.

[0021] Optionally, the industrial scene semantic understanding model is obtained based on the following steps:

[0022] A pre-trained large language model is obtained as a basic semantic understanding model;

[0023] A plurality of historical industrial operation instruction data is obtained, and a plurality of target operation parameter results corresponding to the industrial operation instruction data are obtained;

[0024] Each of the historical industrial operation instruction data is used as a sample, and the target operation parameter result corresponding to each of the historical industrial operation instruction data is used as a sample label corresponding to the sample, to construct an industrial training data set;

[0025] Based on the industrial training data set, the basic semantic understanding model is trained using a low-rank adaptive fine-tuning technology to obtain the industrial scene semantic understanding model.

[0026] Optionally, after the control instruction is generated using the target operation parameter result and the humanoid robot is driven to execute the control instruction, the method further comprises:

[0027] Real-time operation data of the humanoid robot executing the control instruction is obtained.

[0028] based on the operation data and the target operation parameter result, calculate an operation data deviation value;

[0029] based on the operation data deviation value, generate a correction instruction, and correct the humanoid robot using the correction instruction.

[0030] Optionally, the method further comprises:

[0031] in the case where the target operation parameter result fails the safety verification, generating and uploading safety warning information, and terminating the operation process of the humanoid robot.

[0032] In another aspect, an embodiment of the present application provides a humanoid robot instruction understanding device, the device comprising:

[0033] an instruction receiving module, configured to receive industrial operation instruction information in response to an instruction issuing request;

[0034] a scene detection module, configured to detect whether the industrial operation instruction information belongs to a preset industrial scene;

[0035] a semantic understanding module, configured to, in the case where the industrial operation instruction information belongs to a preset industrial scene, input the industrial operation instruction information into an industrial scene semantic understanding model to obtain a target operation parameter result output by the industrial scene semantic understanding model;

[0036] a safety verification module, configured to perform safety verification on the target operation parameter result based on a preset industrial safety rule library;

[0037] a driving execution module, configured to, in the case where the target operation parameter result passes the safety verification, generate a control instruction using the target operation parameter result, and drive a humanoid robot to execute the control instruction;

[0038] The industrial scene semantic understanding model is constructed based on a large language model; and the target operation parameter result comprises an operation object identifier, an action type, an execution precision, and a safety boundary value.

[0039] In another aspect, an embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0040] In another aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0041] In another aspect, an embodiment of the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the method described above.

[0042] The embodiment of the present application improves the scene relevance by detecting whether the industrial operation instruction information conforms to the industrial scene, improves the analysis accuracy of professional terms and operation parameters by using the industrial scene semantic understanding model, performs safety verification by introducing an industrial safety rule library, reduces the operation risk exceeding the threshold, and improves the safety, thereby effectively improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a schematic diagram of an implementation environment of a humanoid robot instruction understanding method provided by an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a flow of a humanoid robot instruction understanding method provided by an embodiment of the present application;

[0045] Figure 3 is a schematic diagram of a flow of driving and controlling a humanoid robot provided by an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a flow of safety verification of a target operation parameter result provided by an embodiment of the present application;

[0047] Figure 5 is a schematic diagram of a flow of training an industrial scene semantic understanding model provided by an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a structure of a humanoid robot instruction understanding device provided by an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0051] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0052] The terms "at least one", "multiple", "each", "any", and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0054] In the related art, there is a cross-end development mode, that is, developing an application program based on a cross-end application framework. The cross-end application framework can support building an application program in a single code library, while running on multiple platform ends, and through sharing code and resources, the development efficiency and code reusability of the application program can be improved. However, it is found in actual application that there are differences between the application programming interfaces adopted by some platform ends and the types of application programming interfaces supported by the cross-end application framework, which makes it difficult for the application program developed based on the cross-end application framework to truly realize cross-end use, affecting the development efficiency of the application program.

[0055] For example, some platform ends may only provide synchronous application programming interfaces, supporting synchronous calls to implement processing of business data. Some application programs developed using the cross-end application framework are difficult to implement cross-end use because the cross-end application framework implements processing of business data based on asynchronous application programming interfaces. In the related art, JSI (JavaScript Interface) is used to perform compatibility processing on the calling mode of the application program developed using the cross-end application framework, which can help to realize consistency of the multi-end code of the application program and reduce the cost of developing the code of the application program. However, this mode is based on synchronous calls to read and write business data in the disk, which is easy to cause serious blocking problems. In some platform ends with poor performance or business scenarios with frequent data storage calls, application program crashes or abnormal termination often occur, resulting in poor stability of the application program and affecting the user experience.

[0056] Therefore, the embodiment of the present application provides a humanoid robot instruction understanding method and related equipment, which improves the scene relevance by detecting whether the industrial operation instruction information conforms to the industrial scene, improves the analysis accuracy of professional terms and operation parameters by using an industrial scene semantic understanding model, reduces the operation risk above the threshold by introducing an industrial safety rule library for safety verification, and improves the safety, thereby effectively improving the user experience.

[0057] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information, and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0058] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. First, a humanoid robot instruction understanding method provided in the embodiments of the present application is described with reference to the accompanying drawings.

[0059] Please refer to Figure 1 , Figure 1 is an implementation environment schematic diagram of a humanoid robot instruction understanding method provided in the embodiments of the present application. In the implementation environment, the main hardware and software subjects involved include a terminal processor 110 and a server 120.

[0060] Specifically, the terminal processor 110 can be installed with a control program of the related humanoid robot instruction understanding method, and the server 120 is a background server of the control program. The terminal processor 110 and the background server 120 are in communication connection. The humanoid robot instruction understanding method provided in the embodiments of the present application can be executed on the terminal processor 110 side.

[0061] The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0062] In addition, the server 120 can also be a node server in a blockchain network.

[0063] The terminal processor 110 and the server 120 can establish a communication connection through a wireless network. The wireless network uses standard communication technology and / or protocol, and the network can be set as the Internet, or any other network, for example, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile or wireless network, a private network or a virtual private network. In addition, the same communication connection mode or different communication connection mode can be used between the above-mentioned software and hardware subjects, and the application does not make specific limitations.

[0064] Of course, it can be understood that Figure 1 The implementation environment shown in FIG. 1 is only some optional application scenarios of the human-shaped robot instruction understanding method provided in the embodiments of the present application, and the actual application is not fixed to the implementation environment shown in FIG. 1. The application does not make specific limitations. Figure 1

[0065] As shown in FIG. 1, Figure 2 Figure 2 FIG. 2 is a flow diagram of a human-shaped robot instruction understanding method provided in an embodiment of the present application, and specifically includes but is not limited to steps 100 to 500.

[0066] Step 100, in response to an instruction issuing request, receiving industrial operation instruction information.

[0067] In the embodiments of the present application, the initial industrial operation instruction information can be obtained from the upper system or the operator. The industrial operation instruction information refers to the industrial instructions such as assembly, handling, maintenance and operation issued by the operator to the human-shaped robot through natural language.

[0068] For example, in the mechanical assembly scene, when the voice information of the operator is recognized to issue an instruction, the voice data of the operator can be obtained through a voice collection device in response to the instruction issuing request, for example, "tighten the M10 bolt on the left workbench to the flange surface with a pre-tightening force of 90N", so as to convert the voice data into text data through voice recognition, thereby obtaining the industrial operation instruction information.

[0069] Step 200, detecting whether the industrial operation instruction information belongs to a preset industrial scene.

[0070] ​​In the embodiments of the present application, when the industrial operation instruction information is acquired, it can be further judged whether the industrial operation instruction information belongs to a preset industrial scene (for example, mechanical assembly, material handling, equipment maintenance, precision debugging, heavy equipment operation, etc.), so as to effectively filter the industrial operation instruction information irrelevant to the preset industrial scene, reduce resource waste, and improve scene relevance.

[0071] As an optional implementation, the detection of whether the industrial operation instruction information belongs to a preset industrial scene comprises:

[0072] The industrial operation instruction information is subjected to word segmentation processing, key words are extracted, and a key word set is formed;

[0073] According to the key word set and a preset industrial scene word library, a scene matching degree of the industrial operation instruction information is calculated;

[0074] In a case where the scene matching degree is greater than or equal to a preset threshold, it is determined that the industrial operation instruction information belongs to a preset industrial scene;

[0075] In a case where the scene matching degree is less than the preset threshold, it is determined that the industrial operation instruction information does not belong to a preset industrial scene.

[0076] In the embodiments of the present application, please refer to Figure 3 , Figure 3 is a flowchart of driving control of a humanoid robot provided in the embodiments of the present application. By subjecting the original industrial operation instruction information to word segmentation processing, key words such as device name, work art language, and operation verb are extracted, a key word set is formed, and then the key word set is matched with a preset industrial scene word library, so as to calculate a scene matching degree. The scene matching degree=(number of successfully matched key words / total number of key words)×100%.

[0077] Further, when the scene matching degree is greater than or equal to a preset threshold (for example, 70%), it can be determined that the key word set has a high repetition of key words related to the industrial field, and then it can be determined that the industrial operation instruction information belongs to a preset industrial scene, and a subsequent semantic understanding process can be further performed; when the scene matching degree is less than the preset threshold, it can be determined that the industrial operation instruction information does not belong to a preset industrial scene, and then the current detection process can be ended, and the next received industrial operation instruction information is waited for. The preset threshold can be set according to the specific application scene requirement, which is not limited in the present application.

[0078] Step 300, in a case where the industrial operation instruction information belongs to a preset industrial scene, the industrial operation instruction information is input to an industrial scene semantic understanding model, so as to acquire a target operation parameter result output by the industrial scene semantic understanding model;

[0079] The industrial scene semantic understanding model is constructed based on a large language model; and the target operation parameter result includes an operation object identifier, an action type, an execution accuracy, and a safety boundary value.

[0080] In the embodiments of the present application, considering that in the prior art, the instruction understanding of a humanoid robot depends on a general semantic model, and the professional nature of an industrial scene is not optimized, there are defects such as low accuracy of industrial term recognition, the present application optimizes and adapts to an industrial scene based on a large language model, improves the analysis accuracy of professional terms and operation parameters, constructs an industrial scene semantic understanding model, and can accurately analyze the target operation parameter result including the operation object identifier, the action type, the execution accuracy, and the safety boundary value from the industrial operation instruction information belonging to the preset industrial scene, so as to facilitate subsequent safety verification.

[0081] For example, the above industrial operation instruction information "tighten the M10 bolt on the left work station to the flange surface with a pre-tightening force of 90N" is input into the industrial scene semantic understanding model, and the target operation parameter output by the industrial scene semantic understanding model is obtained: operation object identifier: "M10 bolt", "flange surface"; action type: "rotary tightening"; execution accuracy: "coaxiality ≤ 0.1mm"; safety boundary value: "pre-tightening force 90N±5N", as the target operation parameter result.

[0082] Step 400, based on the preset industrial safety rule library, safety verification is performed on the target operation parameter result.

[0083] In the embodiments of the present application, as shown in Figure 3 When the target operation parameter result output by the industrial scene semantic understanding model is obtained, the target operation parameter result can be safety verified based on the pre-constructed industrial safety rule library, so as to determine whether the target operation parameter result is a threshold risk operation, thereby effectively improving the safety.

[0084] Specifically, as an optional implementation, the safety verification of the target operation parameter result based on the preset industrial safety rule library includes:

[0085] Based on the operation object identifier and the action type, a safety standard value interval is matched and extracted from the industrial safety rule library;

[0086] Based on the safety boundary value and the execution accuracy, the safety standard value interval is compared to determine the safety verification result of the target operation parameter result.

[0087] In the embodiments of the present application, as shown in Figure 3As shown, based on the operation object identifier and action type, a range of safety standard values ​​is retrieved from the industrial safety rule base. The range of safety standard values ​​can be preset based on experimental data when the industrial safety rule base is built, or it can be updated and adjusted in real time according to the working conditions in actual application.

[0088] Based on the safety boundary value and execution accuracy, the results are compared with the safety standard value range to obtain the safety verification results of the target operation parameters.

[0089] For example, taking the industrial operation instruction information of "tightening the M10 bolt on the left workstation to the flange face with a preload of 90N" as an example, according to the operation object identifier: "M10 bolt", "flange face", action type: "rotation tightening", the safety standard value range of "preload range 70-100N, speed ≤30r / min" is obtained by matching and extracting from the industrial safety rule base. Thus, by comparing the safety boundary value and the execution accuracy, it can be determined that it meets the safety standard value range, and then the safety verification result of the target operation parameter result is determined to be passed, which effectively improves the safety of the humanoid robot performing the operation.

[0090] Furthermore, such as Figure 3 As shown, if the target operation parameter result fails the security verification, a security warning message is generated and uploaded, and the operation process of the humanoid robot is terminated.

[0091] It is understandable that when the safety boundary value and execution accuracy in the target operation parameter result do not meet the range of the matched safety standard value, it can be determined that the target operation parameter result has not passed the safety verification. In this case, the current operation process of the humanoid robot can be terminated, and a safety warning message can be generated and sent to the relevant operators.

[0092] For example, the industrial safety rule base can be pre-constructed as shown in Table 1 below, and stored using a triplet structure (operation object, action type, safety threshold). In practical applications, relevant safety thresholds can be quickly retrieved and matched in the industrial safety rule base using parameters such as operation object identifier and action type, serving as a range of safety standard values. It is understood that Table 1 below only shows some available safety rules, and actual applications are not fixed to those shown in Table 1; this application does not impose specific limitations on this.

[0093] Table 1 Industrial Safety Rule Base

[0094]

[0095] Step 500: If the target operation parameter result passes the security verification, generate control instructions using the target operation parameter result and drive the humanoid robot to execute the control instructions.

[0096] In the embodiment of the present application, as shown in Figure 3 If the target operation parameter result passes the security verification, the control instruction of the humanoid robot can be generated according to the target operation parameter result, and the corresponding action of the humanoid robot can be driven by using the control instruction, so as to complete the corresponding industrial task operation.

[0097] Therefore, the present application improves the scene relevance by detecting whether the industrial operation instruction information conforms to the industrial scene, improves the analysis accuracy of professional terms and operation parameters by using the industrial scene semantic understanding model, reduces the operation risk above the threshold by introducing the industrial safety rule library for security verification, and improves the safety, thereby effectively improving the user experience.

[0098] Specifically, as an optional implementation, after the control instruction is generated by using the target operation parameter result and the humanoid robot executes the control instruction, the method further includes:

[0099] Real-time acquisition of operation data of the humanoid robot executing the control instruction;

[0100] Calculation of an operation data deviation value based on the operation data and the target operation parameter result;

[0101] Generation of a correction instruction based on the operation data deviation value, and correction of the humanoid robot by using the correction instruction.

[0102] In the embodiment of the present application, please refer to Figure 4 , Figure 4 is a flowchart of the security verification of the target operation parameter result provided by the embodiment of the present application. After the security verification and the generation of the control instruction by using the target operation parameter result to drive the humanoid robot to execute the control instruction, the operation data can be collected in real time by using the end sensor of the humanoid robot, and the deviation value can be calculated according to the operation data and the target operation parameter result, so as to generate the correction instruction to correct the operation executed by the humanoid robot.

[0103] Optionally, as shown in Figure 4 , the calculated deviation value can also be compared with a preset deviation threshold value. When the deviation value is less than or equal to the preset deviation threshold value, the humanoid robot continues to execute the current control instruction. When the deviation value is greater than the preset deviation threshold value, the correction instruction is generated according to the deviation value, and the humanoid robot is corrected by using the correction instruction.

[0104] Optionally, as an optional implementation, the industrial scene semantic understanding model is obtained by training based on the following steps:

[0105] An industrial scene semantic understanding model is obtained by training based on the following steps:

[0106] a plurality of historical industrial operation instruction data, and a plurality of target operation parameter results corresponding to the industrial operation instruction data;

[0107] Each of the historical industrial operation instruction data is taken as a sample, and the target operation parameter result corresponding to each of the historical industrial operation instruction data is taken as a sample label corresponding to the sample, to construct an industrial training data set.

[0108] Based on the industrial training data set, the low-rank adaptation fine-tuning technology is used to train the basic semantic understanding model, to obtain the industrial scene semantic understanding model.

[0109] In the embodiments of the present application, please refer to Figure 5 , Figure 5 is a flowchart of training an industrial scene semantic understanding model provided by the embodiments of the present application. First, a plurality of historical industrial operation instruction data can be collected, and FPGA is used to realize high-speed word segmentation as a sample. The operation object identifier, action type, execution accuracy and safety boundary value corresponding to each sample are labeled as sample labels, so as to construct an industrial training data set.

[0110] Further, a pre-trained large language model is obtained as a basic semantic understanding model and is initialized. The low-rank adaptation fine-tuning technology (Low-Rank Adaptation, LoRA) is used, that is, 90% of the model parameters in the basic semantic understanding model are frozen, and only the remaining 10% (usually the higher layer semantic representation layer) is open for fine-tuning, so as to retain the ability of the large language model in general language understanding, and to fine-tune and adapt in the industrial field semantics, while greatly reducing the amount of model parameters that need to be updated and shortening the fine-tuning time.

[0111] Further, a loss function L=0.6xL1+0.4xL2 is designed, wherein L1 is a BERT word vector matching error, which is used to measure the understanding accuracy of the model to the "semantic unit" (such as object, action word) in the instruction, L2 is the MAE error of parameter value prediction, which is used to ensure the accuracy of the numerical prediction, and the corresponding assigned 0.6 and 0.4 weights can effectively ensure that the "semantic correctness" is slightly higher than the priority of "numerical accuracy", so as to complete the training of the basic semantic understanding model and obtain the industrial scene semantic understanding model.

[0112] Wherein, it can be considered that the training process of the industrial scene semantic understanding model is completed after reaching a preset training number; it can also be considered that the training process of the industrial scene semantic understanding model is completed when the training output result of the industrial scene semantic understanding model converges; and it can also be considered that the training process of the industrial scene semantic understanding model is completed when the accuracy of the industrial scene semantic understanding model on the test set meets a preset condition.

[0113] In actual application, after the industrial training data set is constructed, it can also be divided into a training set and a test set, and the training set accounts for 80% of the training data set and the test set accounts for 20% of the training data set. The training set is used for model training, and the test set is used for model verification.

[0114] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of a humanoid robot instruction understanding device provided by the embodiment of the present application. The embodiment of the present application also provides a humanoid robot instruction understanding device, which can implement the above-mentioned humanoid robot instruction understanding method. The device comprises:

[0115] The instruction receiving module 610 is configured to receive industrial operation instruction information in response to an instruction issuing request.

[0116] The scene detection module 620 is configured to detect whether the industrial operation instruction information belongs to a preset industrial scene.

[0117] The semantic understanding module 630 is configured to input the industrial operation instruction information to an industrial scene semantic understanding model to obtain a target operation parameter result output by the industrial scene semantic understanding model in the case where the industrial operation instruction information belongs to the preset industrial scene.

[0118] The safety verification module 640 is configured to perform safety verification on the target operation parameter result based on a preset industrial safety rule library.

[0119] The driving execution module 650 is configured to generate a control instruction by using the target operation parameter result and drive a humanoid robot to execute the control instruction in the case where the target operation parameter result passes the safety verification.

[0120] Wherein, the industrial scene semantic understanding model is constructed based on a large language model; and the target operation parameter result comprises an operation object identifier, an action type, an execution accuracy and a safety boundary value. It can be understood that the contents in the above-mentioned method embodiments are applicable to the present device embodiments. The present device embodiments specifically implement the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method embodiments.

[0121] Please refer to Figure 7 ,Figure 7 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application, and the electronic device comprises:

[0122] The processor 701 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0123] The memory 702 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 702 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the above-mentioned method of the embodiments of the present application.

[0124] The input / output interface 703 is used to realize information input and output.

[0125] The communication interface 704 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0126] The bus 705 is used to transmit information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0127] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize the communication connection between them in the device.

[0128] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned method.

[0129] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiments. The present storage medium embodiments specifically implement the same functions as the above-mentioned method embodiments, and achieve the same beneficial effects as the above-mentioned method embodiments.

[0130] The embodiment of the application further provides a computer program product comprising a computer program which, when executed by a processor, implements the method described above.

[0131] It can be understood that the contents in the method embodiments described above are applicable to the program product embodiments, the program product embodiments specifically implement the functions same as those of the method embodiments described above, and achieve the beneficial effects same as those of the method embodiments described above.

[0132] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The embodiment of the application provides a humanoid robot instruction understanding method and related equipment, which improves the scene correlation by detecting whether the industrial operation instruction information conforms to the industrial scene, improves the analysis accuracy of professional terms and operation parameters by using an industrial scene semantic understanding model, reduces the operation risk exceeding the threshold by introducing an industrial safety rule library for safety verification, and improves the safety, thereby effectively improving the user experience.

[0134] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0135] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0136] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment of the application.

[0137] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0138] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a general order and / or structure unless otherwise indicated. Furthermore, the terms "comprise", "comprising", "has", "having", "includes", "including", "contain", "containing" or any other similar forms are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains items or components does not include items or components not explicitly recited. The terms "a" or "an", as used herein in the detailed description and in the claims, mean "one or more" or "at least one", unless otherwise indicated.

[0139] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0140] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0141] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0142] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0143] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0144] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the rights of the embodiments of the present application is not limited thereto. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A humanoid robot instruction understanding method characterized by comprising: The method comprises the following steps: in response to an instruction issuing request, receiving industrial operation instruction information; detecting whether the industrial operation instruction information belongs to a preset industrial scene; in the case that the industrial operation instruction information belongs to a preset industrial scene, inputting the industrial operation instruction information into an industrial scene semantic understanding model to obtain a target operation parameter result output by the industrial scene semantic understanding model; based on a preset industrial safety rule library, performing safety verification on the target operation parameter result; in the case that the target operation parameter result passes the safety verification, generating a control instruction using the target operation parameter result and driving a humanoid robot to execute the control instruction; wherein the industrial scene semantic understanding model is constructed based on a large language model; the target operation parameter result comprises an operation object identifier, an action type, an execution accuracy and a safety boundary value.

2. The method of claim 1, wherein, The detection of whether the industrial operation instruction information belongs to a preset industrial scene comprises: performing word segmentation processing on the industrial operation instruction information, extracting keywords to form a keyword set; calculating the scene matching degree of the industrial operation instruction information according to the keyword set and a preset industrial scene word library; in the case that the scene matching degree is greater than or equal to a preset threshold, determining that the industrial operation instruction information belongs to a preset industrial scene; in the case that the scene matching degree is less than the preset threshold, determining that the industrial operation instruction information does not belong to a preset industrial scene.

3. The method of claim 1, wherein, The safety verification of the target operation parameter result based on the preset industrial safety rule library comprises: based on the operation object identifier and the action type, matching and extracting a safety standard value interval from the industrial safety rule library; based on the safety boundary value and the execution accuracy, comparing with the safety standard value interval to determine the safety verification result of the target operation parameter result.

4. The method of claim 1, wherein, The industrial scene semantic understanding model is trained based on the following steps: obtain a pre-trained large language model as a basic semantic understanding model; obtain a plurality of historical industrial operation instruction data and a plurality of target operation parameter results corresponding to the industrial operation instruction data; use each of the historical industrial operation instruction data as a sample and the target operation parameter result corresponding to each of the historical industrial operation instruction data as a sample label corresponding to the sample to construct an industrial training data set; based on the industrial training data set, use low-rank adaptive fine-tuning technology to train the basic semantic understanding model to obtain the industrial scene semantic understanding model.

5. The method of claim 1, wherein, After generating a control instruction using the target operation parameter result and driving a humanoid robot to execute the control instruction, the method further comprises: real-time acquisition of operation data of the humanoid robot executing the control instruction; based on the operation data and the target operation parameter result, calculating an operation data deviation value; based on the operation data deviation value, generating a correction instruction and correcting the humanoid robot using the correction instruction.

6. The method of claim 1, wherein, The method further comprises: In the case that the target operation parameter result fails to pass the safety verification, safety warning information is generated and uploaded, and the operation process of the humanoid robot is terminated.

7. A humanoid robot instruction understanding device characterized by comprising: The device comprises: An instruction receiving module configured to receive industrial operation instruction information in response to an instruction issuing request; A scene detecting module configured to detect whether the industrial operation instruction information belongs to a preset industrial scene; A semantic understanding module configured to input the industrial operation instruction information to an industrial scene semantic understanding model to obtain a target operation parameter result output by the industrial scene semantic understanding model in the case that the industrial operation instruction information belongs to a preset industrial scene; A safety verifying module configured to perform safety verification on the target operation parameter result based on a preset industrial safety rule library; A driving execution module configured to generate a control instruction using the target operation parameter result and drive a humanoid robot to execute the control instruction in the case that the target operation parameter result passes the safety verification. The industrial scene semantic understanding model is constructed based on a large language model; and the target operation parameter result comprises an operation object identifier, an action type, an execution accuracy, and a safety boundary value.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images