A humanoid robot multimodal training method and system based on extended reality

Through the multimodal training system of humanoid robots based on extended reality, using the multimodal training platform and information processing module to calculate the training anomaly coefficient, the problem of low accuracy of traditional training methods in complex scenarios is solved, and efficient training and Anomaly monitoring improves robot training capabilities.

CN118809634BActive Publication Date: 2024-12-03SICHUAN WUTONG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411297788.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-12-03
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

Traditional human-type robot training methods are mostly based on single sensor data judgment, with limited monitoring range and low accuracy, making it difficult to achieve efficient training in complex and changeable scenarios.

Method used

Using a human-type robot multimodal training system based on extended reality, the training abnormal coefficient is calculated through the information processing module, combined with time, movement and motion information, the classification judgment module evaluates the training scenario, and generates training qualified or unqualified instructions, and uses The multimodal training platform builds a virtual training environment to realize data acquisition and analysis.

Benefits of technology

It improves the accuracy and comprehensiveness of abnormal monitoring of training operations, enhances training efficiency and accuracy, improves the training capabilities of human-shaped robots, and provides support for education, medical care, services and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118809634B_ABST
    Figure CN118809634B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of robot technology, and in particular to a multimodal training method and system for a humanoid robot based on extended reality, which is used to solve the problem that traditional humanoid robot training methods are mostly based on single sensor data judgment, have the disadvantages of limited monitoring range and low accuracy, and are difficult to achieve efficient training in complex and changeable scenes; the present invention adopts extended reality technology to construct a multimodal virtual training environment, realizes the diversification and customization of robot training scenes, and performs data collection and analysis on the training operation process, which can comprehensively reflect the training operation status of the humanoid robot, improve the accuracy and comprehensiveness of abnormal monitoring, and can timely discover abnormal training operation situations, and realize overall evaluation of the multimodal virtual training environment, thereby improving training efficiency and accuracy, and achieving comprehensive improvement of the training ability of the humanoid robot.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a multimodal training method and system for a humanoid robot based on extended reality. Background Art

[0002] As an important branch of the robotics field, humanoid robots are highly flexible and adaptable and are widely used in industrial manufacturing, service industry, disaster relief and other fields. With the rapid development of computer graphics and simulation technology, extended reality technology has become a bridge connecting the virtual and real worlds, providing new possibilities for robot training. However, in a complex training environment, humanoid robots may face a variety of uncertainties and abnormal conditions, which not only affect the training effect, but may also cause damage to the robot itself. However, traditional humanoid robot training methods are mostly based on single sensor data judgment, which has the disadvantages of limited monitoring range and low accuracy, making it difficult to achieve efficient training in complex and changing scenarios. Summary of the invention

[0003] In order to overcome the above-mentioned technical problems, the purpose of the present invention is to provide a multimodal training method and system for humanoid robots based on extended reality, which solves the problem that traditional humanoid robot training methods are mostly based on single sensor data judgment, have the shortcomings of limited monitoring range and low accuracy, and are difficult to achieve efficient training in complex and changeable scenarios.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A multimodal training system for humanoid robots based on extended reality, comprising:

[0006] An information processing module is used to obtain a training anomaly coefficient XLi according to the training monitoring information, and send the training anomaly coefficient XLi to the classification judgment module;

[0007] The specific process of the information processing module obtaining the training abnormality coefficient XLi is as follows:

[0008] The values ​​of the time information SJ, the movement information YD and the motion information MO are quantized according to a preset information processing function to obtain a training anomaly coefficient XLi, and the training anomaly coefficient XLi is sent to the classification judgment module;

[0009] The information processing function is as follows:

[0010]

[0011] Where:

[0012] κ is the preset error adjustment factor, taking κ=0.938;

[0013] Both π and e are mathematical constants;

[0014] x1, x2 and x3 are respectively preset weight factors corresponding to the set time information SJ, movement information YD and motion information MO, x1, x2 and x3 satisfy x2>x3>x1>1.594, taking x1=1.88, x2=3.07, x3=2.41;

[0015] A classification judgment module is used to classify the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and send the training abnormal scene to the result display module, and obtain the training evaluation information, and obtain the training evaluation value XP according to the training evaluation information, and generate a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and send the training unqualified instruction or the training qualified instruction to the result display module; wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ;

[0016] The result display module is used to display according to abnormal training scenarios, unqualified training instructions and qualified training instructions.

[0017] As a further solution of the present invention: the specific process of the classification judgment module classifying the training scene i is as follows:

[0018] The training anomaly coefficient XLi is compared with the preset training anomaly threshold XLy, and the comparison results are as follows:

[0019] If the training anomaly coefficient XLi ≥ the training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a training anomaly scene, and the training anomaly scene is sent to the result display module;

[0020] If the training anomaly coefficient XLi<training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a qualified training scene.

[0021] As a further solution of the present invention: the specific process of the classification judgment module generating the training unqualified instruction or the training qualified instruction is as follows:

[0022] After the humanoid robot completes the training operation according to all the training scenarios i, the ratio of the number of abnormal training scenarios to the number of training scenarios i is obtained, and marked as an abnormal ratio value YB;

[0023] After the humanoid robot completes the training operation according to all training scenarios i, the average value of all training abnormal coefficients XLi is obtained and marked as the training mean value XJ;

[0024] Get the product of the difference ratio value YB and the training mean value XJ, and mark it as the training evaluation value XP;

[0025] The training evaluation value XP is compared with the preset training evaluation threshold XPy, and the comparison results are as follows:

[0026] If the training evaluation value XP ≥ the training evaluation threshold XPy, a training failure instruction is generated and sent to the result display module;

[0027] If the training evaluation value XP<training evaluation threshold value XPy, a training qualified instruction is generated and sent to the result display module.

[0028] As a further solution of the present invention: the humanoid robot multimodal training system based on extended reality also includes:

[0029] The scenario construction module is used to construct the training scenario i and send the training scenario i to the multimodal training platform.

[0030] As a further solution of the present invention: the specific process of the scenario construction module constructing the training scenario i is as follows:

[0031] Use extended reality technology to construct several highly realistic virtual training environments, and mark them as training scene i in sequence, where i=1, ..., n, n is a positive integer, i is the number of any virtual training environment, and n is the total number of virtual training environments. Then send the training scene i to the multimodal training platform.

[0032] As a further solution of the present invention: the humanoid robot multimodal training system based on extended reality also includes:

[0033] The multimodal training platform is used for users to select training scenarios i, generate training control instructions at the same time, and send the training control instructions to the humanoid robot.

[0034] As a further solution of the present invention: the humanoid robot multimodal training system based on extended reality also includes:

[0035] The humanoid robot is used to perform training operations according to the training scenario i after receiving the training control instructions, generate information acquisition instructions at the same time, and send the information acquisition instructions to the information acquisition module.

[0036] As a further solution of the present invention: the humanoid robot multimodal training system based on extended reality also includes:

[0037] The information acquisition module is used to obtain the training monitoring information after obtaining the information acquisition instruction, and send the training monitoring information to the information processing module; wherein the training monitoring information includes time information SJ, movement information YD and motion information MO.

[0038] As a further solution of the present invention: the specific process of the information acquisition module acquiring the training monitoring information is as follows:

[0039] After obtaining the information acquisition instruction, obtain the generation time of the training control instruction and the time when the humanoid robot receives the training control instruction, obtain the time difference between the two, and mark it as the connection time value JS, obtain the time when the humanoid robot receives the training control instruction and the time when the humanoid robot performs the training operation, obtain the time difference between the two, and mark it as the operation time value CS, quantize the connection time value JS and the operation time value CS, multiply the values ​​of the connection time value JS and the operation time value CS by their corresponding preset proportional coefficients, obtain the sum of the two, and mark it as time information SJ; wherein, the preset proportional coefficients corresponding to the connection time value JS and the operation time value CS are s1 and s2 respectively, and s1 and s2 satisfy s1+s2=1, 0<s1<s2<1, and s1=0.38 and s2=0.62;

[0040] Obtain the moving trajectory of the humanoid robot for training operation and mark it as the actual moving trajectory, obtain the difference between the length of the actual moving trajectory and the length of the preset standard moving trajectory, and mark it as the length value CD, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, obtain the non-overlapping trajectory length of the actual moving trajectory and the preset standard moving trajectory, and mark it as the non-overlapping value FC, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, connect the end points of the actual moving trajectory and the preset standard moving trajectory with a line segment, obtain the area of ​​the area enclosed between the actual moving trajectory and the preset standard moving trajectory, and mark it as the area value MJ, and connect the length value CD, the non-overlapping value FC and area value MJ are quantized, and the values ​​of length value CD, non-repeated value FC and area value MJ are respectively used as exponents of base e, and are respectively marked as length power value, non-repeated power value and area power value, and the length power value, non-repeated power value and area power value are respectively multiplied by their corresponding preset proportional coefficients to obtain the arithmetic square root of the sum of the three, and are marked as movement information YD; wherein e is a mathematical constant, and the preset proportional coefficients corresponding to the length power value, non-repeated power value and area power value are d1, d2 and d3 respectively, and d1, d2 and d3 satisfy d1+d2+d3=1, 0<d1<d2<d3<1, and d1=0.21, d2=0.35, d3=0.44;

[0041] Obtain all preset monitoring joints on the humanoid robot, and mark them as monitoring objects j in sequence, where j=1, ..., m, where m is a positive integer, j is the number of any monitoring joint, and m is the total number of monitoring joints; obtain the motion trajectory of the monitoring object j for training operation, and mark it as the actual motion trajectory YGj; obtain the difference between the length of the actual motion trajectory YGj and the length of the preset standard motion trajectory YGb, and mark it as the length difference CCi; obtain the maximum length difference CCi, and mark it as the longest value ZC; obtain the sum of all the length difference CCi, and mark it as the total length value TC; obtain the product of the longest value ZC and the total length value TC, and mark it as motion information MO;

[0042] The time information SJ, movement information YD and motion information MO are sent to the information processing module.

[0043] As a further solution of the present invention: a multimodal training method for a humanoid robot based on extended reality comprises the following steps:

[0044] Step 1: The scenario construction module constructs the training scenario i and sends the training scenario i to the multimodal training platform;

[0045] Step 2: The user selects a training scenario i using the multimodal training platform, generates a training control instruction, and sends the training control instruction to the humanoid robot;

[0046] Step 3: After receiving the training control instruction, the humanoid robot performs training operations according to the training scenario i, generates an information acquisition instruction, and sends the information acquisition instruction to the information acquisition module;

[0047] Step 4: The information acquisition module acquires the training monitoring information after acquiring the information acquisition instruction, wherein the training monitoring information includes time information SJ, movement information YD and motion information MO, and sends the training monitoring information to the information processing module;

[0048] Step 5: The information processing module obtains the training abnormality coefficient XLi according to the training monitoring information, and sends the training abnormality coefficient XLi to the classification judgment module;

[0049] Step 6: The classification judgment module classifies the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and sends the training abnormal scene to the result display module, and obtains the training evaluation information, wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ, and obtains the training evaluation value XP according to the training evaluation information, and generates a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and sends the training unqualified instruction or the training qualified instruction to the result display module;

[0050] Step 7: The result display module displays the abnormal training scenario, unqualified training instructions, and qualified training instructions.

[0051] Beneficial effects of the present invention:

[0052] The present invention discloses a multimodal training method and system for a humanoid robot based on extended reality. The training operation is performed according to the training scene, and data is collected and analyzed during the training operation process to obtain training monitoring information. The training abnormality coefficient obtained according to the training monitoring information can comprehensively measure the abnormality degree of the training operation process, and the larger the training abnormality coefficient, the higher the abnormality degree. The training scenes with abnormal training operation are then screened out, and then all the training scenes are evaluated as a whole to obtain a training evaluation value. The training evaluation value can comprehensively measure the overall abnormality degree of the training scene, and the larger the training evaluation value, the higher the overall abnormality degree. Finally, the result is displayed.

[0053] The present invention adopts extended reality technology to construct a multimodal virtual training environment, realizes the diversification and customization of robot training scenarios, and collects and analyzes data of the training operation process, which can comprehensively reflect the training operation status of the humanoid robot, improve the accuracy and comprehensiveness of abnormal monitoring, and can timely discover abnormal training operations, and realize overall evaluation of the multimodal virtual training environment, thereby improving training efficiency and accuracy, realizing a comprehensive improvement in the training capabilities of humanoid robots, and providing strong support for the wide application of robots in education, medical care, and service fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described below in conjunction with the accompanying drawings.

[0055] Figure 1 It is a principle block diagram of a multimodal training system for humanoid robots based on extended reality in the present invention;

[0056] Figure 2 It is a flow chart of a multimodal training method for a humanoid robot based on extended reality in the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Embodiment 1:

[0059] See also Figure 1As shown, this embodiment is a humanoid robot multimodal training system based on extended reality, including: a scene construction module, a multimodal training platform, a humanoid robot, an information acquisition module, an information processing module, a classification judgment module and a result display module;

[0060] The scenario construction module is used to construct a training scenario i and send the training scenario i to the multimodal training platform;

[0061] Wherein, the multimodal training platform is used for the user to select a training scenario i, and at the same time generates a training control instruction, and sends the training control instruction to the humanoid robot;

[0062] The humanoid robot is used to perform training operations according to the training scenario i after receiving the training control instruction, and generate an information acquisition instruction at the same time, and send the information acquisition instruction to the information acquisition module;

[0063] The information acquisition module is used to acquire training monitoring information after acquiring the information acquisition instruction, and send the training monitoring information to the information processing module; wherein the training monitoring information includes time information SJ, movement information YD and motion information MO;

[0064] The information processing module is used to obtain the training abnormality coefficient XLi according to the training monitoring information, and send the training abnormality coefficient XLi to the classification judgment module;

[0065] The classification judgment module is used to classify the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and send the training abnormal scene to the result display module, and obtain the training evaluation information, and obtain the training evaluation value XP according to the training evaluation information, and generate a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and send the training unqualified instruction or the training qualified instruction to the result display module; wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ;

[0066] The result display module is used to display according to abnormal training scenarios, unqualified training instructions and qualified training instructions.

[0067] Embodiment 2:

[0068] See also Figure 2 As shown, this embodiment is a multimodal training method for a humanoid robot based on extended reality, comprising the following steps:

[0069] Step 1: The scenario construction module constructs the training scenario i and sends the training scenario i to the multimodal training platform;

[0070] Step 2: The user selects a training scenario i using the multimodal training platform, generates a training control instruction, and sends the training control instruction to the humanoid robot;

[0071] Step 3: After receiving the training control instruction, the humanoid robot performs training operations according to the training scenario i, generates an information acquisition instruction, and sends the information acquisition instruction to the information acquisition module;

[0072] Step 4: The information acquisition module acquires the training monitoring information after acquiring the information acquisition instruction, wherein the training monitoring information includes time information SJ, movement information YD and motion information MO, and sends the training monitoring information to the information processing module;

[0073] Step 5: The information processing module obtains the training abnormality coefficient XLi according to the training monitoring information, and sends the training abnormality coefficient XLi to the classification judgment module;

[0074] Step 6: The classification judgment module classifies the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and sends the training abnormal scene to the result display module, and obtains the training evaluation information, wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ, and obtains the training evaluation value XP according to the training evaluation information, and generates a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and sends the training unqualified instruction or the training qualified instruction to the result display module;

[0075] Step 7: The result display module displays the abnormal training scenario, unqualified training instructions, and qualified training instructions.

[0076] Embodiment 3:

[0077] Based on any of the above embodiments, Embodiment 3 of the present invention is a scene construction module. The function of the scene construction module is to construct a training scene i. The specific process is as follows:

[0078] The scenario construction module uses extended reality technology to construct several highly realistic virtual training environments, and marks them as training scene i in sequence, where i=1, ..., n, n is a positive integer, i is the number of any virtual training environment, and n is the total number of virtual training environments, and sends the training scene i to the multimodal training platform; the virtual training environment includes various scenarios such as home, office, school, hospital, shopping mall, factory, etc.

[0079] Embodiment 4:

[0080] Based on any of the above embodiments, Embodiment 4 of the present invention is a multimodal training platform. The function of the multimodal training platform is to generate training control instructions. The specific process is as follows:

[0081] The humanoid robot is started, and the user uses the multimodal training platform to select a training scenario i, and generates training control instructions at the same time, and sends the training control instructions to the humanoid robot.

[0082] Embodiment 5:

[0083] Based on any of the above embodiments, Embodiment 5 of the present invention is a humanoid robot, and the function of the humanoid robot is to generate an information acquisition instruction. The specific process is as follows:

[0084] After receiving the training control instruction, the humanoid robot performs training operations according to the training scenario i, generates information acquisition instructions at the same time, and sends the information acquisition instructions to the information acquisition module.

[0085] Embodiment 6:

[0086] Based on any of the above embodiments, Embodiment 6 of the present invention is an information acquisition module, and the function of the information acquisition module is to obtain training monitoring information, wherein the training monitoring information includes time information SJ, movement information YD and motion information MO. The specific process is as follows:

[0087] After obtaining the information acquisition instruction, the information acquisition module obtains the generation time of the training control instruction and the time when the humanoid robot receives the training control instruction, obtains the time difference between the two, and marks it as the connection time value JS, obtains the time when the humanoid robot receives the training control instruction and the time when the humanoid robot performs the training operation, obtains the time difference between the two, and marks it as the operation time value CS, quantizes the connection time value JS and the operation time value CS, multiplies the values ​​of the connection time value JS and the operation time value CS by their corresponding preset proportional coefficients, obtains the sum of the two, and marks it as time information SJ; wherein, the preset proportional coefficients corresponding to the connection time value JS and the operation time value CS are s1 and s2 respectively, and s1 and s2 satisfy s1+s2=1, 0<s1<s2<1, and s1=0.38 and s2=0.62;

[0088] The information acquisition module obtains the moving trajectory of the humanoid robot for training operation and marks it as the actual moving trajectory, obtains the difference between the length of the actual moving trajectory and the length of the preset standard moving trajectory, and marks it as the length value CD, overlaps the starting point of the actual moving trajectory with the preset standard moving trajectory, obtains the non-overlapping trajectory length of the actual moving trajectory and the preset standard moving trajectory, and marks it as the non-overlapping value FC, overlaps the starting point of the actual moving trajectory with the preset standard moving trajectory, connects the end point of the actual moving trajectory and the preset standard moving trajectory with a line segment, obtains the area of ​​the area enclosed between the actual moving trajectory and the preset standard moving trajectory, and marks it as the area value MJ, and the length value CD, The non-repeating value FC and the area value MJ are quantized, and the values ​​of the length value CD, the non-repeating value FC and the area value MJ are respectively used as the exponents of the base e, and are respectively marked as the length power value, the non-repeating power value and the area power value, and the length power value, the non-repeating power value and the area power value are respectively multiplied by their corresponding preset proportional coefficients to obtain the arithmetic square root of the sum of the three, and are marked as movement information YD; wherein e is a mathematical constant, and the preset proportional coefficients corresponding to the length power value, the non-repeating power value and the area power value are d1, d2 and d3 respectively, and d1, d2 and d3 satisfy d1+d2+d3=1, 0<d1<d2<d3<1, and d1=0.21, d2=0.35, d3=0.44;

[0089] The information acquisition module obtains all preset monitoring joints on the humanoid robot, and marks them as monitoring objects j in sequence, where j=1, ..., m, where m is a positive integer, j is the number of any monitoring joint, and m is the total number of monitoring joints, obtains the motion trajectory of the monitoring object j for training operation, and marks it as the actual motion trajectory YGj, obtains the difference between the length of the actual motion trajectory YGj and the length of the preset standard motion trajectory YGb, and marks it as the length difference CCi, obtains the maximum length difference CCi, and marks it as the longest value ZC, obtains the sum of all the length difference CCi, and marks it as the total length value TC, obtains the product of the longest value ZC and the total length value TC, and marks it as motion information MO;

[0090] The information acquisition module sends the time information SJ, movement information YD and motion information MO to the information processing module.

[0091] Embodiment 7:

[0092] Based on any of the above embodiments, Embodiment 7 of the present invention is an information processing module. The function of the information processing module is to obtain the training abnormality coefficient XLi. The specific process is as follows:

[0093] The information processing module performs quantization operation on the values ​​of the time information SJ, the movement information YD and the motion information MO according to the preset information processing function to obtain the training abnormality coefficient XLi, and sends the training abnormality coefficient XLi to the classification judgment module;

[0094] The information processing function is as follows:

[0095]

[0096] Where:

[0097] κ is the preset error adjustment factor, taking κ=0.938;

[0098] Both π and e are mathematical constants;

[0099] x1, x2 and x3 are preset weight factors corresponding to the set time information SJ, movement information YD and motion information MO respectively. x1, x2 and x3 satisfy x2>x3>x1>1.594, and x1=1.88, x2=3.07, x3=2.41 are taken.

[0100] Embodiment 8:

[0101] Based on any of the above embodiments, Embodiment 8 of the present invention is a classification judgment module. The function of the classification judgment module is to classify the training scene i and generate a training unqualified instruction or a training qualified instruction. The specific process is as follows:

[0102] The classification judgment module compares the training anomaly coefficient XLi with the preset training anomaly threshold XLy. The comparison results are as follows:

[0103] If the training anomaly coefficient XLi ≥ the training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a training anomaly scene, and the training anomaly scene is sent to the result display module;

[0104] If the training anomaly coefficient XLi is less than the training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a qualified training scene;

[0105] After the humanoid robot completes the training operation according to all the training scenarios i, the classification judgment module obtains the ratio between the number of abnormal training scenarios and the number of training scenarios i, and marks it as an abnormal ratio value YB;

[0106] The classification judgment module obtains the average value of all training abnormal coefficients XLi after the humanoid robot completes the training operation according to all training scenarios i, and marks it as the training mean value XJ;

[0107] The classification judgment module obtains the product of the difference ratio value YB and the training mean value XJ, and marks it as the training evaluation value XP;

[0108] The classification judgment module compares the training evaluation value XP with the preset training evaluation threshold XPy. The comparison results are as follows:

[0109] If the training evaluation value XP ≥ the training evaluation threshold XPy, a training failure instruction is generated and sent to the result display module;

[0110] If the training evaluation value XP<training evaluation threshold value XPy, a training qualified instruction is generated and sent to the result display module.

[0111] Embodiment 9:

[0112] Based on any of the above embodiments, Embodiment 9 of the present invention is a result display module. The function of the result display module is to perform display. The specific process is as follows:

[0113] The result display module displays the training abnormal scene;

[0114] After receiving the instruction of training failure, the result display module displays the words "training failure";

[0115] The result display module displays the words "Training Passed" after receiving the training pass instruction.

[0116] Based on the above embodiments 1-9, the working principle of the present invention is as follows:

[0117] The present invention discloses a multimodal training method and system for a humanoid robot based on extended reality. The training scene is constructed by a scene construction module. The user selects the training scene by using a multimodal training platform and generates a training control instruction. After the humanoid robot receives the training control instruction, the training operation is performed according to the training scene. The information acquisition instruction is generated at the same time. The information acquisition module obtains the information acquisition instruction and then obtains the training monitoring information, wherein the training monitoring information includes time information, movement information and motion information. The information processing module obtains the training abnormality coefficient according to the training monitoring information. The classification judgment module classifies the training scene into a training abnormality scene and a training qualified scene according to the training abnormality coefficient, and obtains the training evaluation information, wherein the training evaluation information includes a difference ratio value and a training mean value. The training evaluation value is obtained according to the training evaluation information, and a training unqualified instruction or a training qualified instruction is generated according to the training evaluation value. The result display module displays the training abnormal scene, the training unqualified instruction and the training qualified instruction. The system performs the training operation according to the training scene and evaluates the training operation. The process collects and analyzes data to obtain training monitoring information. The training abnormality coefficient obtained according to the training monitoring information can comprehensively measure the abnormality degree of the training operation process, and the larger the training abnormality coefficient, the higher the abnormality degree. Then, the training scenes with abnormal training operations are screened out, and then all training scenes are evaluated as a whole to obtain a training evaluation value. The training evaluation value can comprehensively measure the overall abnormality degree of the training scene, and the larger the training evaluation value, the higher the overall abnormality degree. Finally, the result is displayed; the present invention adopts extended reality technology to construct a multimodal virtual training environment, realizes the diversification and customization of robot training scenes, and collects and analyzes data on the training operation process, which can comprehensively reflect the training operation status of the humanoid robot, improve the accuracy and comprehensiveness of abnormality monitoring, and can timely discover abnormal training operations, and realize overall evaluation of the multimodal virtual training environment, thereby improving training efficiency and accuracy, realizing a comprehensive improvement in the training ability of humanoid robots, and providing strong support for the wide application of robots in education, medical care, and service fields.

[0118] It should be further explained that the above formulas are obtained by technicians in this field based on their own work experience and by collecting a large amount of data for software simulation, and a formula close to the actual value is selected. The coefficients in the formula are set by technicians in this field based on actual conditions.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0120] The above contents are merely examples and explanations of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the invention or exceed the scope defined by the claims, they shall all fall within the protection scope of the present invention.

Claims

1. A multimodal training system for humanoid robots based on extended reality, characterized in that: include: An information acquisition module, used to acquire training monitoring information after acquiring an information acquisition instruction, and send the training monitoring information to the information processing module; wherein the training monitoring information includes time information SJ, movement information YD and motion information MO; The specific process of the information acquisition module acquiring training monitoring information is as follows: After obtaining the information acquisition instruction, obtain the generation time of the training control instruction and the time when the humanoid robot receives the training control instruction, obtain the time difference between the two, and mark it as the connection time value JS, obtain the time when the humanoid robot receives the training control instruction and the time when the humanoid robot performs the training operation, obtain the time difference between the two, and mark it as the operation time value CS, quantize the connection time value JS and the operation time value CS, multiply the values ​​of the connection time value JS and the operation time value CS by their corresponding preset proportional coefficients, obtain the sum of the two, and mark it as time information SJ; wherein the preset proportional coefficients corresponding to the connection time value JS and the operation time value CS are s1 and s2 respectively; Obtain the moving trajectory of the humanoid robot for training operation and mark it as the actual moving trajectory, obtain the difference between the length of the actual moving trajectory and the length of the preset standard moving trajectory, and mark it as the length value CD, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, obtain the non-overlapping trajectory length of the actual moving trajectory and the preset standard moving trajectory, and mark it as the non-overlapping value FC, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, connect the end points of the actual moving trajectory and the preset standard moving trajectory with a line segment, and obtain the difference between the actual moving trajectory and the preset standard moving trajectory. The area of ​​the enclosed area is marked as the area value MJ, the length value CD, the non-repeating value FC and the area value MJ are quantized, the values ​​of the length value CD, the non-repeating value FC and the area value MJ are respectively used as the exponents of the base e, and they are respectively marked as the length power value, the non-repeating power value and the area power value, and the length power value, the non-repeating power value and the area power value are respectively multiplied by their corresponding preset proportional coefficients to obtain the arithmetic square root of the sum of the three, and marked as the movement information YD; wherein e is a mathematical constant, and the preset proportional coefficients corresponding to the length power value, the non-repeating power value and the area power value are d1, d2 and d3 respectively; Obtain all preset monitoring joints on the humanoid robot, and mark them as monitoring objects j in sequence, where j=1, ..., m, where m is a positive integer, j is the number of any monitoring joint, and m is the total number of monitoring joints; obtain the motion trajectory of the monitoring object j for training operation, and mark it as the actual motion trajectory YGj; obtain the difference between the length of the actual motion trajectory YGj and the length of the preset standard motion trajectory YGb, and mark it as the length difference CCi; obtain the maximum length difference CCi, and mark it as the longest value ZC; obtain the sum of all the length difference CCi, and mark it as the total length value TC; obtain the product of the longest value ZC and the total length value TC, and mark it as motion information MO; Send the time information SJ, movement information YD and motion information MO to the information processing module; An information processing module is used to obtain a training anomaly coefficient XLi according to the training monitoring information, and send the training anomaly coefficient XLi to the classification judgment module; The specific process of the information processing module obtaining the training abnormality coefficient XLi is as follows: The values ​​of the time information SJ, the movement information YD and the motion information MO are quantized according to a preset information processing function to obtain a training abnormality coefficient XLi; The information processing function is as follows: Where: κ is the preset error adjustment factor, taking κ=0.938; Both π and e are mathematical constants; x1, x2 and x3 are respectively preset weight factors corresponding to the set time information SJ, movement information YD and motion information MO; A classification judgment module is used to classify the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and send the training abnormal scene to the result display module, and obtain the training evaluation information, and obtain the training evaluation value XP according to the training evaluation information, and generate a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and send the training unqualified instruction or the training qualified instruction to the result display module; wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ; The result display module is used to display according to abnormal training scenarios, unqualified training instructions and qualified training instructions.

2. The multimodal training system for humanoid robots based on extended reality according to claim 1, characterized in that: The specific process of the classification judgment module classifying the training scene i is as follows: The training anomaly coefficient XLi is compared with the preset training anomaly threshold XLy, and the comparison results are as follows: If the training anomaly coefficient XLi ≥ the training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a training anomaly scene, and the training anomaly scene is sent to the result display module; If the training anomaly coefficient XLi<training anomaly threshold XLy, the training scene i corresponding to the training anomaly coefficient XLi is marked as a qualified training scene.

3. The multimodal training system for humanoid robots based on extended reality according to claim 1, characterized in that: The specific process of the classification judgment module generating a training failure instruction or a training success instruction is as follows: After the humanoid robot completes the training operation according to all the training scenarios i, the ratio of the number of abnormal training scenarios to the number of training scenarios i is obtained, and marked as an abnormal ratio value YB; After the humanoid robot completes the training operation according to all training scenarios i, the average value of all training abnormal coefficients XLi is obtained and marked as the training mean value XJ; Get the product of the difference ratio value YB and the training mean value XJ, and mark it as the training evaluation value XP; The training evaluation value XP is compared with the preset training evaluation threshold XPy, and the comparison results are as follows: If the training evaluation value XP ≥ the training evaluation threshold XPy, a training failure instruction is generated and sent to the result display module; If the training evaluation value XP<training evaluation threshold value XPy, a training qualified instruction is generated and sent to the result display module.

4. The multimodal training system for humanoid robots based on extended reality according to claim 1, characterized in that: Also includes: The scenario construction module is used to construct the training scenario i and send the training scenario i to the multimodal training platform.

5. The humanoid robot multimodal training system based on extended reality according to claim 4, characterized in that: The specific process of constructing the training scene i by the scene construction module is as follows: Use extended reality technology to construct several highly realistic virtual training environments, and mark them as training scene i in sequence, where i=1, ..., n, n is a positive integer, i is the number of any virtual training environment, and n is the total number of virtual training environments. Then send the training scene i to the multimodal training platform.

6. The humanoid robot multimodal training system based on extended reality according to claim 1, characterized in that: Also includes: The multimodal training platform is used for users to select training scenarios i, generate training control instructions at the same time, and send the training control instructions to the humanoid robot.

7. The humanoid robot multimodal training system based on extended reality according to claim 1, characterized in that: Also includes: The humanoid robot is used to perform training operations according to the training scenario i after receiving the training control instructions, generate information acquisition instructions at the same time, and send the information acquisition instructions to the information acquisition module.

8. A multimodal training method for a humanoid robot based on extended reality, characterized in that: The following steps are involved: Step 1: The scenario construction module constructs the training scenario i and sends the training scenario i to the multimodal training platform; Step 2: The user selects a training scenario i using the multimodal training platform, generates a training control instruction, and sends the training control instruction to the humanoid robot; Step 3: After receiving the training control instruction, the humanoid robot performs training operations according to the training scenario i, generates an information acquisition instruction, and sends the information acquisition instruction to the information acquisition module; Step 4: The information acquisition module acquires the training monitoring information after acquiring the information acquisition instruction, wherein the training monitoring information includes time information SJ, movement information YD and motion information MO, and sends the training monitoring information to the information processing module; The specific process of the information acquisition module acquiring training monitoring information is as follows: After obtaining the information acquisition instruction, obtain the generation time of the training control instruction and the time when the humanoid robot receives the training control instruction, obtain the time difference between the two, and mark it as the connection time value JS, obtain the time when the humanoid robot receives the training control instruction and the time when the humanoid robot performs the training operation, obtain the time difference between the two, and mark it as the operation time value CS, quantize the connection time value JS and the operation time value CS, multiply the values ​​of the connection time value JS and the operation time value CS by their corresponding preset proportional coefficients, obtain the sum of the two, and mark it as time information SJ; wherein the preset proportional coefficients corresponding to the connection time value JS and the operation time value CS are s1 and s2 respectively; Obtain the moving trajectory of the humanoid robot for training operation and mark it as the actual moving trajectory, obtain the difference between the length of the actual moving trajectory and the length of the preset standard moving trajectory, and mark it as the length value CD, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, obtain the non-overlapping trajectory length of the actual moving trajectory and the preset standard moving trajectory, and mark it as the non-overlapping value FC, overlap the starting point of the actual moving trajectory and the preset standard moving trajectory, connect the end points of the actual moving trajectory and the preset standard moving trajectory with a line segment, and obtain the difference between the actual moving trajectory and the preset standard moving trajectory. The area of ​​the enclosed area is marked as the area value MJ, the length value CD, the non-repeating value FC and the area value MJ are quantized, the values ​​of the length value CD, the non-repeating value FC and the area value MJ are respectively used as the exponents of the base e, and they are respectively marked as the length power value, the non-repeating power value and the area power value, and the length power value, the non-repeating power value and the area power value are respectively multiplied by their corresponding preset proportional coefficients to obtain the arithmetic square root of the sum of the three, and marked as the movement information YD; wherein e is a mathematical constant, and the preset proportional coefficients corresponding to the length power value, the non-repeating power value and the area power value are d1, d2 and d3 respectively; Obtain all preset monitoring joints on the humanoid robot, and mark them as monitoring objects j in sequence, where j=1, ..., m, where m is a positive integer, j is the number of any monitoring joint, and m is the total number of monitoring joints; obtain the motion trajectory of the monitoring object j for training operation, and mark it as the actual motion trajectory YGj; obtain the difference between the length of the actual motion trajectory YGj and the length of the preset standard motion trajectory YGb, and mark it as the length difference CCi; obtain the maximum length difference CCi, and mark it as the longest value ZC; obtain the sum of all the length difference CCi, and mark it as the total length value TC; obtain the product of the longest value ZC and the total length value TC, and mark it as motion information MO; Step 5: The information processing module obtains the training abnormality coefficient XLi according to the training monitoring information, and sends the training abnormality coefficient XLi to the classification judgment module; The specific process of the information processing module obtaining the training abnormality coefficient XLi is as follows: The values ​​of the time information SJ, the movement information YD and the motion information MO are quantized according to a preset information processing function to obtain a training abnormality coefficient XLi; The information processing function is as follows: Where: κ is the preset error adjustment factor, taking κ=0.938; Both π and e are mathematical constants; x1, x2 and x3 are respectively preset weight factors corresponding to the set time information SJ, movement information YD and motion information MO; Step 6: The classification judgment module classifies the training scene i into a training abnormal scene and a training qualified scene according to the training abnormality coefficient XLi, and sends the training abnormal scene to the result display module, and obtains the training evaluation information, wherein the training evaluation information includes the difference ratio value YB and the training mean value XJ, and obtains the training evaluation value XP according to the training evaluation information, and generates a training unqualified instruction or a training qualified instruction according to the training evaluation value XP, and sends the training unqualified instruction or the training qualified instruction to the result display module; Step 7: The result display module displays the abnormal training scenario, unqualified training instructions, and qualified training instructions.