Diagnosis method, device and equipment based on key frame reinforcement learning model

Abstract

The application provides a diagnosis method and device based on a key frame reinforcement learning model, and relates to the technical field of robot reinforcement learning, comprising: acquiring first information; acquiring second information; searching for the minimum Q value in the Q values output by each step in the second information; calculating the average Q value of all steps according to the second information; calculating the difference between the Q value output by each step before the step corresponding to the minimum Q value and the average Q value respectively, and obtaining a difference set; searching for the minimum difference in the difference set, wherein the Q value output by the step corresponding to the minimum difference is the closest Q value, finding the corresponding key frame in the first information according to the time point corresponding to the closest Q value, and diagnosing the failure cause of the robot according to the key frame. The application eliminates the uncertainty of the robot in the reinforcement learning model training process, and improves the failure diagnosis efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot reinforcement learning, in particular to a key frame-based reinforcement learning model diagnosis method, device and equipment. BACKGROUND

[0002] Reinforcement learning is a trial-and-error method in which a reinforcement learning model continuously improves its strategy through cycles of failure and optimization. Currently, experts mainly use statistical methods to evaluate the performance of reinforcement learning models. However, statistical indicators often mask many details, resulting in uncertainty in the reinforcement learning process, and when a robot fails, it cannot quickly and accurately find the cause of the failure. SUMMARY

[0003] The purpose of the present application is to provide a key frame-based reinforcement learning model diagnosis method, device and equipment to improve the above problems. In order to achieve the above purpose, the technical solutions adopted by the present application are as follows:

[0004] In a first aspect, the present application provides a key frame-based reinforcement learning model diagnosis method, comprising:

[0005] Obtain first information, the first information includes a training video of a round of reinforcement learning model training of a robot failure, the reinforcement learning model is a reinforcement learning model based on AC framework;

[0006] Obtain second information, the second information includes the Q value output by each step in a round of reinforcement learning model training;

[0007] Search for the minimum Q value in the Q value output by each step in the second information;

[0008] Calculate the average Q value of all steps from the step corresponding to the minimum Q value according to the second information;

[0009] Calculate the difference between the Q value output by each step from the step corresponding to the minimum Q value and the average Q value in a round of reinforcement learning model training, respectively, to obtain a difference set;

[0010] Search for the minimum difference in the difference set, the Q value output by the step corresponding to the minimum difference is the closest Q value, find the corresponding key frame in the first information according to the time corresponding to the closest Q value, and diagnose the failure cause of the robot according to the key frame.

[0011] In a second aspect, the present application further provides a key frame-based reinforcement learning model diagnosis device, characterized in that it comprises:

[0012] The first acquisition module is configured to acquire first information, wherein the first information comprises a training video of a round of reinforcement learning model training of a robot failure, and the reinforcement learning model is an AC framework-based reinforcement learning model.

[0013] The second acquisition module is configured to acquire second information, wherein the second information comprises Q values output by each step in a round of the reinforcement learning model training.

[0014] The first search module is configured to search for a minimum Q value in the Q values output by each step in the second information.

[0015] The first calculation module is configured to calculate an average Q value of all steps from the step corresponding to the minimum Q value forward according to the second information.

[0016] The second calculation module is configured to calculate a difference value between the Q value output by each step from the step corresponding to the minimum Q value forward and the average Q value in a round of the reinforcement learning model training, respectively, to obtain a difference value set.

[0017] The second search module is configured to search for a minimum difference value in the difference value set, wherein the Q value output by the step corresponding to the minimum difference value is the closest Q value, a corresponding key frame is found in the first information according to a time point corresponding to the closest Q value, and a failure cause of the robot is diagnosed according to the key frame.

[0018] In a third aspect, the present application further provides a key frame-based reinforcement learning model diagnosis device, comprising:

[0019] A memory is configured to store a computer program.

[0020] A processor is configured to implement steps of the key frame-based reinforcement learning model diagnosis method when the computer program is executed.

[0021] In a fourth aspect, the present application further provides a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of the above-mentioned key frame-based reinforcement learning model diagnosis method.

[0022] The present application has the following beneficial effects: the key frame-based reinforcement learning model diagnosis method of the present application can quickly find out a key frame of a robot at the time of failure, experts can identify weaknesses of the reinforcement learning model by analyzing the key frame, and thus corresponding measures can be taken to improve the performance, thereby eliminating the uncertainty of the robot in the reinforcement learning model training process and improving the robot failure diagnosis efficiency.

[0023] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by means of the instrumentalities and combinations particularly pointed out in the written description and claims hereof. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0025] Figure 1 The diagnostic method flow chart of the key frame-based reinforcement learning model described in the embodiments of the present application;

[0026] Figure 2 The diagnostic device structure schematic diagram of the key frame-based reinforcement learning model described in the embodiments of the present application;

[0027] Figure 3 The diagnostic device structure schematic diagram of the key frame-based reinforcement learning model described in the embodiments of the present application.

[0028] In the figure, 701 is a first acquisition module, 702 is a second acquisition module, 703 is a first search module, 704 is a first calculation module, 705 is a second calculation module, 706 is a second search module, 707 is a third acquisition module, 708 is a segmentation module, 709 is a conversion module, 710 is a normalization module, 711 is a hierarchical clustering module, 712 is an analysis module, 713 is a fourth acquisition module, 714 is a splicing module, 715 is a fifth calculation module, 716 is a drawing module, 717 is a third calculation module, 718 is a fourth calculation module, 719 is a repetition module, 800 is a diagnostic device of a key frame-based reinforcement learning model, 801 is a processor, 802 is a memory, 803 is a multimedia component, 804 is an I / O interface, and 805 is a communication component. DETAILED DESCRIPTION

[0029] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0030] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0031] Embodiment 1

[0032] The embodiment provides a diagnosis method based on a key frame-based reinforcement learning model.

[0033] The research and development idea of the present application is: for robot control tasks, an incorrect action usually does not immediately lead to failure, but has a significant impact after a period of time. For example, when a robot steps on the edge of an obstacle, it may gradually lose balance, continue to walk for a distance, until the posture becomes uncontrollable, and finally falls down. Since this process is gradual, it is difficult for us to accurately identify when the reinforcement learning model makes an incorrect decision through statistical measures, and it is also unclear how these errors gradually develop and eventually lead to failure. Therefore, after the robot fails, domain experts begin to spend a lot of time repeatedly watching training videos frame by frame to manually search for key frames that can reflect the cause of the failure, however, this way leads to a very time-consuming diagnosis process, and the results are often not accurate enough. Therefore, the present applicant has developed the present application, aiming to eliminate the uncertainty of the robot in the reinforcement learning model training process, improve the efficiency of robot failure diagnosis, and thus more effectively locate the incorrect action of the robot.

[0034] Referring to Figure 1 , the method includes steps S100, S200, S300, S400, S500 and S600.

[0035] Step S100: obtaining first information, the first information comprising a training video of a round of reinforcement learning model training of a robot failure, the reinforcement learning model being an AC framework-based reinforcement learning model.

[0036] The training video is composed of video frames, each video frame corresponding to a time point, the time step of the video frame being consistent with the time step of the reinforcement learning model training, and the training video comprising a key frame capable of reflecting a robot failure cause.

[0037] Step S200: obtaining second information, the second information comprising Q values output by each step in a round of the reinforcement learning model training;

[0038] Step S300: searching for a minimum Q value in the Q values output by each step in the second information;

[0039] Step S400: calculating average Q values of all the steps according to the second information;

[0040] Step S500: calculating a difference between the Q value output by each step before the step corresponding to the minimum Q value and the average Q value, respectively, to obtain a difference set;

[0041] Step S600: searching for a minimum difference in the difference set, the Q value output by the step corresponding to the minimum difference being the closest Q value, finding a corresponding key frame in the first information according to a time point corresponding to the closest Q value, and diagnosing a failure cause of the robot according to the key frame.

[0042] The key frame refers to a frame corresponding to a key action in a reinforcement learning training video that has the greatest impact on the final result (success / failure). Finding the key frame is helpful for experts to diagnose the reinforcement learning model to analyze possible error causes of the model.

[0043] Applicants have found that the Q value output by the AC framework-based reinforcement learning model represents the evaluation of the model on the action, and the evaluation is the lowest when the model fails. The Q value continuously decreases from the key frame to the time point when the model fails, and the failure is usually caused by the gradual accumulation of previous erroneous actions (i.e., the key frame), and the erroneous action corresponds to the minimum Q value output by the AC framework-based reinforcement learning model. Therefore, the Q value output by the AC framework-based reinforcement learning model tends to gradually decrease after the step corresponding to the key frame, and eventually decreases to a minimum value, so it is necessary to calculate the difference between the Q value output by each step before the step corresponding to the minimum Q value and the average Q value, and the step in which the Q value output by each step before the step corresponding to the minimum Q value is closest to the average Q value, thereby finding the corresponding key frame.

[0044] The method further comprises:

[0045] Step S101: Obtain third information, the third information including all time series and all environment states in a round of the reinforcement learning model training in step S100;

[0046] Step S102: Segment all the time series according to all the environment states; we manually segment all sequences in a round according to the environment state, and the basis for segmentation is the change of the environment, for example, the robot is climbing uphill, downhill, crossing obstacles, etc. Since the environment will affect the decision of the reinforcement learning model, segmenting the environment can eliminate its influence and prepare for the next clustering step. For example, the division of labor between the left leg and the right leg is different when the robot is climbing uphill and crossing.

[0047] Step S103: Convert all the segmented time series into position sequences to obtain uneven position sequences;

[0048] The reason for converting the time series into position sequences is that the reinforcement learning model may take different times to complete the same task. For example, one reinforcement learning model may take longer to climb uphill, while another reinforcement learning model may quickly complete the same action. Converting their time series into position sequences can reduce the differences caused by time;

[0049] Step S104: Normalize the uneven position sequences based on an interpolation algorithm to obtain uniform position sequences;

[0050] Since the position corresponding to each time is uneven, the converted uneven position sequences can be interpolated to uniform position sequences by interpolation method, and the uniform position sequences are normalized. The above processing is for the subsequent hierarchical clustering;

[0051] Step S105: Hierarchical clustering of the uniform position sequences based on a hierarchical clustering algorithm to obtain hierarchical clustering results, the hierarchical clustering results including several action patterns in a round of the reinforcement learning model training;

[0052] Step S106: Analyze the differences between the hierarchical clustering results of the reinforcement learning model training of different rounds.

[0053] Unlike simple and intuitive actions in video games, robots are controlled by continuous motor signals, which makes it challenging to directly analyze the raw signals. In post-analysis, domain experts usually need to spend a lot of time repeatedly watching training videos at different stages. Since reinforcement learning models need different times to pass through different obstacles, experts need to manually adjust their action clips to compare their behaviors. In order to search for typical behavior patterns, experts need to manually filter action clips with contextual information such as environment, reward, Q value, etc. This is a very time-consuming process, usually relying on expert experience. In addition, the obtained action patterns are not comprehensive enough, lacking representativeness. Therefore, the present application can cluster the robot's continuous actions without semantics into several typical action patterns through the above steps S101-S106, thereby effectively extracting the action patterns of the reinforcement learning model and helping domain experts understand the semantics of the robot's continuous actions.

[0054] The step S106 specifically comprises:

[0055] Obtaining fifth information, the fifth information comprising motion patterns of a previous round of successful reinforcement learning model training and motion patterns of a next round of successful reinforcement learning model training;

[0056] Judging whether the motion patterns in the first round of successful reinforcement learning model training are consistent with the motion patterns in the second round of successful reinforcement learning model training.

[0057] The step S106 further comprises:

[0058] According to the second information and the hierarchical clustering result, calculating average Q values corresponding to each action pattern in a round of reinforcement learning model training of the reinforcement learning model training;

[0059] Comparing the sizes of the average Q values corresponding to each action pattern, and judging the effectiveness of each action pattern according to the comparison result.

[0060] The step S105 specifically comprises:

[0061] Step S1051: Taking each segmented sequence in the uniform position sequence as an individual category, calculating two-by-two DTW distances between categories based on the DTW algorithm, and merging two categories corresponding to the first minimum DTW distance obtained by calculation to form a new category;

[0062] The first minimum DTW distance is the minimum DTW distance among the two-by-two DTW distances calculated based on the DTW algorithm.

[0063] Step S1052: calculating DTW distances between each of the remaining classes and the new class based on the DTW algorithm, and merging two classes corresponding to the second minimum DTW distance obtained by calculation to form another new class;

[0064] The second minimum DTW distance is the minimum DTW distance among the DTW distances between each of the remaining classes and the new class calculated based on the DTW algorithm.

[0065] Step S1053: iteratively performing step S1052 until the number of clusters is less than the preset threshold of the number of clusters, and outputting a hierarchical clustering result.

[0066] The reason for calculating the DTW distances between each of the remaining classes and the new class based on the DTW algorithm is that the action mode may have a shift in the position sequence. For example, when the robot climbs down a step, it may be located in the middle or edge of the previous step. Using the DTW distance for clustering can match the best correspondence relationship of different position sequences, which helps to eliminate the possible shift of the action mode in the position sequence. In addition, in order to facilitate the exploration of clustering results of different granularities, we use a hierarchical clustering method. Users can further merge or split the hierarchical clustering results according to experience, and compare the evolution of action modes at different training rounds.

[0067] The method further comprises:

[0068] Step S107: obtaining fourth information, wherein the fourth information comprises an environment feature vector and an action vector, the environment feature is an environment feature vector preset according to all the environment states in step S101, and the action vector is an action vector output by the reinforcement learning model according to an input preset action instruction;

[0069] Step S108: performing vector splicing on the environment feature vector and the action vector to obtain a spliced vector;

[0070] Step S109: calculating SHAP values of the spliced vector based on the SHAP algorithm;

[0071] Step S1010: obtaining a SHAP heat map based on a heat map drawing function according to the SHAP values, and analyzing the importance of the environment and the action according to the SHAP heat map.

[0072] The application can be visualized and operated in application, and specifically includes a statistical view, a single round view, an evolution view and a feature view.

[0073] Among them, the statistical view summarizes the training statistical information of the reinforcement learning model, including three line charts and one stacked area chart. All charts use the training set as the horizontal axis, representing the entire training process over a period of time. The line chart tracks three basic statistics of training from left to right: time steps, average reward per episode, and average Q value per episode. The said "episode" means a round of training of the reinforcement learning model, and the time step represents the length of the event, indicating the efficiency of the reinforcement learning model in completing the task within a period of time. The average reward is an indicator of learning progress provided by the environment, as a steady increase in average reward will indicate that the reinforcement learning model is making progress. The Q value represents the expected score of the reinforcement learning model for its own behavior given by the Critic network, which provides important contextual information when analyzing the behavior of the reinforcement learning model. A high Q value indicates that the reinforcement learning model believes it has made the right decision. The stacked area chart demonstrates the distribution of rewards / q values obtained by the reinforcement learning model in episodes. The vertical axis represents the proportion of rewards / q values within different ranges, and the horizontal axis represents the number of episodes. It provides insights into the variability of rewards / q values and can be used to identify any trends or patterns in the performance of the reinforcement learning model. For example, a decrease in low rewards and a steady increase in high rewards can indicate that the reinforcement learning model has found the right action strategy. On the other hand, if most of the rewards of the reinforcement learning model are still low, with fewer high rewards, it can indicate that the reinforcement learning model has encountered a bottleneck and is struggling to generalize its strategy to new situations. All charts in this view are linked. The user can modify the episode input to adjust the gray dashed line and view the statistical data of different episodes. In addition, the stacked area chart supports switching the display content, such as rewards or Q values. By clicking the corresponding tab, the user can trigger an update event to switch the stacked area chart.

[0074] Among them, the single round view aims to present a detailed summary of each round of training of the reinforcement learning model and provide key frame previews for quickly identifying failed actions. The view is composed of multiple rows, and each row is divided into two parts: the left part shows the summary of this round, and the right part shows the key frame and the reward / q value line chart. The left part includes three horizontal bar charts and six indicator blocks. Each horizontal bar chart represents the time step, reward, and q value, respectively, and the black solid line represents the y-axis. The color of the indicator block maps the performance of the reinforcement learning model when facing different obstacles, with red blocks indicating failure, green blocks indicating success, and gray blocks indicating no data. The right part consists of the key frame and the line chart below. The key frame is determined by the method described in the previous section, and the line chart shows how the reward / Q value evolves over time in this round. The orange arrow on the horizontal axis indicates the current step, which defaults to $t_{key}$ corresponding to the key frame. We set the value represented by the horizontal axis in the line chart to the average of the reward / Q value, and the value above the axis represents a value higher than the average, and the value below the axis represents a value lower than the average. This means that the intersection between the curve and the axis may be a key frame. The view supports vertical scrolling, and users can manually input intervals to specify the circular intervals between each row. The key frame and the line chart are linked, and users can use left and right arrow interactions to adjust the key frame. In addition, experts believe that rewards reflect the environment's evaluation of behavior and are important background information for analyzing errors. Therefore, support users to switch the content displayed in the line chart by clicking different tabs.

[0075] Among them, the evolution view aims to provide a global overview of the test environment and show the evolution of action patterns through the node-link diagram in the figure, allowing users to explore and compare the differences in patterns over time between different reinforcement learning models. The evolution diagram is divided into two parts: environment overview and node-link diagram. The environment overview shows the location of all obstacles that the robot needs to traverse, which helps users align different action sequences. The coordinate axes above represent the range of each obstacle. The red and blue curves correspond to the trajectory projection of the first and second rounds selected by the user in the episode view, and the black dashed line represents the final position of the robot. The node-link diagram shows all the patterns determined by the hierarchical clustering algorithm, which is used to compare the differences between different patterns and the evolution of behavior. The node-link diagram contains multiple columns, each corresponding to the obstacle segment indicated by the coordinate axes above. Each column contains multiple blocks, with one block corresponding to a motion pattern cluster within the segment. For example, the first column contains six blocks, indicating that there are six motion patterns within the flat area corresponding to the first segment. The scattered points in the block represent the actions of the robot, with green representing left leg actions and yellow representing right leg actions. The position (x, y) of the scattered points maps to the actions of the upper and lower joints, so a point can determine the action of the leg. Action mapping can map the actions of reinforcement learning models to a two-dimensional plane to help users better understand the structure of the action space. If the average q value of the motion pattern represented by a block is lower than the average q value of other motion patterns, the boundary of the block will be red, indicating that the effectiveness of the pattern is poor. The links between blocks represent the connections between different motion patterns. A complete link through several blocks represents a round of training. Similarly, the color of the line is used to distinguish between the first and second rounds. Users can select the first and second rounds in the view for comparison, or directly modify the input box in the evolution diagram. In addition, due to the possibility of occlusion between scattered points, users can switch the type of scattered points to be displayed by clicking the corresponding tab.

[0076] Among them, the feature view aims to explore the importance of environmental features and reveal their impact on the decision in the graph. This view consists of three interrelated parts: timeline, importance heatmap (i.e., the SHAP heatmap), and action sequence graph. Different colors in the timeline represent different obstacles, and the shaded area represents the selected analysis time range. Users can drag the left and right sliders respectively to determine the start / end time, or directly click and drag the shaded area to slide and filter the time period. Changes in the timeline trigger the redrawing of the importance heatmap and the action sequence graph. The vertical axis of the importance heatmap represents environmental features, and the horizontal axis represents the timeline of analysis. Each color in the heatmap grid represents the importance of the feature to the decision, with red indicating a positive impact and blue indicating a negative impact. The action sequence graph contains multiple groups of action examples. The first and last samples of the sequence represent the robot's actions at the start / end time determined by the timeline, and the intermediate samples are obtained by uniform sampling. The orange arrows represent the torque direction at the four joints, and the length of the arrow represents the size of the force. This helps experts have a more intuitive understanding of the behavior of the reinforcement learning model. The feature view supports multiple analysis modes. Users can compare the impact of different environmental features on the reinforcement learning model at the same training stage by dragging the timeline. For example, experts can compare which features the reinforcement learning model pays attention to when going up and down the stairs to understand whether the model can distinguish different scenarios and take appropriate strategies. In addition, users can select features of different rounds through the buttons in the event view to compare the evolution of decisions at different training stages. By analyzing the distribution of color mapping in the feature heatmap, users can explore the decisions of the reinforcement learning model from the perspective of critics and check whether the decisions are stable during training.

[0077] To verify the effectiveness of the present application, the applicant further illustrates the present application through the following verification experiments:

[0078] Verification of overcoming obstacles: After locating the key frame before the robot crosses the obstacle in the single round view, we click the right arrow to increase the time and observe how the robot crosses the obstacle. We find that the first time the robot crosses the block is an accidental behavior. Since the left leg is blocked by the block, when the torque of the right leg is large, the robot as a whole is lifted above the block, thereby overcoming the obstacle.

[0079] Verifying differences in behavior patterns: During the training process, we observed that the reinforcement learning model first successfully crossed all obstacles at the 750th episode. However, it was not until the 2400th episode that the reinforcement learning model was able to consistently complete the task. This led the domain experts to wonder what happened during the transition period and why the model took so long to master the skill of crossing obstacles. Therefore, we selected the 750th set and the 2400th set as the 1st set and the 2nd set, respectively, and examined their action patterns in the evolution view. After comparing, we found that these two events belonged to different clusters in columns 1, 3, 4, 6, 7, 9, 10, and 14, indicating that their action strategies when facing these obstacles were inconsistent. Based on this observation, the domain experts hypothesized that the action pattern of the 750th episode was not stable enough, and the successful completion of all obstacles was a coincidence. It was not until the 2400th episode that the model truly mastered the skill of crossing obstacles.

[0080] Verifying the association between features and decisions: Among all obstacles, experts were most interested in "trap" because it required the reinforcement learning model to learn how to adjust its pace, accurately step on the edge of the trap, and then cross it with the other leg. Experts wanted to know if the reinforcement learning model had really learned this rule or relied on luck. We selected the features corresponding to the 1250th episode, in which the reinforcement learning model successfully crossed the trap. In the evolution view, we selected the range corresponding to the trap by dragging the time axis. Using the SHAP heat map, we found that the value of row 7 was significantly higher, which corresponds to whether the left leg is in contact with the ground. Experts believe that the reinforcement learning model pays high attention to whether the left foot is in contact with the ground during the first half of the trap crossing, indicating that the reinforcement learning model has successfully mastered the key to crossing the trap.

[0081] Embodiment 2:

[0082] As shown in the Figure 2 The embodiment provides a diagnosis device of a key frame-based reinforcement learning model, and the device comprises:

[0083] A first acquisition module 701 is configured to acquire first information, wherein the first information comprises a training video of a round of reinforcement learning model training in which a robot fails, and the reinforcement learning model is an AC framework-based reinforcement learning model.

[0084] A second acquisition module 702 is configured to acquire second information, wherein the second information comprises Q values output by each step in a round of reinforcement learning model training.

[0085] A first search module 703 is configured to search for a minimum Q value in the Q values output by each step in the second information.

[0086] The first calculation module 704 is configured to calculate an average Q value of all the steps according to the second information;

[0087] The second calculation module 705 is configured to calculate a difference value between a Q value output by each step before the step corresponding to the minimum Q value and the average Q value, respectively, to obtain a difference value set;

[0088] The second search module 706 is configured to search for a minimum difference value in the difference value set, the minimum difference value corresponding to a Q value output by a step closest to a Q value, find a corresponding key frame in the first information according to a time point corresponding to the Q value closest to the Q value, and diagnose a failure cause of the robot according to the key frame.

[0089] The diagnosis apparatus further comprises:

[0090] The third acquisition module 707 is configured to acquire third information, the third information including all time sequences and all environment states in a round of the reinforcement learning model training;

[0091] The environment states include undulating terrains and obstacles such as steps, traps and obstacles that hinder the robot from advancing.

[0092] The segmentation module 708 is configured to perform segmentation processing on all the time sequences according to all the environment states;

[0093] The conversion module 709 is configured to convert all the segmented time sequences into position sequences to obtain uneven position sequences;

[0094] The normalization module 710 is configured to perform normalization processing on the uneven position sequences based on an interpolation algorithm to obtain uniform position sequences;

[0095] The hierarchical clustering module 711 is configured to perform hierarchical clustering on the uniform position sequences based on a hierarchical clustering algorithm to obtain a hierarchical clustering result, the hierarchical clustering result including a plurality of action patterns in a round of the reinforcement learning model training;

[0096] The analysis module 712 is configured to analyze differences between hierarchical clustering results of different rounds of the reinforcement learning model training.

[0097] The hierarchical clustering module comprises:

[0098] The third calculation module 717 is configured to take each segmented sequence in the uniform position sequence as a category, calculate DTW distances between each two categories based on a DTW algorithm, and merge two categories corresponding to a first minimum DTW distance to form a new category;

[0099] The fourth calculation module 718 is configured to calculate DTW distances between each two categories of the new category and the remaining categories based on the DTW algorithm, and merge two categories corresponding to a second minimum DTW distance to form another new category.

[0100] The repeating module 719 is configured to repeatedly calculate until the number of clusters is less than a preset cluster threshold, and output a hierarchical clustering result.

[0101] The analysis module 712 is specifically configured to:

[0102] The fifth information includes a motion pattern in a first round of successful reinforcement learning model training and a motion pattern in a next round of successful reinforcement learning model training.

[0103] The analysis module 712 is specifically configured to:

[0104] The second information and the hierarchical clustering result are used to calculate average Q values corresponding to each motion pattern in a round of reinforcement learning model training.

[0105] The analysis module 712 is specifically configured to:

[0106] The diagnostic device comprises:

[0107] The fourth acquisition module 713 is configured to acquire fourth information, wherein the fourth information includes an environment feature vector and an action vector, the environment feature vector is a preset feature vector according to all the environment states, and the action vector is an action vector output by the reinforcement learning model according to an input preset action instruction.

[0108] The splicing module 714 is configured to splice the environment feature vector and the action vector to obtain a spliced vector.

[0109] The fifth calculation module 715 is configured to calculate SHAP values of the spliced vector based on a SHAP algorithm.

[0110] The drawing module 716 is used to draw a SHAP heatmap based on the SHAP value using a heatmap drawing function, and to analyze the importance of the environment and actions based on the SHAP heatmap.

[0111] It should be noted that the specific manner in which each module performs its operation in the apparatus described in the above embodiments has been described in detail in the embodiments of the method, and will not be elaborated here.

[0112] Example 3:

[0113] Corresponding to the above method embodiments, this embodiment also provides a diagnostic method and device for a keyframe-based reinforcement learning model. The diagnostic method and device for a keyframe-based reinforcement learning model described below can be referred to in correspondence with the diagnostic method for a keyframe-based reinforcement learning model described above.

[0114] Figure 3 This is a block diagram of a diagnostic method device 800 based on a keyframe-based reinforcement learning model, according to an exemplary embodiment. Figure 3 As shown, the diagnostic method device 800 for the keyframe-based reinforcement learning model may include a processor 801 and a memory 802. The diagnostic method device 800 for the keyframe-based reinforcement learning model may also include one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0115] The processor 801 is configured to control overall operations of the device 800 for diagnosing the keyframe-based reinforcement learning model, so as to complete all or part of the steps in the method for diagnosing the keyframe-based reinforcement learning model. The memory 802 is configured to store various types of data to support operations of the device 800 for diagnosing the keyframe-based reinforcement learning model. For example, the data can include instructions of any application or method for operating on the device 800 for diagnosing the keyframe-based reinforcement learning model, and application-related data such as contact data, sent and received messages, pictures, audio, video, and the like. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, for example, static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 803 can include a screen and an audio component. The screen can be a touch screen, for example, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 802 or transmitted through the communication component 805. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 805 is configured to perform wired or wireless communication between the device 800 for diagnosing the keyframe-based reinforcement learning model and other devices. The wireless communication, for example, Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include a Wi-Fi module, a Bluetooth module, an NFC module.

[0116] In an example embodiment, the key frame based reinforcement learning model diagnosis method device 800 can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements for executing the above-mentioned key frame based reinforcement learning model diagnosis method.

[0117] In another example embodiment, a computer readable storage medium including program instructions that, when executed by a processor, implement the steps of the above-mentioned key frame based reinforcement learning model diagnosis method is also provided. For example, the computer readable storage medium can be the above-mentioned memory 802 including program instructions that can be executed by the processor 801 of the key frame based reinforcement learning model diagnosis method device 800 to complete the above-mentioned key frame based reinforcement learning model diagnosis method.

[0118] Embodiment 4:

[0119] Corresponding to the above method embodiments, in this embodiment, a readable storage medium is also provided, and the readable storage medium described below can be referred to in conjunction with the above-mentioned key frame based reinforcement learning model diagnosis method.

[0120] A readable storage medium, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the above-mentioned key frame based reinforcement learning model diagnosis method of the method embodiments.

[0121] The readable storage medium can be specifically a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various readable storage media that can store program codes.

[0122] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0123] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A diagnostic method based on a key frame-based reinforcement learning model, characterized by, include: Obtain first information, which includes a training video of a round of reinforcement learning model training when the robot malfunctions, wherein the reinforcement learning model is a reinforcement learning model based on the AC framework; Obtain second information, which includes the Q-values ​​output at each step in one round of training of the reinforcement learning model; Search the second information for the minimum Q value among the Q values ​​output by each of the steps; Calculate the average Q value of all the steps based on the second information; Calculate the difference between the Q value output by each step preceding the step corresponding to the minimum Q value and the average Q value to obtain a set of differences; The minimum difference is searched in the set of differences. The minimum difference corresponds to the Q value output in the step that is closest to the Q value. The corresponding key frame is found in the first information based on the time corresponding to the closest Q value. The cause of the robot's failure is diagnosed based on the key frame. 2.The method of diagnosing a keyframe-based reinforcement learning model according to claim 1, wherein The diagnostic method further includes: Obtain third information, which includes all time series and all environmental states in one round of training of the reinforcement learning model; All time series are segmented according to all environmental states; All the segmented time series are converted into location series to obtain a non-uniform location series; The non-uniform position sequence is normalized using an interpolation algorithm to obtain a uniform position sequence. The uniform position sequence is clustered hierarchically based on a hierarchical clustering algorithm to obtain hierarchical clustering results, which include several action patterns in one round of training of the reinforcement learning model. The differences in hierarchical clustering results between different rounds of training of the reinforcement learning model were analyzed. 3.The method of diagnosing a keyframe-based reinforcement learning model according to claim 2, characterized in that The uniform position sequence is subjected to hierarchical clustering based on a hierarchical clustering algorithm to obtain hierarchical clustering results, including: Each segment sequence in the uniform position sequence is treated as a separate category. The pairwise DTW distance between each category is calculated based on the DTW algorithm. The two categories corresponding to the first minimum DTW distance are merged to form a new category. The pairwise DTW distances between the remaining categories and the new category are calculated based on the DTW algorithm. The two categories corresponding to the calculated second minimum DTW distance are then merged to form another new category. This process continues until the number of clusters is less than a preset clustering threshold, at which point the calculation stops and the hierarchical clustering results are output.

4. The diagnostic method for a keyframe-based reinforcement learning model according to claim 2, characterized in that... The diagnostic method further includes: Obtain fourth information, which includes an environmental feature vector and an action vector. The environmental feature vector is a preset feature vector based on all the environmental states, and the action vector is an action vector output by the reinforcement learning model based on the input preset action command. The environmental feature vector and the action vector are concatenated to obtain the concatenated vector; The SHAP value of the concatenated vector is calculated based on the SHAP algorithm; A SHAP heatmap is generated based on the SHAP value using a heatmap drawing function, and the importance of the environment and actions is analyzed based on the SHAP heatmap.

5. The diagnostic method for a keyframe-based reinforcement learning model according to claim 2, characterized in that... Analyze the differences between the hierarchical clustering results trained by the reinforcement learning model in different rounds, including: Obtain the fifth piece of information, which includes the motion pattern of the previous round of successful reinforcement learning model training and the motion pattern of the next round of successful reinforcement learning model training. Determine whether the motion pattern in the previous successful reinforcement learning model training is consistent with the motion pattern in the next successful reinforcement learning model training.

6. A diagnostic device based on a keyframe-based reinforcement learning model, characterized in that, include: The first acquisition module is used to acquire first information, which includes a training video of a round of reinforcement learning model training in which the robot malfunctions, and the reinforcement learning model is a reinforcement learning model based on the AC framework. The second acquisition module is used to acquire second information, which includes the Q-values ​​output in each step of the reinforcement learning model training in one round. A first search module is configured to search for the minimum Q value among the Q values ​​output by each of the steps in the second information. A first calculation module is configured to calculate the average Q value of all the steps based on the second information. The second calculation module is used to calculate the difference between the Q value output by each step preceding the step corresponding to the minimum Q value and the average Q value, and obtain a set of differences. The second search module is used to search for the minimum difference in the difference set, where the minimum difference corresponds to the Q value output in the step that is closest to the Q value. Based on the time corresponding to the closest Q value, the module finds the corresponding key frame in the first information and diagnoses the cause of the robot's failure based on the key frame.

7. The diagnostic device for a keyframe-based reinforcement learning model according to claim 6, characterized in that, The diagnostic device further includes: The third acquisition module is used to acquire third information, which includes all time series and all environmental states in one round of training of the reinforcement learning model. The segmentation module is used to segment all the time series according to all the environmental states; A conversion module is used to convert all the segmented time series into location series to obtain a non-uniform location series. A normalization module is used to normalize the non-uniform position sequence based on an interpolation algorithm to obtain a uniform position sequence. A hierarchical clustering module is used to perform hierarchical clustering on the uniform position sequence based on a hierarchical clustering algorithm to obtain hierarchical clustering results, wherein the hierarchical clustering results include several action patterns in one round of training of the reinforcement learning model. An analysis module is used to analyze the differences between the hierarchical clustering results of the reinforcement learning model training in different rounds.

8. The diagnostic device for a keyframe-based reinforcement learning model according to claim 7, characterized in that, The hierarchical clustering module includes: The third calculation module is used to treat each segment sequence in the uniform position sequence as a separate category, calculate the pairwise DTW distance between each category based on the DTW algorithm, and merge the two categories corresponding to the first minimum DTW distance to form a new category. The fourth calculation module is used to calculate the pairwise DTW distance between the remaining categories and the new category based on the DTW algorithm, and merge the two categories corresponding to the calculated second minimum DTW distance to form another new category; The repeating module is used to continue the calculation until the number of clusters is less than a preset cluster number threshold, and then output the hierarchical clustering result.

9. The diagnostic device for a keyframe-based reinforcement learning model according to claim 8, characterized in that, The diagnostic device includes: The fourth acquisition module is used to acquire fourth information, which includes an environmental feature vector and an action vector. The environmental feature vector is a feature vector preset based on all the environmental states, and the action vector is an action vector output by the reinforcement learning model according to the input preset action command. A splicing module is used to splice the environmental feature vector and the action vector to obtain a spliced ​​vector; The fifth calculation module is used to calculate the SHAP value of the concatenated vector based on the SHAP algorithm; A drawing module is used to draw a SHAP heatmap based on the SHAP value using a heatmap drawing function, and to analyze the importance of the environment and actions based on the SHAP heatmap.

10. A diagnostic device based on a keyframe-based reinforcement learning model, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the diagnostic method of the keyframe-based reinforcement learning model as described in any one of claims 1 to 4 when executing the computer program.

Images