Method and device for determining package falling of robot and robot package falling detection system

Abstract

The invention provides a method and device for determining package falling of a robot and a robot package falling detection system. The method comprises the steps that load state data are obtained, and the load state data comprise one or more of the type, the size, the shape, the position and the posture of a load currently borne by the robot; an analysis model is constructed, the analysis model is obtained by training through an LSTM algorithm by using multiple sets of training data, each set of training data in the multiple sets of training data comprises historical load state data obtained in a historical time period and historical analysis results corresponding to the historical load state data, and the historical load state data and the historical analysis results correspond to the historical load state data; the historical analysis result is used for representing whether package falling occurs in the historical carrying process or not; and inputting the load state data into the analysis model to obtain an analysis result corresponding to the load state data. The method solves the problem that in the prior art, the time of packet dropping detection is late, and the efficiency of an assembly line is affected.

Description

Technical Field

[0001] The present application relates to the field of robot handling technology, and in particular, to a method, device, computer program product, and robot package switching detection system for determining package switching of a robot. Background Art

[0002] In the automatic unpacking and loading system, the collaborative robot first grabs a bundle of carton paper from the material box and puts it into the aligner. The clamp cuts and extracts the kraft paper, and the aligner arranges the carton paper and pushes it to the production line. Due to the quality of the material and the shaking of the robot during the handling process, the kraft paper may break, causing the carton paper to scatter and damage the material. In this case, it is necessary to discover the carton paper swap as soon as possible and notify the workers to deal with it to avoid affecting the production line.

[0003] However, the timing of traditional packet loss detection is relatively late, which will affect the efficiency of the pipeline. Summary of the invention

[0004] The main purpose of the present application is to provide a method, device, computer program product and robot packet switching detection system for determining packet switching of a robot, so as to at least solve the problem in the prior art that the timing of packet switching detection is too late, which affects the efficiency of the assembly line.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for determining package substitution of a robot is provided, comprising: obtaining load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot; constructing an analysis model, wherein the analysis model is trained by an LSTM algorithm using multiple sets of training data, and each set of training data in the multiple sets of training data includes historical load status data obtained within a historical time period and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether package substitution has occurred during historical handling; inputting the load status data into the analysis model to obtain the analysis results corresponding to the load status data.

[0006] Optionally, obtaining load status data includes: obtaining initial load status data, wherein the initial load status data is data of the state of the load obtained at an initial moment; preprocessing the initial load status data to obtain the load status data, wherein the preprocessing method includes one or more of data cleaning, data standardization and data normalization.

[0007] Optionally, before constructing the analysis model, the method also includes: constructing a logic gate of the cell state at the Nth moment, wherein N≥2, the cell state is a channel for transmitting information in the LSTM architecture, the logic gate includes a forget gate, an input gate and an output gate, the logic gate is used to control and regulate the information flow of the cell state, the forget gate is used to use a sigmoid function to determine whether the cell state at the Nth moment retains the information of the cell state at the N-1th moment, the input gate is used to use a sigmoid function and a tanh function to determine whether the newly input information is retained in the cell state at the Nth moment, and the output gate is used to use a sigmoid function to determine the output information of the cell state at the Nth moment.

[0008] Optionally, before building the analysis model, the method also includes: building an architecture layer of the LSTM network, wherein the architecture layer includes a hidden layer, an LSTM layer and a fully connected layer, wherein the hidden layer is used to store and update the internal state of the LSTM network, the LSTM layer is used to learn the dependencies between data, and the fully connected layer is used to map the features output by the LSTM layer to corresponding tasks and output them.

[0009] Optionally, before obtaining the load status data, the method further includes: obtaining force control data, wherein the force control data is the torque during the operation of the robot; when the force control data is less than a torque threshold and the robot fails to transport the material to the target position, determining that a package swap has occurred.

[0010] Optionally, after inputting the load status data into the analysis model to obtain the analysis results corresponding to the load status data, the method further includes: generating a line graph based on the analysis results, wherein the horizontal axis of the line graph is the time when the robot is working, and the vertical axis of the line graph is the analysis result, wherein the analysis result is displayed in a first color when a packet is dropped, and the analysis result is displayed in a second color when no packet is dropped; and displaying the line graph in a display interface.

[0011] Optionally, after inputting the load status data into the analysis model to obtain an analysis result corresponding to the load status data, the method further includes: generating warning information when the analysis result indicates packet loss; and sending the warning information to a target terminal.

[0012] According to another aspect of the present application, a device for determining package drop of a robot is provided, comprising: a first acquisition unit, for acquiring load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot; a first construction unit, for constructing an analysis model, wherein the analysis model is trained by an LSTM algorithm using multiple sets of training data, each set of the multiple sets of training data includes historical load status data acquired within a historical time period, and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether package drop has occurred during historical handling; a processing unit, for inputting the load status data into the analysis model to obtain analysis results corresponding to the load status data.

[0013] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the methods for determining packet loss of the robot are implemented.

[0014] According to another aspect of the present application, a robot packet dropping detection system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for determining packet dropping of any one of the robots described.

[0015] By applying the technical solution of the present application, the collected load status data is used to train the LSTM algorithm model. During the robot handling process, the load status data is collected to determine in real time whether package substitution has occurred. In this way, anomalies (such as package substitution) can be detected during the handling process, thereby avoiding the impact of the late timing of package substitution detection in traditional detection methods on the production line efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for executing a method for determining packet loss of a robot provided in an embodiment of the present application is shown;

[0018] Figure 2 A schematic flow chart of a method for determining packet loss of a robot provided in accordance with an embodiment of the present application is shown;

[0019] Figure 3 The schematic diagram of the structure of the automatic unpacking and feeding system is shown;

[0020] Figure 4 A schematic diagram of the process of detecting abnormal load status of the automatic unpacking and loading collaborative robot of the present solution is shown;

[0021] Figure 5 A schematic diagram showing partial load status data of a collaborative robot;

[0022] Figure 6 A schematic diagram showing the feature evidence data of the LSTM model;

[0023] Figure 7 A schematic diagram of a plurality of logic gates is shown;

[0024] Figure 8 shows a schematic diagram of a logic gate;

[0025] Fig. 9 shows a schematic diagram of a forget gate;

[0026] Fig.10 A schematic diagram of an input gate is shown;

[0027] Fig.11 A schematic diagram of an update gate is shown;

[0028] Fig.12 A schematic diagram of an output gate is shown;

[0029] Fig.13 A schematic diagram showing the architectural layers of LSTM;

[0030] Fig.14 A schematic diagram showing the drop in loss for test data;

[0031] Fig.15 A schematic diagram of the generated line graph is shown;

[0032] Fig.16 A structural block diagram of a device for determining packet switching of a robot provided according to an embodiment of the present application is shown.

[0033] The above drawings include the following reference numerals:

[0034] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0039] LSTM (Long Short-Term Memory): It is a time recurrent neural network designed to solve the long-term dependency problem of general RNN (recurrent neural network). All RNNs have a chain form of repeated neural network modules. In the standard RNN, this repeated structural module has only a very simple structure, such as a tanh layer.

[0040] Collaborative robots: As the name suggests, robots and humans can work together on the production line, giving full play to the efficiency of robots and the intelligence of humans. This type of robot is not only cost-effective, but also safe and convenient, which can greatly promote the development of manufacturing enterprises.

[0041] As introduced in the background technology, the timing of packet switching detection in the prior art is relatively late, which will affect the efficiency of the assembly line. To solve the above problem, the embodiments of the present application provide a method, device, computer program product and robot packet switching detection system for determining packet switching of a robot.

[0042] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0043] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a method for determining packet loss of a robot according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0044] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0045] In this embodiment, a method for determining packet loss of a robot running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0046] Figure 2 FIG. 1 is a flow chart of a method for determining a packet drop of a robot according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0047] Step S201, acquiring load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot;

[0048] Specifically, the robot collects information about the load status during the handling process, which is the basis for determining whether an abnormality (such as package drop) occurs during the handling process. The so-called "load status data" includes detailed attributes of the load currently carried by the robot, such as the type, size, shape, position, and posture of the load. These attributes can provide comprehensive information about the physical state of the load and the robot's grasping and handling method, allowing the model to more accurately analyze the stability of the handling process based on these details.

[0049] Step S202, constructing an analysis model, wherein the analysis model is obtained by training with an LSTM algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical load status data acquired in a historical time period, and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether packet loss occurs during historical transportation;

[0050] Specifically, the analysis model is obtained through offline training of the LSTM algorithm, with the purpose of learning and mastering the relationship between the load status data and the handling results (such as whether a packet is dropped) during the historical handling process. The LSTM model can effectively process time series data to predict changes in the load status. The training data contains historical load status data and corresponding historical analysis results, where the historical analysis results are used to mark whether a packet drop event occurred during the historical handling process.

[0051] Assume that there is a set of historical load status data, including sensor data such as load current, speed, acceleration, etc. when the robot moved paper cartons every time in the past six months, as well as a mark of whether a package was dropped during each move. These data are used to train the LSTM model, so that the model can learn the behavior pattern of predicting whether the move is normal under various load status data.

[0052] The constructed analysis model can predict whether packages will be dropped during the transportation process based on the input load status data, thereby providing early warning and avoiding impact on the production line.

[0053] Step S203: input the load status data into the analysis model to obtain analysis results corresponding to the load status data.

[0054] Specifically, during the real-time handling process, this step is responsible for inputting the current load status data into the analysis model constructed in step S202 in real time to obtain the prediction analysis results of the current handling process. This process is dynamic, that is, as the robot's handling action proceeds, the load status data is continuously collected and analyzed and predicted to ensure that anomalies can be immediately identified when they occur.

[0055] Through this embodiment, the collected load status data is used to train the LSTM algorithm model. During the robot handling process, the load status data is collected to determine in real time whether package substitution has occurred. In this way, anomalies (such as package substitution) can be detected during the handling process, thereby avoiding the impact of the late timing of package substitution detection in traditional detection methods on the production line efficiency.

[0056] Specifically, the traditional way to detect package switching is to install sensors in the aligner or on the fixture, but this method has some problems. The aligner can only detect package switching after the strip of paper is placed in it, which is too late and takes time for workers to handle, affecting the efficiency of the assembly line. The sensor on the fixture cannot identify the situation where a strip of paper is sucked into the kraft paper as a package switching. Figure 3 As shown in the figure, in the automatic unpacking and feeding system, the collaborative robot first grabs a bundle of carton paper from the material box and puts it into the aligner. The fixture cuts and extracts the kraft paper. The aligner arranges the carton paper and pushes it to the production line. Due to the quality of the material and the shaking of the robot during the handling process, the kraft paper may break, causing the carton paper to scatter and damage the material. In this case, it is necessary to detect the carton paper swap as soon as possible and notify the workers to deal with it to avoid affecting the production line. The traditional detection of package swapping is to install sensors in the aligner or on the fixture, but this method has some problems. The aligner can only detect the package swap when the carton paper is placed in it. The timing is late, and the workers also need time to deal with it, which affects the efficiency of the production line. The sensor on the fixture cannot recognize the situation of sucking kraft paper and a carton paper as a package swap.

[0057] Common load status abnormalities are divided into the following two situations:

[0058] 1. Grasping failure: When the clamp sucks the carton paper, it fails to grab the carton paper successfully due to reasons such as excessive load, broken kraft paper or air leakage in the suction cup.

[0059] 2. Package switching: During the process of transporting carton paper, the kraft paper breaks due to shaking and load, causing the carton paper to scatter. During package switching, the clamp often sucks the kraft paper and a piece of carton paper.

[0060] Therefore, the solution of the present application monitors the load status data of the collaborative robot during normal operation and abnormal operation, pre-processes the data samples, and trains the machine learning prediction model (i.e., analysis model) offline. When the load data of the robot's real-time operation is input into the model, it is predicted whether the box paper will be dropped during the handling process.

[0061] Specifically, in the offline modeling stage, the main processes are as follows: Figure 4 As shown in the figure, by installing a six-dimensional sensor on the base of the collaborative robot, multi-dimensional data such as robot posture data, speed, acceleration, joint load current loop data, and robot end force sensor data can be obtained in real time. Load status data samples are shown in the figure below. Figure 5 As shown in the figure, the background service reads the robot's multi-dimensional data in real time and draws a curve graph. When the robot's load state is abnormal, the multi-dimensional data curve read above will fluctuate greatly, indicating that the multi-dimensional data is closely related to the abnormal load state and can be used as a basis for predicting the abnormal load state of the robot.

[0062] Figure 6 The data cube input to the LSTM model (i.e., the analysis model) is shown. The first dimension is the number of samples, the second dimension is time, and the third dimension is the number of features. For this example, the samples can be understood as the automatic unpacking and loading systems deployed in different factories. The time axis is per second, and the features are the posture data, speed and acceleration related to the load state, the joint load current loop, and the end force sensor data.

[0063] In the specific implementation process, obtaining the load status data can be achieved through the following steps: obtaining initial load status data, wherein the initial load status data is data of the state of the load obtained at the initial moment; preprocessing the initial load status data to obtain the load status data, wherein the preprocessing method includes one or more of data cleaning, data standardization and data normalization.

[0064] In this solution, data standardization and normalization can avoid the impact of differences in dimensions and numerical ranges in the data on the model training process, making the model more stable. Data cleaning helps remove irrelevant or erroneous information, avoids overfitting of the model to these noisy data during training, and improves the generalization ability of the model.

[0065] Specifically, at the beginning of the robot handling task, the initial state data of the load (such as box paper) will be collected immediately. This data can include information such as the weight, size, shape, position and posture of the load, as well as the force control data of the robot when grasping and carrying the load, such as joint load current, end force sensor reading, etc. The acquired data is used as the initial load state data, which provides the starting point information of the time series for subsequent anomaly detection.

[0066] Preprocessing is a key step to ensure the accuracy of model training and prediction results. The preprocessing process can include operations such as data cleaning, data standardization, and data normalization. The purpose of data cleaning is to remove noise, handle missing values ​​or outliers, and improve data quality. Data standardization and data normalization are both data scaling techniques used to adjust data to the same scale to avoid certain features from having an adverse effect on model training due to too large or too small a numerical range.

[0067] Data cleaning: Removing or correcting inconsistent, erroneous, or invalid data points, such as removing noise from sensor signals or handling missing values ​​in data.

[0068] Data standardization: Adjust the data to a standard normal distribution with a mean of 0 and a standard deviation of 1 to eliminate the dimension effect in the data.

[0069] Data normalization: Scaling the data to a fixed range, such as [0, 1], helps the model learn the relationship between features more effectively.

[0070] Specifically, the training data is the load status data of the automatic unpacking and feeding system running for half a year, from 2023-10-1 to 2024-3-31, and the test data is from 2024-4-1 to 2024-4-30. Because the factory is on vacation, the automatic feeding system is not running, and the data during the vacation period needs to be removed. Before using the data in the LSTM model, the data needs to be normalized to improve the performance and stability of the model. Normalize the original data. Let x be the original data, x s For the normalized data, we have:

[0071]

[0072] Among them, x max and x min are the maximum and minimum values ​​of the original data respectively.

[0073] Specifically, by installing a six-dimensional sensor on the base of the robot, the robot's force control data is obtained, so that the robot stops running when a collision occurs. In addition, after analyzing the robot's force control data, it was found that the loss of packages during the grasping and handling process will cause changes in the load status data. Therefore, by collecting the load status data of the robot during normal operation and abnormal operation, the data samples are fused and standardized, and a machine learning prediction model is built offline to continuously optimize the prediction model. During the collaborative robot's handling of paper strips, the real-time load data monitored is input into the prediction model to predict whether the package will be lost during the handling process, so that abnormalities can be discovered early and material scattering can be handled as soon as possible.

[0074] In some embodiments, before constructing the analysis model, the method further includes the following steps: constructing a logic gate of the cell state at the Nth moment, wherein N≥2, the cell state is a channel for transmitting information in the LSTM architecture, the logic gate includes a forget gate, an input gate and an output gate, the logic gate is used to control and regulate the information flow of the cell state, the forget gate is used to use a sigmoid function to determine whether the cell state at the Nth moment retains the information of the cell state at the N-1th moment, the input gate is used to use a sigmoid function and a tanh function to determine whether the newly input information is retained in the cell state at the Nth moment, and the output gate is used to use a sigmoid function to determine the output information of the cell state at the Nth moment.

[0075] In this scheme, LSTM controls the flow of information in the cell state through logic gates, which can effectively handle long-term dependencies in sequence data. Even if the information is far apart in the sequence, it can be correctly remembered and used by the model. The forget gate mechanism allows LSTM to forget unimportant or outdated information, which helps reduce the computational burden of the model, avoid overfitting, and improve the model's generalization ability on new data. The input gate and output gate mechanisms enable LSTM to selectively update and output information, thereby providing more accurate results in prediction tasks. The logic gate mechanism enables LSTM to have adaptive learning capabilities, that is, the model can dynamically adjust the information retention and update strategy according to the characteristics of the data, which helps the model maintain high performance in different environments.

[0076] Specifically, in LSTM networks, the cell state is the key channel for storing and transmitting sequence information. Logic gates, including forget gates, input gates, and output gates, are used to control the flow of information in the cell state. At each time step (or moment) N, these logic gates decide which information is retained, updated, or output based on the current input and the hidden state of the previous time step (N-1). N ≥ 2 means that starting from the second time step of the sequence, LSTM begins to use the logic gate mechanism to manage the information in the cell state.

[0077] Forget gate: Use the sigmoid activation function to decide whether to retain the information in the previous moment (N-1) unit state. The output range of the sigmoid function is between 0 and 1, indicating the proportion of retained information, 0 means complete forgetfulness, and 1 means complete retention. In this way, the forget gate can selectively forget past information and prevent useless or outdated information from interfering with current predictions.

[0078] Input gate: Combining the sigmoid and tanh activation functions, the sigmoid function determines the extent to which the new input information is preserved, while the tanh function generates candidate values ​​for the new input information. After multiplying the outputs of the two, it is determined which new input information will be stored in the cell state at the current time (N), thereby updating the cell state.

[0079] Output gate: The sigmoid function is also used to determine how much information of the current cell state (N) is output to the model output, and how much information will be passed as a hidden state to the next moment (N+1). The output gate controls the flow of information from the cell state to the model output, ensuring that the output information is filtered and most relevant to the current task.

[0080] Specifically, we build an LSTM model. LSTM (Long Short-Term Memory) is a special type of RNN that is able to learn long-term patterns. All recurrent neural networks have the form of a chain of repeating modules of neural networks. LSTM also has this chain-like structure, but the repeating modules have a different structure. These modules have four, not one, neural network layer, such as Figure 7 As shown, Figure 7 The structure of A on the left is the same as the network structure in the middle, so it is not highlighted. Figure 7 The structure of A on the right is the same as the network structure in the middle, so it is not highlighted.

[0081] The key to LSTM is the cell state, the horizontal line running through the top of the diagram. LSTM is able to remove or add information to the node to change the state of information flow, which is carefully regulated by structures called gates (logic gates are simply gates), such as Figure 8 As shown. A gate is a node that can selectively pass information. They consist of a sigmoid neural network layer and point-by-point multiplication operations. The sigmoid layer outputs a number between 0 and 1 that describes how much of each information vector should pass through. A value of zero means "don't let anything through", while a value of 1 means "let everything through". LSTM has three such gates that are used to protect and control the state of the information flow vector.

[0082] Forget gate, such as Fig. 9As shown, the highlighted part is the structure of the forget gate, and the input is the output h of the previous unit t-1 and the current time input data x t Note that the sigmoid module here does not only have a sigmoid activation function, but is actually a sigmoid layer, which includes matrix multiplication with the weight and adding bias. Output f t The output value range is 0-1, which determines what information is allowed to pass and what information is retained. t The output is 0, then the state information of the previous unit is completely forgotten and will not be input into the current unit; if f t If the output is 1, the state information of the previous unit is fully retained by the current unit. If we take the load current change above as an example, in order to correctly predict that the next state is that the current increases, the key information that the current has been increasing before should be retained as completely as possible in C t-1 The information is then passed to the prediction unit level by level. The Forget gate will retain more relevant information and forget more irrelevant information. However, it does not mean that it will not pass, but a certain percentage will be retained.

[0083] f t =σ(W f .[h t-1 ,x t ]+b f ). W is the weight, h is the hidden state, h is the bias, and h is the offset.

[0084] input gate, such as Fig.10 As shown in the figure, the highlighted part is the structure of the input gate, which determines which new input information can be retained in the cell state. The first sigmoid layer is called the input gate, and the output i t It is a control signal that controls which new input information can be updated to the cell state. The tanh layer next to it generates new information. For example, in the case of load circuit changes, more important key data is output, and unimportant non-captured current data information passes through i t The control can reduce the output to the cell state. For key information, such as capturing the current data when loading, i t It should be controlled to be saved in the cell state as much as possible. t and The result of the multiplication determines what kind of information can be updated to the cell state.

[0085] i t =σ(W i.[h t-1 ,x t ]+b i );

[0086]

[0087] update gate, such as Fig.11 As shown, the highlighted part is the structure of the update gate, which concatenates the new information with the cell state that needs to retain information to obtain a new cell state.

[0088]

[0089] Output gate, such as Fig.12 As shown, the highlighted part is the structure of the output gate, which determines what information is output. t Controls which information needs to be output to the next unit. The output information is determined by the cell state.

[0090] o t =σ(W o .[h t-1 ,x t ]+b o );

[0091] h t =o t *tanh(C t ).

[0092] LSTM solves the long-term dependency problem of traditional RNN through the gating mechanism, making it more effective in processing long-term dependencies in sequence data. The three main gates of LSTM are:

[0093] Forget gate: Determines which previous information to discard. By using a sigmoid function, the gate produces a value between 0 and 1, where 0 means completely discarded and 1 means completely retained.

[0094] Input gate: determines what new information is added to the cell state, and is controlled by a combination of the new information generated by the sigmoid function and a tanh function.

[0095] Output gate: determines which information is output from the cell state and affects the output of the next time step.

[0096] These gate mechanisms work together to ensure that the LSTM unit can dynamically control the inflow, outflow, and forgetting of information as needed, thereby achieving better memory performance when processing time series data.

[0097] In the specific implementation process, before building the analysis model, the above method also includes the following steps: building an architecture layer of the LSTM network, wherein the above architecture layer includes a hidden layer, an LSTM layer and a fully connected layer, wherein the above hidden layer is used to store and update the internal state of the above LSTM network, the above LSTM layer is used to learn the dependency relationship between data, and the above fully connected layer is used to map the features output by the above LSTM layer to the corresponding tasks and output them.

[0098] In this solution, the existence of the hidden layer enables LSTM to store long-term dependency information in the sequence, avoiding the problem that long-term dependencies are difficult to capture in traditional RNNs, and improving the model's ability to process long sequence data. The LSTM layer can effectively learn the dependencies between elements in the sequence through the logic gate mechanism, including not only short-term adjacent elements, but more importantly, it can handle the associations between elements in the long span, thereby performing better in prediction and classification tasks. The fully connected layer integrates the sequence features output by the LSTM layer into a fixed-size vector, further refines the features, and outputs them to specific tasks, which helps improve the model's prediction accuracy and decision-making ability.

[0099] Specifically, hidden layer: The hidden layer in the LSTM network usually refers to the hidden state update mechanism of its internal loop structure. It stores and updates the internal state of the LSTM network, which contains the historical memory of the sequence information and plays a key role in the prediction of subsequent time steps.

[0100] LSTM Layer: The LSTM layer is the core of the LSTM network. It contains multiple LSTM units, each of which uses a logic gate mechanism (forget gate, input gate, and output gate) to control and regulate the flow of information. The purpose of the LSTM layer is to learn and capture dependencies in sequence data, especially long-term dependencies, which are difficult for traditional RNNs (recurrent neural networks) to handle.

[0101] Fully Connected Layer: The fully connected layer is used to map the features output by the LSTM layer to the output space of a specific task, such as classification or regression tasks. It integrates the sequence features extracted by the LSTM layer into a fixed-size vector as the basis for the final prediction or decision.

[0102] The hidden layer stores the memory of historical load status data, which helps the model understand the changing context of the current load status. The LSTM layer can predict whether a packet drop may occur in the next handling process by learning the dependency between historical load status data and packet drop events. The fully connected layer maps the features output by the LSTM layer to a single prediction result, that is, whether a packet drop occurs, providing an intuitive output that facilitates subsequent decision-making and abnormal alarms. In short, the architectural layer design of the LSTM network enables the model to handle complex dependencies in sequence data, store and update long-term memory, and finally output to specific tasks through the fully connected layer, which has important application value in the detection of packet drop in automatic unpacking and loading collaborative robots, and improves the accuracy of prediction and the response speed of the system.

[0103] Specifically, the architecture of LSTM is as follows Fig.13 As shown in the figure, a LSTM network is formed using 1 hidden layer, 2 LSTM layers, and 1 fully connected layer. The output is lstm = nn.LSTM (input_size, hidden_size, num_layers), where input_size is the feature dimension of the input data, hidden_size is the dimension of the hidden layer in the LSTM, and num_layers is the number of layers of the recurrent neural network.

[0104] Because 26 data such as pose data, load current loop, etc. are used as input and 1 prediction result is output, we set input_size to 26 and output_size to 1:

[0105] model=LSTMModel(input_size=26, output_size=1).

[0106] In the solution of this application, the LSTM network consists of the following parts:

[0107] Hidden layer 1: This refers to the hidden layer units inside the LSTM network. Each layer of LSTM units contains the above-mentioned gating mechanism. The hidden layer units are responsible for processing the complex temporal dependencies in the sequence data.

[0108] 2-layer LSTM layer: indicates that the network uses a two-layer LSTM structure. Multi-layer LSTM can learn more complex and deeper time series patterns and improve the prediction ability of the model. Each layer of LSTM has an independent gating mechanism. The output of the previous layer is used as the input of the next layer. Through multi-layer superposition processing, longer time dependencies in the data can be captured.

[0109] 1 fully connected layer: located after the LSTM layer, it is used to map the features output by the LSTM layer to the actual output. The fully connected layer performs the final processing on the features extracted by the LSTM layer to generate the prediction results. In this solution, the fully connected layer is used to generate a prediction of whether packet loss has occurred from the output of the LSTM layer.

[0110] The gating mechanism is a core component inside the LSTM layer. In each layer of LSTM units, the gating mechanism helps the unit decide which information to keep, which to discard, and which to add to the current state. These decisions are made based on the input data and the hidden state of the previous time step, ensuring that the LSTM can effectively handle dynamic changes in sequence data. LSTM layer and fully connected layer: After the LSTM layer processes the sequence data, it passes the output to the fully connected layer. The fully connected layer adjusts the weights and biases of the output features of the LSTM layer, and finally generates a binary classification output through an activation function (such as sigmoid), that is, whether a packet drop event is detected. The role of the fully connected layer is to transform the output of the LSTM layer so that it can be directly used for decision-making or prediction.

[0111] In some embodiments, before obtaining the load status data, the method further includes the following steps: obtaining force control data, wherein the force control data is the torque during the operation of the robot; when the force control data is less than a torque threshold and the robot fails to transport the material to the target position, it is determined that a package swap has occurred.

[0112] In this solution, based on the force control data and preset torque threshold during the operation of the collaborative robot, packet loss events can be detected in real time and accurately, which not only improves the accuracy of detection, but also reduces dependence on hardware sensors. Even when the sensors may be limited or unreliable, the changes in force control data can be used to effectively judge packet loss.

[0113] Specifically, force control data refers to the torque data of the collaborative robot during operation, which is obtained through equipment such as six-dimensional force sensors installed at the base or joints of the robot. Torque data can reflect the interaction force between the robot and the load, including but not limited to the weight of the load, the force distribution during the handling process, and the dynamic force changes of the robot during the handling process. These data are crucial to determine whether the load is safely and stably controlled during the handling process.

[0114] The torque threshold is a preset torque strength standard used to distinguish normal handling operations from possible abnormal situations. When the robot is handling materials, the torque value should be kept within a certain range under normal circumstances. If the force control data is lower than this threshold, it means that the robot may not be able to grasp or control the material stably, which may be a precursor to a package swap event.

[0115] When the force control data is less than the torque threshold and the robot fails to move the material to the target location, it can be determined that a package swap has occurred. There are two key conditions:

[0116] Torque below threshold: This indicates that the force connection between the robot and the material is unstable, and the material may have partially or completely fallen out of the robot gripper.

[0117] The material did not reach the target location: This means that the material was scattered or detached due to package substitution, affecting the normal handling process, and the material was not successfully placed at its preset target location.

[0118] The torque threshold may be 2 N·m, 15 N·m, 0.5 N·m, or any other feasible threshold.

[0119] In the robot's working environment, the coordinates of the target position need to be predefined. This is usually part of the robot program and is determined according to the production line layout and workflow. The target position can be any fixed position on the production line, such as a grading machine, assembly table, etc.

[0120] The robot's position and posture are monitored in real time using the robot's built-in position tracking system (such as built-in joint encoders) or external position tracking equipment (such as vision systems, laser radars, etc.). During the handling process, the robot control system will continuously update its position and posture information, which needs to be compared with the coordinates of the target position.

[0121] You can use visual sensors (such as cameras, depth cameras) or contact sensors (such as pressure sensors, proximity sensors) to monitor the state of the material. Visual sensors can check whether the material is present at the target location, whether it is missing or scattered. Contact sensors can detect the contact status between the robot gripper and the material, or the contact status of the material at the target location.

[0122] The robot's real-time position is compared with the target position coordinates to determine whether the robot has reached the predetermined position. A certain distance threshold (for example, a position deviation of less than 1 cm) is used to determine whether the robot is close enough to the target position, which helps reduce the error of position detection. Combined with the material status monitoring results, if the robot reaches the target position and the material status is normal (that is, the material is completely placed at the target position without being scattered or missing), it can be considered that the material has been successfully moved.

[0123] During the specific implementation process, after the above-mentioned load status data is input into the above-mentioned analysis model to obtain the analysis results corresponding to the above-mentioned load status data, the above-mentioned method also includes the following steps: generating a line graph based on the above-mentioned analysis results, wherein the horizontal axis of the above-mentioned line graph is the time when the above-mentioned robot is working, and the vertical axis of the above-mentioned line graph is the above-mentioned analysis result, wherein the above-mentioned analysis result is displayed in a first color when the package is dropped, and the above-mentioned analysis result is displayed in a second color when the package is not dropped; the above-mentioned line graph is displayed in the display interface.

[0124] In this solution, by displaying the analysis results in real-time in the form of a line graph, operators can immediately identify the package switching incident and maintain a comprehensive understanding of the production line status even during the processing process, so as to respond in time and reduce production stoppages and material losses. The color-coded line graph provides an intuitive visualization method, making the difference between package switching incidents and normal operations clear at a glance, reducing the difficulty of interpreting data and improving monitoring efficiency.

[0125] Specifically, based on the analysis results, a line graph is generated for visual display. The horizontal axis of the line graph is the time when the robot works, which can be a timestamp or an operation sequence number, indicating the time dimension of the data. The vertical axis is the analysis result, that is, the packet drop probability predicted by the model or the binary judgment result (packet drop / no packet drop).

[0126] In order to distinguish between packet drop and non-packet drop, the analysis results in the line chart are color-coded. When the prediction result indicates packet drop, the data point or line in the line chart is displayed in the first color (such as red) to attract attention; when the prediction result indicates that no packet drop has occurred, it is displayed in the second color (such as green) to indicate that everything is normal.

[0127] The generated line graph is displayed in real time on the display interface of the operator or the monitoring system to facilitate real-time monitoring and rapid response.

[0128] Specifically, we use Python's matplotlib library to visualize the prediction results. i is the value of the i-th data point in the prediction result, A i is the value of the ith data point in the actual data, d i is the date of the i-th data point in the actual data, then the image of the prediction result can be expressed as:

[0129]

[0130]

[0131] After multiple training and parameter optimization, Fig.14It can be seen that the loss of the test data dropped quickly and reached convergence:

[0132] Let's look at the comparison between the predicted results and the actual data. Fig.15 As shown in the figure, the shapes are basically consistent. When the training model predicts values ​​greater than 0.8 in the future, it can be considered that packet loss anomalies will occur, which is basically consistent with the actual results. In addition, the model has a little lag in prediction, which can be optimized through some other strategies later.

[0133] In some embodiments, after inputting the above-mentioned load status data into the above-mentioned analysis model and obtaining the analysis results corresponding to the above-mentioned load status data, the above-mentioned method also includes the following steps: when the above-mentioned analysis results indicate packet loss, generating early warning information; and sending the above-mentioned early warning information to the target terminal.

[0134] In this solution, the real-time generation and sending of warning information enables operators or automation systems to respond immediately to packet substitution events and take necessary measures (such as stopping robot operations, adjusting production processes, dispatching personnel for inspection, etc.), thereby significantly reducing the impact of abnormal events on the production line and improving the stability and efficiency of the production line.

[0135] Specifically, the warning information usually contains key information such as the specific description of the event, the time of occurrence, the robot operation status, etc., so that the recipient can quickly understand the situation.

[0136] The generated warning information will be sent to the target terminal through the network or communication interface. The target terminal can be an operator's handheld device (such as a smartphone, tablet), a workstation, a display screen in the monitoring room, or a control module in the automation system. By sending the warning information to the target terminal, it ensures that key personnel or systems can receive the alarm of abnormal events in real time.

[0137] In addition, during the online monitoring phase, e.g. Figure 4 As shown in the figure, the trained model is deployed to the production environment in advance. During the production process, the real-time load status data of the robot monitored, namely the robot posture data, speed, acceleration, joint load current loop data and robot end force sensor data, are preprocessed. The preprocessed robot load status data is input into the LSTM model to predict whether abnormal situations such as package substitution will occur at the next moment during the robot handling process, and the real-time load status monitoring results are output. If package substitution occurs, the control program will alarm according to the monitoring results and notify the workers to go to the site as soon as possible for processing.

[0138] In summary, the main innovation of the solution of this application is to obtain the force control data collected by the six-dimensional sensor of the collaborative robot under different conditions to determine whether the package is dropped during the robot's handling process. The load status data collected during normal operation and abnormal operation are used to train the LSTM algorithm prediction model, and offline modeling is performed to avoid wasting computing resources. During the process of grabbing the box paper, the prediction model inputs the load status data of the robot during operation monitored in real time to predict whether the package is dropped.

[0139] In summary, the solution of the present application obtains force control data collected by the six-dimensional sensor of the collaborative robot under different conditions, and discovers the difference in load status data during abnormal operation and normal operation, which provides conditions for judging whether abnormalities such as package substitution occur during the robot's handling process.

[0140] The load status data collected during normal and abnormal operation of the robot is used to train the LSTM algorithm prediction model, and offline modeling is performed to avoid wasting computing resources. During the process of grabbing paper strips, the prediction model inputs the load status data of the robot during operation monitored in real time to predict whether a package is dropped. If a package is dropped, the worker is notified as soon as possible to handle it, thereby improving production efficiency.

[0141] The embodiment of the present application also provides a device for determining the packet loss of a robot. It should be noted that the device for determining the packet loss of a robot in the embodiment of the present application can be used to execute the method for determining the packet loss of a robot provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0142] The following is an introduction to the robot's packet loss determination device provided in an embodiment of the present application.

[0143] Fig.16 1 is a structural block diagram of a device for determining packet loss of a robot according to an embodiment of the present application. Fig.16 As shown, the device comprises:

[0144] A first acquisition unit 10 is used to acquire load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot;

[0145] A first construction unit 20 is used to construct an analysis model, wherein the analysis model is obtained by training with an LSTM algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical load status data acquired in a historical time period and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether packet loss occurs during historical transportation;

[0146] The processing unit 30 is used to input the above-mentioned load state data into the above-mentioned analysis model to obtain the analysis result corresponding to the above-mentioned load state data.

[0147] Through this embodiment, the collected load status data is used to train the LSTM algorithm model. During the robot handling process, the load status data is collected to determine in real time whether package substitution has occurred. In this way, anomalies (such as package substitution) can be detected during the handling process, thereby avoiding the impact of the late timing of package substitution detection in traditional detection methods on the production line efficiency.

[0148] During the specific implementation process, the first acquisition unit includes an acquisition module and a preprocessing module. The acquisition module is used to acquire initial load state data, wherein the initial load state data is data on the state of the load acquired at the initial moment; the preprocessing module is used to preprocess the initial load state data to obtain the load state data, wherein the preprocessing method includes one or more of data cleaning, data standardization and data normalization.

[0149] In this solution, data standardization and normalization can avoid the impact of differences in dimensions and numerical ranges in the data on the model training process, making the model more stable. Data cleaning helps remove irrelevant or erroneous information, avoids overfitting of the model to these noisy data during training, and improves the generalization ability of the model.

[0150] In some embodiments, the device further includes a second construction unit, which is used to construct a logic gate of the cell state at the Nth moment before constructing the analysis model, wherein N≥2, the cell state is a channel for transmitting information in the LSTM architecture, the logic gate includes a forget gate, an input gate and an output gate, the logic gate is used to control and regulate the information flow of the cell state, the forget gate is used to use a sigmoid function to determine whether the cell state at the Nth moment retains the information of the cell state at the N-1th moment, the input gate is used to use a sigmoid function and a tanh function to determine whether the newly input information is retained in the cell state at the Nth moment, and the output gate is used to use a sigmoid function to determine the output information of the cell state at the Nth moment.

[0151] In this scheme, LSTM controls the flow of information in the cell state through logic gates, which can effectively handle long-term dependencies in sequence data. Even if the information is far apart in the sequence, it can be correctly remembered and used by the model. The forget gate mechanism allows LSTM to forget unimportant or outdated information, which helps reduce the computational burden of the model, avoid overfitting, and improve the model's generalization ability on new data. The input gate and output gate mechanisms enable LSTM to selectively update and output information, thereby providing more accurate results in prediction tasks. The logic gate mechanism enables LSTM to have adaptive learning capabilities, that is, the model can dynamically adjust the information retention and update strategy according to the characteristics of the data, which helps the model maintain high performance in different environments.

[0152] During the specific implementation process, the above-mentioned device also includes a third construction unit, which is used to construct an architecture layer of the LSTM network before constructing the analysis model, wherein the above-mentioned architecture layer includes a hidden layer, an LSTM layer and a fully connected layer, wherein the above-mentioned hidden layer is used to store and update the internal state of the above-mentioned LSTM network, the above-mentioned LSTM layer is used to learn the dependency relationship between data, and the above-mentioned fully connected layer is used to map the features output by the above-mentioned LSTM layer to the corresponding tasks and output them.

[0153] In this solution, the existence of the hidden layer enables LSTM to store long-term dependency information in the sequence, avoiding the problem that long-term dependencies are difficult to capture in traditional RNNs, and improving the model's ability to process long sequence data. The LSTM layer can effectively learn the dependencies between elements in the sequence through the logic gate mechanism, including not only short-term adjacent elements, but more importantly, it can handle the associations between elements in the long span, thereby performing better in prediction and classification tasks. The fully connected layer integrates the sequence features output by the LSTM layer into a fixed-size vector, further refines the features, and outputs them to specific tasks, which helps improve the model's prediction accuracy and decision-making ability.

[0154] In some embodiments, the device further includes a second acquisition unit and a determination unit, wherein the second acquisition unit is used to acquire force control data before acquiring the load status data, wherein the force control data is the torque during the operation of the robot; and the determination unit is used to determine that a package has been swapped when the force control data is less than a torque threshold and the robot has not transported the material to the target position.

[0155] In this solution, based on the force control data and preset torque threshold during the operation of the collaborative robot, packet loss events can be detected in real time and accurately, which not only improves the accuracy of detection, but also reduces dependence on hardware sensors. Even when the sensors may be limited or unreliable, the changes in force control data can be used to effectively judge packet loss.

[0156] During the specific implementation process, the above-mentioned device also includes a first generating unit and a display unit. The first generating unit is used to generate a line graph according to the above-mentioned analysis result after inputting the above-mentioned load status data into the above-mentioned analysis model and obtaining the analysis result corresponding to the above-mentioned load status data, wherein the horizontal axis of the above-mentioned line graph is the time when the above-mentioned robot is working, and the vertical axis of the above-mentioned line graph is the above-mentioned analysis result, wherein the above-mentioned analysis result is displayed in a first color when the package is dropped, and the above-mentioned analysis result is displayed in a second color when the package is not dropped; the display unit is used to display the above-mentioned line graph in the display interface.

[0157] In this solution, by displaying the analysis results in real-time in the form of a line graph, operators can immediately identify the package switching incident and maintain a comprehensive understanding of the production line status even during the processing process, so as to respond in time and reduce production stoppages and material losses. The color-coded line graph provides an intuitive visualization method, making the difference between package switching incidents and normal operations clear at a glance, reducing the difficulty of interpreting data and improving monitoring efficiency.

[0158] In some embodiments, the above-mentioned device also includes a second generating unit and a sending unit. The second generating unit is used to generate early warning information after inputting the above-mentioned load status data into the above-mentioned analysis model to obtain the analysis result corresponding to the above-mentioned load status data, when the above-mentioned analysis result indicates packet loss; the sending unit is used to send the above-mentioned early warning information to the target terminal.

[0159] In this solution, the real-time generation and sending of warning information enables operators or automation systems to respond immediately to packet substitution events and take necessary measures (such as stopping robot operations, adjusting production processes, dispatching personnel for inspection, etc.), thereby significantly reducing the impact of abnormal events on the production line and improving the stability and efficiency of the production line.

[0160] The above-mentioned device for determining the packet switching of the robot includes a processor and a memory, and the above-mentioned first acquisition unit, the first construction unit and the processing unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to implement corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0161] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the timing of packet loss detection in the prior art is late, which affects the efficiency of the pipeline.

[0162] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0163] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method for determining the packet loss of the robot.

[0164] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the method for determining packet loss of the robot is executed when the program is running.

[0165] An embodiment of the present invention provides a device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the steps of the method for determining the packet loss of the robot are implemented. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0166] A computer program product includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining packet loss of the robot described above in each embodiment of the present application are implemented.

[0167] The present application also provides a robot packet dropping detection system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for determining packet dropping of any of the above-mentioned robots.

[0168] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0174] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0175] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0177] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining packet loss of a robot, characterized in that: include: Acquire load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot; Constructing an analysis model, wherein the analysis model is obtained by training with an LSTM algorithm using multiple sets of training data, wherein each set of training data in the multiple sets of training data includes historical load status data acquired in a historical time period and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether packet loss occurs during historical transportation; The load state data is input into the analysis model to obtain analysis results corresponding to the load state data.

2. The method according to claim 1, characterized in that Obtain load status data, including: Acquire initial load state data, wherein the initial load state data is data of the load state acquired at an initial moment; The initial load state data is preprocessed to obtain the load state data, wherein the preprocessing method includes one or more of data cleaning, data standardization and data normalization.

3. The method according to claim 1, characterized in that Before constructing the analysis model, the method further includes: Construct a logic gate of a cell state at the Nth moment, wherein N≥2, the cell state is a channel for transmitting information in an LSTM architecture, the logic gate comprises a forget gate, an input gate and an output gate, the logic gate is used to control and regulate the information flow of the cell state, the forget gate is used to use a sigmoid function to determine whether the cell state at the Nth moment retains the information of the cell state at the N-1th moment, the input gate is used to use a sigmoid function and a tanh function to determine whether the newly input information is retained in the cell state at the Nth moment, and the output gate is used to use a sigmoid function to determine the output information of the cell state at the Nth moment.

4. The method according to claim 1, characterized in that Before constructing the analysis model, the method further includes: Construct an architecture layer of an LSTM network, wherein the architecture layer includes a hidden layer, an LSTM layer and a fully connected layer, wherein the hidden layer is used to store and update the internal state of the LSTM network, the LSTM layer is used to learn the dependency relationship between data, and the fully connected layer is used to map the features output by the LSTM layer to the corresponding task and output it.

5. The method according to claim 1, characterized in that Before acquiring the load status data, the method further includes: Acquiring force control data, wherein the force control data is the torque during the operation of the robot; When the force control data is less than the torque threshold and the robot fails to transport the material to the target location, it is determined that a package is swapped.

6. The method according to any one of claims 1 to 5, characterized in that After inputting the load state data into the analysis model to obtain the analysis result corresponding to the load state data, the method further includes: Generate a line graph according to the analysis result, wherein the horizontal axis of the line graph is the time when the robot is working, and the vertical axis of the line graph is the analysis result, wherein the analysis result is displayed in a first color when the package is dropped, and the analysis result is displayed in a second color when the package is not dropped; The line graph is displayed in a display interface.

7. The method according to any one of claims 1 to 5, characterized in that After inputting the load state data into the analysis model to obtain the analysis result corresponding to the load state data, the method further includes: If the analysis result indicates packet loss, generating a warning message; The warning information is sent to the target terminal.

8. A device for determining packet switching of a robot, characterized in that: include: A first acquisition unit, configured to acquire load status data, wherein the load status data includes one or more of the type, size, shape, position and posture of the load currently carried by the robot; A first construction unit is used to construct an analysis model, wherein the analysis model is obtained by training with an LSTM algorithm using multiple sets of training data, each set of training data in the multiple sets of training data includes historical load status data acquired in a historical time period and historical analysis results corresponding to the historical load status data, wherein the historical analysis results are used to characterize whether packet loss occurs during a historical transport process; The processing unit is used to input the load status data into the analysis model to obtain the analysis result corresponding to the load status data.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for determining packet switching of a robot described in any one of claims 1 to 7 are implemented.

10. A robot packet switching detection system, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for determining packet dropping of the robot described in any one of claims 1 to 7.

Images