Industrial production line fault early warning method based on multi-modal data fusion

Abstract

The invention belongs to the technical field of industrial equipment fault early warning, and provides an industrial production line fault early warning method based on multi-modal data fusion, and the method comprises the steps: carrying out the real-time monitoring of parts on a production conveying belt in a monitoring period, obtaining a part conveying track, constructing a part conveying track sequence, and carrying out the real-time monitoring of the part conveying track. The motion trail of each part in the space can be accurately presented through the part conveying trail sequence, the position information of the parts at different time points can be accurately recorded, continuous trail data are formed, the motion trend of the parts in the whole monitoring period can be analyzed, and the monitoring accuracy is improved. According to the method, track change prediction is carried out on a plurality of part conveying tracks in the part conveying track sequence to predict whether the part conveying tracks in the part conveying track sequence collide or not, so that whether the two part conveying tracks intersect in the space or not can be accurately judged, and potential collision risks are found in advance; and the situation that robot grabbing is affected due to track deviation caused by collision of the parts is avoided.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial equipment fault early warning, and in particular to an industrial production line fault early warning method based on multi-modal data fusion. BACKGROUND

[0002] In the field of industrial production, especially in the production line involving production conveying belts, accurate grabbing and placing of parts are crucial to ensure the smoothness of the production process and the quality of products. However, existing industrial production line fault early warning methods have many problems in actual application, which affect production efficiency and product quality.

[0003] In the prior art, the motion trajectory of a part may be affected by various factors and undergo complex changes, making it difficult to accurately determine whether two part conveying trajectories will intersect using simple distance judgments or empirical models, thus leading to potential collision risks that cannot be discovered in time.

[0004] Secondly, after a part collides, existing methods often cannot accurately determine the trajectory deviation of the part after collision, causing the robot to be unable to take targeted grabbing measures. For example, for a part whose trajectory deviates after collision, the robot is likely to fail to grab if it does not have corresponding compensation measures. For a part whose trajectory does not deviate after collision but whose arrival time changes, the robot is also likely to miss the grabbing opportunity if it does not have early warning grabbing time.

[0005] Therefore, the application provides an industrial production line fault early warning method based on multi-modal data fusion. SUMMARY

[0006] To make up for the deficiencies of the prior art and solve at least one technical problem raised in the background.

[0007] The technical solution adopted by the application to solve its technical problems is:

[0008] An industrial production line fault early warning method based on multi-modal data fusion, comprising:

[0009] In a monitoring period, real-time monitoring of parts on a production conveying belt is performed, and a part conveying trajectory sequence is constructed;

[0010] Trajectory change prediction is performed on multiple part conveying trajectories in the part conveying trajectory sequence, collision of the part conveying trajectories is predicted, and potential collision parts are identified;

[0011] After simulating collision of the potential collision parts, analysis is performed to obtain identification and differentiation results;

[0012] The identification and differentiation results include collision deviation parts or collision non-deviation parts.

[0013] For the collision offset parts, the grabbing compensation amount is obtained, for the collision non-offset parts, the grabbing time of the collision non-offset parts is warned, the warning grabbing time is obtained, and the part grabbing work on the conveying belt is completed.

[0014] As a further scheme of the present application, the construction process of the part conveying trajectory sequence is as follows:

[0015] The monitoring period is equally divided into a plurality of monitoring nodes;

[0016] A space trajectory monitoring model is constructed, in the space trajectory monitoring model, the space coordinate point of each part at each monitoring node is obtained, all the space coordinate points of the part in the monitoring period are connected, and the part conveying trajectory is obtained.

[0017] The part conveying trajectory corresponding to each part is sorted from small to large according to the corresponding initial space coordinate distance, and a part conveying trajectory sequence is constructed.

[0018] As a further scheme of the present application, the initial space coordinate distance is obtained in the following manner:

[0019] The space coordinate point corresponding to the first monitoring node in the monitoring period of each part is obtained as the initial space coordinate, and the distance between the initial space coordinate of each part and the origin coordinate on the space trajectory monitoring model is obtained as the initial space coordinate distance through a two-dimensional coordinate distance formula.

[0020] As a further scheme of the present application, the trajectory change of the plurality of part conveying trajectories is predicted in the following process:

[0021] In the part conveying trajectory sequence, two part conveying trajectories are randomly selected as a target trajectory analysis group, and the endpoint coordinates on one of the part conveying trajectories and the endpoint coordinates on the other part conveying trajectory are respectively taken as fitting endpoint coordinates, and a reference trajectory fitting line and a comparison trajectory fitting line are fitted.

[0022] As a further scheme of the present application, the potential collision parts are identified in the following process:

[0023] The equations corresponding to the reference trajectory fitting line and the comparison trajectory fitting line are obtained respectively, which are the reference fitting equation and the comparison fitting equation, a simultaneous equation set is constructed, and the simultaneous equation set is solved.

[0024] If the simultaneous equation set has a real number solution, the target trajectory analysis group is marked as a potential collision group, and the parts corresponding to the potential collision group are marked as potential collision parts.

[0025] As a further scheme of the present application, after simulating the collision of the potential collision part, the following process is performed for analysis:

[0026] The post-collision coordinate points of the potential collision part are obtained respectively, and the real solution of the simultaneous equations is taken as the X coordinate at the time of collision, and substituted into the simultaneous equations to obtain the Y coordinate at the time of collision, as the coordinate point at the time of collision;

[0027] The post-collision coordinate points of the potential collision part are connected with the coordinate points at the time of collision respectively, to obtain two post-collision simulation trajectories, namely the reference post-collision simulation trajectory and the comparison post-collision simulation trajectory.

[0028] As a further scheme of the present application, the potential collision part is identified and distinguished, and the following process is performed:

[0029] The included angle between the reference post-collision simulation trajectory and the X axis on the space trajectory monitoring model, and the included angle between the comparison post-collision simulation trajectory and the X axis on the space trajectory monitoring model are obtained, and the trigonometric functions are used for calculation respectively to obtain the reference post-collision simulation slope corresponding to the reference post-collision simulation trajectory, and the comparison post-collision simulation slope corresponding to the comparison post-collision simulation trajectory.

[0030] As a further scheme of the present application, the process of identifying and distinguishing the results is as follows:

[0031] The grasping range on the space trajectory monitoring model of the robot is extracted, and the base model collision identification range and the comparison model collision identification range are obtained respectively;

[0032] If the reference post-collision simulation slope exists in the base model collision identification range, it is a collision non-offset part;

[0033] If the reference post-collision simulation slope does not exist in the base model collision identification range, it is a collision offset part;

[0034] If the comparison post-collision simulation slope exists in the comparison model collision identification range, it is a collision non-offset part;

[0035] If the comparison post-collision simulation slope does not exist in the comparison model collision identification range, it is a collision offset part.

[0036] As a further scheme of the present application, for the collision offset part, the grasping compensation amount is obtained, and the following process is performed:

[0037] The grasping range on the space trajectory monitoring model of the robot is extracted, and the center point of the grasping range is obtained, and a perpendicular line perpendicular to the post-collision simulation trajectory corresponding to the collision offset part is drawn, and the perpendicular point coordinates are extracted, and the perpendicular point coordinates and the center point coordinates are input into the coordinate point distance formula to output the offset distance value;

[0038] Subtract the offset distance value from the radius corresponding to the grabbing range, take the absolute value, and output the grabbing compensation amount.

[0039] As a further scheme of the application, for the collision non-offset part, the grabbing time of the collision non-offset part is prewarned to obtain a prewarning grabbing time, and the process is as follows:

[0040] Based on the simulation collision slope corresponding to the collision non-offset part, a straight line intersecting the grabbing range is drawn with the post-collision coordinate point as an auxiliary point, and a prewarning intersection point is extracted;

[0041] The prewarning intersection point and the post-collision coordinate point are input into the coordinate point distance formula to output a prewarning grabbing distance;

[0042] The instantaneous moving speed between the coordinate point and the post-collision coordinate point distance of the collision non-offset part at the corresponding collision is obtained, and mean value processing is performed to obtain a predicted moving speed;

[0043] The prewarning grabbing distance and the predicted moving speed are ratio calculated to output a prewarning grabbing time.

[0044] The beneficial effects of the application are as follows:

[0045] 1、The application can monitor the parts on the production conveying belt in real time within a monitoring period, obtain the part conveying trajectory, and construct a part conveying trajectory sequence, so that the movement trajectory of each part in space can be accurately presented through the part conveying trajectory sequence, the robot can intuitively understand the movement path of the part, the position information of the part at different time points can be accurately recorded, continuous trajectory data is formed, which is helpful for analyzing the movement trend of the part in the whole monitoring period, the trajectory change of multiple part conveying trajectories in the part conveying trajectory sequence is predicted, whether the part conveying trajectories in the part conveying trajectory sequence collide is predicted, so that whether two part conveying trajectories will intersect in space can be accurately judged, potential collision risks can be found in advance, and the robot grabbing is affected due to the collision of the part causing the trajectory offset.

[0046] 2、The present application simulates the potential collision parts and identifies and distinguishes the potential collision parts to obtain collision offset parts or collision non-offset parts, can accurately judge the offset of the part trajectory in space after collision, identify and distinguish the potential collision parts, know the trajectory change of the part after collision in advance, early warning of possible problems in production process, so that the robot can take targeted grabbing measures according to the identification result, for collision offset parts, obtain the grabbing compensation, the robot can accurately adjust the grabbing position according to the grabbing compensation, improve the accuracy of grabbing, reduce the failure of grabbing caused by part collision offset, for collision non-offset parts, the grabbing time of the collision non-offset parts is warned, the warning grabbing time is obtained, which is helpful for the robot to plan the grabbing time in advance, avoid missing the grabbing opportunity due to the change of part arrival time, reduce the part omission rate, ensure that the parts can be accurately placed in the assembly position of the production line, improve the assembly quality. BRIEF DESCRIPTION OF DRAWINGS

[0047] The present application will be further described below in conjunction with the drawings.

[0048] Figure 1 is a step flow chart of an industrial production line fault early warning method based on multi-modal data fusion of the present application;

[0049] Figure 2 is a condition judgment schematic diagram of an industrial production line fault early warning method based on multi-modal data fusion of the present application. DETAILED DESCRIPTION

[0050] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application will be further described below in conjunction with specific embodiments.

[0051] Embodiment 1

[0052] As shown in Figure 1 When the robot grabs parts on the production conveying belt, the parts on the production conveying belt may collide, causing the part trajectory on the production conveying belt to deviate, so that the robot cannot accurately grab the parts on the production conveying belt, causing part omission and assembly failure on the industrial production line. Therefore, the industrial production line fault early warning method based on multi-modal data fusion of the present application includes:

[0053] Step 1: Real-time monitoring of parts on the production conveying belt is performed within a monitoring period to obtain part conveying trajectories and construct a part conveying trajectory sequence.

[0054] It should be noted that the monitoring period is the total time of real-time monitoring of parts on the production conveying belt.

[0055] In some embodiments, a spatial trajectory monitoring model is constructed with the length and width of the production conveyor belt as the Y-axis and X-axis, respectively;

[0056] The monitoring period is equally divided into several monitoring nodes;

[0057] It should be noted that the interval time between adjacent monitoring nodes is equal;

[0058] In the spatial trajectory monitoring model, the spatial coordinate point of each part at each monitoring node is obtained Where j represents the jth part on the production conveyor belt, and i represents the coordinate point of the part on the production conveyor belt at the ith monitoring node;

[0059] All spatial coordinate points of the part in the monitoring period are connected to obtain the part transmission trajectory;

[0060] The part transmission trajectory corresponding to each part is sorted in ascending order according to the corresponding initial spatial coordinate distance, and a part transmission trajectory sequence is constructed;

[0061] Where the initial spatial coordinate distance is obtained as follows:

[0062] The spatial coordinate point corresponding to the first monitoring node in the monitoring period of each part is obtained as the initial spatial coordinate, and the distance between the initial spatial coordinate of each part and the origin coordinate on the spatial trajectory monitoring model is obtained by a two-dimensional coordinate distance formula as the initial spatial coordinate distance JL c ;

[0063] Specifically, the two-dimensional coordinate distance formula is: Where, represents the initial spatial coordinate corresponding to the jth part on the production conveyor belt;

[0064] In detail, the purpose of constructing the part transmission trajectory sequence is:

[0065] Purpose one: from the spatial dimension, the movement trajectory of each part in space can be accurately presented, so that the robot can intuitively understand the movement path of the part, and when the part trajectory deviates due to collision, the degree and direction of deviation can be found and judged in time, providing basis for robot adjustment of grabbing strategy, thereby improving the accuracy of robot grabbing parts and reducing the failure of grabbing due to part position deviation;

[0066] Purpose two: from the time dimension, the position information of the part at different time points can be accurately recorded to form continuous trajectory data, which is helpful for analyzing the movement trend of the part in the entire monitoring period and discovering potential collision risks and trajectory abnormalities in time;

[0067] Step 2: Predict trajectory changes for multiple part transport trajectories within the part transport trajectory sequence, predict whether collisions will occur between the part transport trajectories within the part transport trajectory sequence, and identify potentially colliding parts.

[0068] In some embodiments, two part transport trajectories are arbitrarily selected within the part transport trajectory sequence as a target trajectory analysis group;

[0069] Within the target trajectory analysis group, the endpoint coordinates of one of the part conveying trajectories are used as the fitted endpoint coordinates, where the fitted endpoint coordinates include the initial spatial coordinates and the tail point spatial coordinates of the part conveying trajectory.

[0070] Connect the initial spatial coordinates and the tail point spatial coordinates to obtain the baseline trajectory fitting line;

[0071] Similarly, the endpoint coordinates on another part conveying trajectory are used as the fitting endpoint coordinates, where the fitting endpoint coordinates include the initial spatial coordinates and the tail point spatial coordinates of the part conveying trajectory.

[0072] Connect the initial spatial coordinates and the tail point spatial coordinates to obtain the fitting line of the comparison trajectory;

[0073] Obtain the equations corresponding to the baseline trajectory fitting line and the comparison trajectory fitting line, which are the baseline fitting equation and the comparison fitting equation, respectively.

[0074] Wherein, the benchmark fitting equation is: y jz =K jz ×X+b jz K jz Let b represent the slope of the fitted line of the baseline trajectory. jz This is represented as the constant corresponding to the fitted line of the baseline trajectory;

[0075] Compare and fit the equation: y bd =K bd ×X+b bd K bd The slope of the fitted trajectory line is represented by b. bd This is represented as the constant corresponding to the fitted trajectory line;

[0076] Based on the benchmark fitting equation and the comparison fitting equation, a system of simultaneous equations is constructed and solved.

[0077] If the simultaneous equations have real solutions, it means that the baseline trajectory fitting line and the comparison trajectory fitting line intersect, and a collision warning signal is displayed. The target trajectory analysis group corresponding to the collision warning signal is marked as a potential collision group, and the parts corresponding to the potential collision group are marked as potential collision parts.

[0078] If the system of simultaneous equations has non-real solutions, it means that the baseline trajectory fitting line and the comparison trajectory fitting line do not intersect, and a non-collision warning signal is generated.

[0079] It should be noted that the purpose of solving a system of simultaneous equations is:

[0080] Objective 1: From a spatial perspective, it is possible to accurately determine whether two part transport trajectories will intersect in space, detect potential collision risks in advance, and avoid parts from being affected by trajectory deviation due to collisions, thus preventing the robot from grasping them.

[0081] Objective 2: From a time perspective, it can accurately record the position information of parts at different points in time, predict the movement trend of parts throughout the monitoring period, and promptly detect potential collision risks, providing the robot with a lead time to take corresponding measures.

[0082] Objective 3: From the perspective of individual parts, it is possible to analyze and compare the motion trajectories of each part separately, and accurately distinguish the motion states and collision risks of different parts;

[0083] The specific implementation plan of this embodiment is as follows: During the monitoring period, the parts on the production conveyor belt are monitored in real time to obtain the part conveying trajectory and construct a part conveying trajectory sequence. The part conveying trajectory sequence can accurately present the motion trajectory of each part in space, so that the robot can intuitively understand the motion path of the part. Moreover, it can accurately record the position information of the part at different time points, forming continuous trajectory data, which helps to analyze the motion trend of the part throughout the monitoring period. The trajectory change prediction is performed on multiple part conveying trajectories in the part conveying trajectory sequence to predict whether the part conveying trajectories in the part conveying trajectory sequence will collide. Thus, it can accurately determine whether two part conveying trajectories will intersect in space, detect potential collision risks in advance, and avoid the part from being affected by trajectory deviation due to collision, which would affect the robot's grasping.

[0084] Example 2

[0085] like Figure 1 As shown in Example 1, the industrial production line fault early warning method based on multimodal data fusion described in this embodiment of the invention includes:

[0086] Step 3: After simulating a collision with potential collision parts, analyze the results and identify and distinguish the potential collision parts to obtain the identification and distinction results;

[0087] The identification and differentiation results include collision-offset parts or collision-non-offset parts;

[0088] In some embodiments, after simulating a collision with a potential collision component, the post-collision coordinates of the potential collision component are obtained.

[0089] It should be noted that the potential collision part corresponds to a potential collision group, and there are two potential collision parts in the potential collision group, so after simulating the collision of the potential collision part, the post-collision coordinates of the two potential collision parts in the potential collision group are two;

[0090] The real solution of the simultaneous equations is taken as the X coordinate at the time of collision, and substituted into the simultaneous equations to obtain the Y coordinate at the time of collision as the coordinate point at the time of collision;

[0091] The post-collision coordinate points of the potential collision part are connected with the coordinate points at the time of collision respectively to obtain two post-collision trajectories, namely the reference post-collision trajectory and the comparison post-collision trajectory;

[0092] The angle between the reference post-collision trajectory and the X axis on the space trajectory monitoring model is obtained, and the trigonometric function is used for calculation to obtain the reference post-collision slope corresponding to the reference post-collision trajectory;

[0093] Similarly, the angle between the comparison post-collision trajectory and the X axis on the space trajectory monitoring model is obtained, and the trigonometric function is used for calculation to obtain the comparison post-collision slope corresponding to the comparison post-collision trajectory;

[0094] The grasping range on the space trajectory monitoring model where the robot is located is extracted;

[0095] It should be noted that the grasping range can be approximately circular;

[0096] For example, based on the post-collision coordinate point corresponding to the reference post-collision trajectory, two tangent lines tangent to the grasping range are drawn, marked as reference upper tangent line and reference lower tangent line;

[0097] The reference upper slope and the reference lower slope corresponding to the reference upper tangent line and the reference lower tangent line are obtained respectively to construct the reference post-collision identification range;

[0098] If the reference post-collision slope exists in the reference post-collision identification range, it means that the potential collision part corresponding to the reference post-collision trajectory has no trajectory deviation after collision, which is a non-deviation collision part;

[0099] If the reference post-collision slope does not exist in the reference post-collision identification range, it means that the potential collision part corresponding to the reference post-collision trajectory has trajectory deviation after collision, which is a deviation collision part;

[0100] Similarly, based on the post-collision coordinate point corresponding to the comparison post-collision trajectory, two tangent lines tangent to the grasping range are drawn, marked as comparison upper tangent line and comparison lower tangent line;

[0101] respectively, to construct a comparison simulation collision identification range;

[0102] If the comparison simulation collision slope exists in the comparison simulation collision identification range, it indicates that the potential collision part corresponding to the comparison simulation post-collision trajectory is not offset after collision, which is a collision non-offset part;

[0103] If the comparison simulation collision slope does not exist in the comparison simulation collision identification range, it indicates that the potential collision part corresponding to the comparison simulation post-collision trajectory is offset after collision, which is a collision offset part;

[0104] Step four: based on the identification and differentiation results, for the collision offset part, the grabbing compensation amount is obtained, for the collision non-offset part, the grabbing time of the collision non-offset part is pre-warned, the pre-warning grabbing time is obtained, and the part grabbing work on the conveying belt is completed;

[0105] In some embodiments, if the potential collision parts are all collision offset parts, the grabbing compensation amount is obtained, and the process is as follows:

[0106] Extract the grabbing range on the space trajectory monitoring model of the robot, and obtain the center point of the grabbing range;

[0107] Take the center point as an auxiliary point, draw a perpendicular line perpendicular to the post-collision trajectory corresponding to the collision offset part, extract the perpendicular point coordinates, and input the perpendicular point coordinates and the center point coordinates into the coordinate point distance formula to output the offset distance value;

[0108] Subtract the offset distance value from the radius corresponding to the grabbing range, take the absolute value, and output the grabbing compensation amount;

[0109] It should be noted that the post-collision trajectory includes the reference post-collision trajectory and the comparison post-collision trajectory

[0110] If the potential collision parts are all collision non-offset parts, the pre-warning grabbing time is obtained, and the process is as follows:

[0111] Take the simulation collision slope corresponding to the collision non-offset part as the basis, take the post-collision coordinate point as an auxiliary point, draw a straight line intersecting the grabbing range, and extract the pre-warning intersection point;

[0112] Input the pre-warning intersection point and the post-collision coordinate point into the coordinate point distance formula to output the pre-warning grabbing distance;

[0113] Obtain the instantaneous moving speed between the pre-collision coordinate point and the post-collision coordinate point of the collision non-offset part, and perform mean value processing to obtain the predicted moving speed;

[0114] The pre-warning grabbing distance is divided by the predicted moving speed to obtain a pre-warning grabbing time.

[0115] It should be noted that the simulation collision slope includes a reference simulation collision slope and a comparison simulation collision slope.

[0116] The embodiment specifically implements the following: the potential collision part is simulated to collide, and the potential collision part is identified and distinguished to obtain a collision offset part or a collision non-offset part, so that the offset of the part trajectory in space after collision can be accurately judged, the potential collision part is identified and distinguished, the trajectory change of the part after collision is known in advance, the problems that may occur in the production process are warned in advance, the robot can take targeted grabbing measures according to the identification result, the grabbing offset amount is obtained for the collision offset part, the robot can accurately adjust the grabbing position according to the grabbing offset amount, the accuracy of grabbing is improved, the grabbing failure caused by the collision offset of the part is reduced, the grabbing time of the collision non-offset part is warned for the collision non-offset part to obtain a pre-warning grabbing time, which is helpful for the robot to plan the grabbing time in advance, avoids missing the grabbing opportunity due to the change of the part arrival time, reduces the part omission rate, ensures that the part can be accurately placed in the assembly position of the production line, and improves the assembly quality.

[0117] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for early warning of industrial production line faults based on multimodal data fusion, characterized in that: include: During the monitoring period, the parts on the production conveyor belt are monitored in real time to construct a sequence of part conveying trajectories; The trajectory change prediction is performed on multiple part transport trajectories within the part transport trajectory sequence to predict whether a collision will occur and to identify potentially colliding parts. The identification and differentiation results are obtained by simulating collisions with potential collision parts and then analyzing the results. The identification and differentiation results include collision-offset parts or collision-non-offset parts; For collision-offset parts, obtain the gripping compensation amount; for collision-non-offset parts, issue an early warning for the gripping time of the collision-non-offset parts, obtain the early warning gripping time, and complete the gripping of parts on the conveyor belt.

2. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The process of constructing the part transfer trajectory sequence is as follows: The monitoring cycle is divided into several monitoring nodes of equal length; Construct a spatial trajectory monitoring model, obtain the spatial coordinates of each part at each monitoring node within the spatial trajectory monitoring model, and connect all spatial coordinates of the part within the monitoring cycle to obtain the part's transmission trajectory; The part transport trajectory corresponding to each part is sorted in ascending order according to the distance of the corresponding initial spatial coordinates to construct a part transport trajectory sequence.

3. The industrial production line fault early warning method based on multimodal data fusion according to claim 2, characterized in that: The initial spatial coordinate distance is obtained as follows: Obtain the spatial coordinates of the first monitoring node for each part within the monitoring cycle as the initial spatial coordinates, and use the two-dimensional coordinate distance formula to obtain the distance between the initial spatial coordinates of each part and the coordinates of the origin on the spatial trajectory monitoring model as the initial spatial coordinate distance.

4. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The process of predicting trajectory changes for multiple parts during transport is as follows: Within the part transport trajectory sequence, two part transport trajectories are arbitrarily selected as the target trajectory analysis group. The endpoint coordinates of one part transport trajectory and the endpoint coordinates of the other part transport trajectory are used as the fitting endpoint coordinates to obtain the baseline trajectory fitting line and the comparison trajectory fitting line.

5. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The process for identifying potentially colliding parts is as follows: Obtain the equations corresponding to the baseline trajectory fitting line and the comparison trajectory fitting line respectively, which are the baseline fitting equation and the comparison fitting equation. Construct a system of simultaneous equations and solve the system of simultaneous equations. If the system of simultaneous equations has a real solution, then the target trajectory analysis group is marked as the potential collision group, and the parts corresponding to the potential collision group are marked as potential collision parts.

6. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The analysis is performed after simulating a collision with the potentially colliding parts, as follows: The post-collision coordinates of the potential colliding parts are obtained respectively, and the real solution of the simultaneous equations is used as the X coordinate at the time of collision. The X coordinate at the time of collision is then substituted into the simultaneous equations to obtain the Y coordinate at the time of collision, which is used as the coordinate point at the time of collision. By connecting the post-collision coordinates of the potential colliding parts with the post-collision coordinates, two simulated post-collision trajectories are obtained: the baseline simulated post-collision trajectory and the comparison simulated post-collision trajectory.

7. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The process for identifying and distinguishing potential collision components is as follows: Obtain the angle between the baseline simulated collision trajectory and the X-axis of the spatial trajectory monitoring model, as well as the angle between the comparison simulated collision trajectory and the X-axis of the spatial trajectory monitoring model. Calculate the baseline simulated collision slope corresponding to the baseline simulated collision trajectory and the comparison simulated collision slope corresponding to the comparison simulated collision trajectory using trigonometric functions.

8. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: The process of identifying and distinguishing the results is as follows: Extract the grasping range on the spatial trajectory monitoring model of the robot, and obtain the basic model collision recognition range and the comparative model collision recognition range respectively; If the baseline simulated collision slope exists within the baseline collision identification range, then it is a non-offset collision part; If the baseline simulated collision slope is not within the baseline collision identification range, then it is a collision offset part; If the slope of the simulated collision is within the range of the collision identification, then it is a non-offset collision part; If the slope of the simulated collision is not within the range of the collision identification, then it is a collision offset part.

9. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: For collision-offset parts, the grab compensation amount is obtained as follows: Extract the grasping range on the spatial trajectory monitoring model where the robot is located, obtain the center point of the grasping range, draw a perpendicular line to the simulated trajectory after the collision of the offset part, extract the coordinates of the perpendicular point, and input the coordinates of the perpendicular point and the center point into the coordinate point distance formula to output the deviation distance value. The difference between the deviation distance value and the radius corresponding to the grab range is taken, and the absolute value is output as the grab compensation amount.

10. The industrial production line fault early warning method based on multimodal data fusion according to claim 1, characterized in that: For collision-free non-offset parts, an early warning is issued regarding the grasping time of these parts, and the early warning grasping time is obtained. The process is as follows: Based on the simulated collision slope corresponding to the non-offset part, the coordinate point after the collision is used as an auxiliary point to draw a straight line that intersects with the grab range, and the warning intersection point is extracted. Input the intersection point of the warning and the coordinate point after the collision into the coordinate point distance formula, and the output will be the warning capture distance; The instantaneous moving speed of the non-offset part during the collision is obtained between the distance between the coordinate point at the time of the collision and the coordinate point after the collision, and then averaged to obtain the expected moving speed. The ratio of the warning capture distance to the expected moving speed is calculated, and the warning capture time is output.

Images