An assembly misoperation detection method based on video data

Abstract

The application provides an assembly misoperation detection method based on video data, and the application extracts operation actions and operation paths by aiming at operation videos of users on equipment, and operation failure risk levels of each operation action are calculated; aiming at equipment assembly misoperation, misoperation reason nodes are located according to misoperation action fault trees; and equipment failure types possibly caused by misoperation are determined according to operation failure risk levels of each misoperation action and failure reason nodes; the application can effectively improve the accuracy of equipment assembly misoperation diagnosis, and can also compare non-standard assembly misoperation action data with a standard assembly operation video database as a benchmark, find abnormal data for early warning, avoid abnormalities caused by assembly misoperation, and ensure assembly quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of assembly misoperation detection, and particularly relates to an assembly misoperation detection method based on video data. BACKGROUND

[0002] In the production line of an enterprise, an operator needs to operate a device according to a fixed process to realize product processing; in the use scene of a finished device, an operator often needs to operate and use the finished device according to a fixed operation step. However, when the operator operates the device, due to lack of understanding of the operation process or non-standard operation of the device parts, misoperation often occurs, which often leads to device failure. However, for the operator, when misoperation leads to device failure, the operator often cannot lock the operation process or operation action that causes the problem in multiple operation processes, resulting in production downtime or low efficiency.

[0003] To solve the above problems, the present application provides an assembly misoperation detection method based on video data. SUMMARY

[0004] In view of the problem that, when an operator misoperates a device and the device fails, the operator cannot effectively locate the failure cause and cannot detect misoperation, the present application provides an assembly misoperation detection method based on video data, which comprises the following steps:

[0005] S1. Obtain operation video data of a user for a device, wherein the operation video data is video data of the user operating the device according to a predetermined operation process;

[0006] S2. Obtain multiple single operation actions and operation paths of the user for the device in the operation video data;

[0007] The operation path is a sequence of multiple single operation actions of the user for the device;

[0008] The single operation action is an action process of the user for single operation of the device;

[0009] S3. Calculate a failure risk level of the user operation according to the multiple single operation actions and operation paths of the user for the device;

[0010] S4. When the device fails, locate a failure cause node in an operation action fault tree according to a device failure type;

[0011] S5. Determine a device assembly misoperation cause according to the failure risk level calculation result in step S3 and the failure cause node in step S4.

[0012] Step S1, acquiring user operation video data on the device, specifically includes:

[0013] A 3D camera is used to acquire video data of the user's operation of the device.

[0014] Step S2, which obtains multiple single-action actions and operation paths of the user on the device from the operation video data, specifically includes the following steps:

[0015] S21. Based on the three-dimensional coordinate information of each point of the device in the video image, perform three-dimensional modeling of the device to form a three-dimensional model of the device;

[0016] The three-dimensional model of the device includes the three-dimensional coordinate information of each point in the device in the video image;

[0017] The three-dimensional model of the device includes the three-dimensional position information of each component in the video image;

[0018] The three-dimensional position information of each component includes the component name and component marker box;

[0019] The component marking box is the area in the video image where the component is marked with a rectangular frame;

[0020] S22. Identify the three-dimensional region of the user's hand in the video image, and dynamically track the three-dimensional region of the hand;

[0021] S23. When the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model, record the start time of a single operation and the video frame corresponding to the start time of the single operation.

[0022] The hand's movements are continuously tracked until the three-dimensional region of the hand does not overlap with the three-dimensional region of the three-dimensional model. The end time of a single operation and the corresponding video frame are then recorded.

[0023] The video segment between the video frame corresponding to the start time of the single operation and the video frame corresponding to the end time of the single operation is taken as the video segment of this single operation.

[0024] S24. Repeat steps S22 to S23 until the user completes all operation procedures and obtains video clips of multiple single operation actions;

[0025] S25. The video segments of the multiple single operation actions are segmented to form multiple segmented video segments, which serve as the multiple single operation actions in step S2.

[0026] S26. Combine the multiple video segments from step S25 in chronological order to form the operation path in step S2.

[0027] Step S23 specifically includes the following steps:

[0028] S231. When the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model, the region where the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model is obtained;

[0029] S232. Obtain the component names and component marker boxes corresponding to the overlapping areas in the video image;

[0030] S233. Determine the first region based on the component marking frame and the three-dimensional area size of the user's hand in step S232;

[0031] S234. Adjust the shooting angle and focal length of the 3D camera according to the first area determined in step S233.

[0032] Step S233 determines the first region based on the component marking frame in step S232 and the three-dimensional area size of the user's hand, specifically including the following steps:

[0033] S2331. Overlay the three-dimensional area of ​​the user's hand with the component marking frame to form a second area;

[0034] S2332. Obtain the maximum width of the second region in the horizontal direction and the maximum height in the vertical direction, and set the first region on the video image according to the maximum width and maximum height; the first region completely covers the second region.

[0035] Step S234 adjusts the shooting angle and focal length of the 3D camera according to the first region determined in step S233, specifically including the following steps:

[0036] S2341. Obtain the center position of the first region;

[0037] S2342. Adjust the angle of the 3D camera to move the center position of the first area to the center position of the video image;

[0038] S2343. Calculate the area ratio A of the component marking frame in the first region to the area of ​​the first region; divide the area of ​​the first region by the area of ​​the preset region to obtain the ratio B of the area of ​​the first region to the area of ​​the preset region.

[0039] When the area ratio A is greater than the preset threshold T1, the focal length of the 3D camera is adjusted so that the width of the first region is equal to the width of the preset region.

[0040] When the area ratio A is less than the preset threshold T1, the product C of the ratio A and the proportion B is calculated. When the product C is greater than the preset threshold T2, the focal length of the 3D camera is adjusted so that the width of the first region is equal to the width of the preset region.

[0041] When the area ratio A is less than the preset threshold T1, the product C of the ratio A and the proportion B is calculated. When the product C is less than the preset threshold T2, the focal length of the 3D camera is adjusted so that the area ratio of the component marking box in the first region to the area ratio of the preset region is equal to the preset threshold T2.

[0042] Step S232, which obtains the component names and component marker boxes corresponding to the overlapping areas in the video image, specifically includes the following steps:

[0043] S2321. When the overlapping area falls within a component marking frame, the component marking frame and the component name are used as the component marking frame and component name in step S232;

[0044] When the overlapping area falls within two or more component marking frames, the overlapping area is compared with the overlapping area of ​​the two or more component marking frames respectively, and the component name and component marking frame corresponding to the area with the largest overlapping area are taken as the component name and component marking frame in step S232.

[0045] Step S25 involves segmenting the video clips of the multiple single-operation actions to form multiple segmented video clips, specifically including:

[0046] S251. Obtain one video segment from multiple single-operation video clips, obtain multiple video frame images in the video segment, and obtain the name of the component corresponding to the position where the three-dimensional region of the hand and the three-dimensional region of the three-dimensional model overlap in each video frame image;

[0047] S252. Traverse the component names in the multiple video frame images in step S251. When the component name changes, segment the video segment; divide the video segment into two or more video segments; and record the start time and end time of the operation of the segmented video segments.

[0048] If the name of the component remains unchanged, the video segment will not be segmented.

[0049] S253. For video segments with multiple single operation actions, repeat steps S251 to S252 until all video segments with single operation actions have been processed, forming multiple segmented video segments.

[0050] The video clip includes operation video data, component names, operation start time, and operation end time;

[0051] Each segmented video fragment corresponds to a single operation in step S2.

[0052] Step S26 combines the multiple video segments from step S25 in chronological order to form the operation path in step S2, specifically including the following steps:

[0053] S261. Combine multiple video segments according to the order of their start and end times to form an operation path;

[0054] S262. Label each video segment in the operation path formed in step S261 with numbers in ascending order of time.

[0055] Step S3 calculates the fault risk level of the user's operation based on the user's multiple single operation actions and operation paths on the device, specifically including the following steps:

[0056] S31. Obtain the video segment corresponding to a single operation action and the name of the component corresponding to the single operation action, and obtain the guiding operation action corresponding to the component name;

[0057] The guided operation actions are pre-stored guided operation video clips;

[0058] S32. Compare the video clips corresponding to the single operation action obtained in step S31 with the video clips corresponding to the guided operation action to obtain a type of operation risk value;

[0059] S33. For all single operation actions, execute steps S31 to S32 to obtain the operation risk value of one type for all single operation actions;

[0060] S34. Based on the operation path and the guided operation path, obtain the Class II operation risk values ​​for all individual operation actions;

[0061] The guided operation path is a pre-stored sequence of guided operation actions arranged in chronological order;

[0062] Each guided operation action in the guided operation path is labeled with a numerical number in ascending order according to the chronological sequence.

[0063] S35. Based on the Class I operational risk values ​​of all single operation actions obtained in step S33 and the Class II operational risk values ​​obtained in step S34, obtain the fault risk level of all single operation actions.

[0064] Step S32 compares the video segment corresponding to the single operation action obtained in step S31 with the guidance operation video segment corresponding to the guidance operation action to obtain a type of operation risk value, which specifically includes the following steps:

[0065] S321. Obtain the sequence of user hand actions and operation durations for the user's hand on the operation points of the components in the instruction operation video clip in chronological order;

[0066] S322. Based on the operation point location obtained in step S321, query the user's hand operation sequence and operation duration corresponding to the operation point location in the video segment corresponding to the single operation action obtained in step S31;

[0067] S323. Calculate the similarity of the user's hand operation sequence in step S321 according to the pose similarity comparison algorithm to obtain the operation similarity D;

[0068] S324. Calculate the similarity between the operation duration in step S321 and the operation duration in step S322 to obtain the operation duration similarity I;

[0069] S325. Based on the similarity value D of the operation action and the similarity value I of the operation duration, calculate the value of a class of operation risk.

[0070] The method for calculating the operation duration similarity I in step S324 is as follows:

[0071] Compare the operation time obtained in step S321 with the operation time obtained in step S322, and use the ratio of the minimum value to the maximum value as the operation time similarity I.

[0072] Step S325 calculates a type of operational risk value based on the similarity value D of the operation action and the similarity value I of the operation duration, specifically including:

[0073] F1=δ1D+δ2I

[0074] Where F1 is a type of operational risk value; δ1 and δ2 are the first weight and the second weight, respectively; where δ1>δ2.

[0075] Step S34 obtains the Class II operational risk values ​​for all individual operational actions based on the operational path and the guided operational path, specifically including:

[0076] S341. Obtain the sequence of operation actions in the operation path, and obtain the following data structure S based on the sequence of operation actions: {S1, J1, S2, ... S...} i J i ,…,J n-1,S n};

[0077] Si stores the name of the component corresponding to the i-th operation.

[0078] Ji represents the interval between the i-th operation and the (i+1)-th operation.

[0079] The number of operations in the operation path is n;

[0080] Obtain the sequence of guided operation actions in the guided operation path, and obtain the following data structure ZS based on the sequence of operation actions: {ZS1, ZJ1, ZS2, ... ZS...} i ZJ i ,…,ZJ n-1 ZS n};

[0081] Among them, ZS i The name of the component corresponding to the i-th guided operation action is stored in the middle;

[0082] ZJ i The interval between the i-th guided action and the (i+1)-th guided action;

[0083] The number of guided operation actions in the guided operation path is n;

[0084] S342. Calculate the similarity based on data structure S and data structure ZS to obtain the value of the second type of operational risk.

[0085] Step S342 specifically includes the following steps:

[0086] S3421. Set the operation action adjustment coefficient array E{E1, E2, ..., E...} i ,…,E n};

[0087] S3422. Regarding S i When S i With ZS i When the component names are inconsistent, set S i The corresponding adjustment factor E i This is the first adjustment factor value;

[0088] When S i With ZS i The component names are consistent, J i-1 With ZJ i-1 When the values ​​are inconsistent, set S i The corresponding adjustment factor E i This is the second adjustment factor value;

[0089] When Si With ZS i The component names are consistent, J i-1 With ZJ i-1 When the values ​​are the same, set S. i The corresponding adjustment factor E i This is the third adjustment factor value;

[0090] The first adjustment coefficient value is greater than the second adjustment coefficient value;

[0091] The second adjustment factor value is greater than the third adjustment factor value;

[0092] The third adjustment coefficient is 1;

[0093] S3423. Adjust the operation action coefficient array E{E1, E2, ..., E i E n} is the second type of operational risk value mentioned in step S342.

[0094] Step S35 obtains the fault risk level of all single operation actions based on the Class I operation risk values ​​obtained in step S33 and the Class II operation risk values ​​obtained in step S34, specifically including:

[0095] S351. Multiply the value of a single operation risk by the operation adjustment coefficient corresponding to the single operation, and use the resulting multiplication value as the fault risk level corresponding to the single operation.

[0096] S352. For all single operation actions, repeat step S351 to obtain the fault risk level of each single operation action.

[0097] When a device malfunctions, step S4 involves locating the cause node in the operation action fault tree based on the device malfunction type. Specifically, this includes:

[0098] S41. Construct a fault tree for operation actions based on the user's historical operation data of the device;

[0099] The fault tree includes multiple subtrees;

[0100] Each subtree includes a root node, intermediate nodes, and leaf nodes;

[0101] The root node of the subtree stores the device fault type, while the intermediate nodes and leaf nodes are fault cause nodes; the fault cause nodes store the erroneous operation type for the component.

[0102] The root node is directed to the intermediate node by a directed path, and the intermediate node is directed to the leaf node by a directed path.

[0103] The directed path is a path with a correlation value; the correlation value is the correlation between two nodes.

[0104] S42. When equipment malfunctions, locate the fault cause node in the operation action fault tree according to the equipment fault type;

[0105] Step S42, which locates the fault cause node, specifically includes the following steps:

[0106] S421. Based on the equipment fault type, locate the root node of one of the multiple subtrees in the fault tree, and take the root node as the current node;

[0107] S422. When the current node points to multiple lower-level nodes, obtain these multiple lower-level nodes as the fault cause nodes;

[0108] When the current node points to a single subordinate node, the subordinate node is taken as the current node, and step S422 is repeated until the current node points to multiple subordinate nodes, and these multiple subordinate nodes are taken as the fault cause node; when the current node has no subordinate nodes after repeating step S422, the current node is taken as the fault cause node.

[0109] Step S5 determines the cause of equipment failure based on the failure risk level calculation result in step S3 and the failure cause node in step S4, and specifically includes the following steps:

[0110] S51. Obtain the fault risk level calculation result in step S3, and sort the fault risk level results in descending order to obtain the component names and video clips corresponding to the first three fault risk level results in the sorting results.

[0111] S52. Compare the component names corresponding to the first three fault risk level fault results obtained in step S51 with the component names corresponding to the fault cause nodes obtained in step S4. If the component name is among the component names corresponding to the first three fault risk level fault results obtained in step S51 and the component names corresponding to the fault cause nodes obtained in step S4, then provide feedback to the user as the cause of the equipment assembly misoperation.

[0112] The beneficial effects of this invention are as follows:

[0113] 1. In this invention, the user's operation is described based on a single operation action and operation path, and the fault risk level is calculated based on the two factors of operation action and operation path, which improves the accuracy of the user's diagnosis of equipment faults caused by equipment operation errors.

[0114] 2. In this invention, a video operation segment is obtained when the user's hand overlaps with the component area; and the video operation segment is segmented according to the operation on different components in the video operation segment to form multiple segmented video segments, which provides accurate data support for the comparative analysis of subsequent operation actions and further improves the accuracy of identifying user operation errors.

[0115] 3. In this invention, when the user's hand overlaps with the component area, the user's hand area and the component area are adjusted to the middle area of ​​the video image, and the angle and focal length of the camera are adjusted according to the size of the component and the preset area size, so that the video image can clearly and completely record the user's hand operation, thereby improving the accuracy of identifying user operation errors.

[0116] 4. This invention obtains a first-class operation risk value by comparing and analyzing a single operation action with the guided operation action, obtains a second-class operation risk value by comparing and analyzing the operation path with the guided operation path, and obtains the fault risk level based on the first-class and second-class operation risk values. The above algorithm comprehensively considers many factors such as the standardization of operation actions, the duration, and the interval between operation actions, thereby improving the accuracy of identifying user operation errors.

[0117] 5. This invention constructs an operation action fault tree based on the user's historical operation data, and locates the fault cause node based on the operation action fault tree, thereby improving the accuracy of fault cause diagnosis and assembly misoperation diagnosis;

[0118] 6. This invention combines the fault risk level with the fault tree method to obtain the final cause of equipment failure, thereby improving the accuracy of equipment failure cause diagnosis and assembly misoperation diagnosis.

[0119] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above description and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are provided and described in detail below. Attached image description:

[0120] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0121] Figure 1 This is a flowchart of an assembly misoperation detection method based on video data. Detailed implementation method:

[0122] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0123] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0124] This paper proposes a method for detecting assembly errors based on video data. The method includes the following steps:

[0125] S1. Acquire user operation video data of the device, wherein the operation video data is video data of the current user operating the device according to a predetermined operation procedure;

[0126] Further, step S1, acquiring user operation video data on the device, specifically includes:

[0127] A 3D camera is used to acquire video data of the user's operation of the device.

[0128] S2. Obtain multiple single operation actions and operation paths of the user on the device from the operation video data;

[0129] The operation path is a sequence of multiple single operation actions performed by the user on the device;

[0130] The single operation action refers to the process by which a user performs a single operation on the device;

[0131] Step S2, which obtains multiple single-action actions and operation paths of the user on the device from the operation video data, specifically includes the following steps:

[0132] S21. Based on the three-dimensional coordinate information of each point of the device in the video image, perform three-dimensional modeling of the device to form a three-dimensional model of the device;

[0133] The three-dimensional model of the device includes the three-dimensional coordinate information of each point in the device in the video image;

[0134] The three-dimensional model of the device includes the three-dimensional position information of each component in the video image;

[0135] The three-dimensional position information of each component includes the component name and component marker box;

[0136] The component marking box is the area in the video image where the component is marked with a rectangular frame;

[0137] S22. Identify the three-dimensional region of the user's hand in the video image, and dynamically track the three-dimensional region of the hand;

[0138] S23. When the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model, record the start time of a single operation and the video frame corresponding to the start time of the single operation.

[0139] The hand's movements are continuously tracked until the three-dimensional region of the hand does not overlap with the three-dimensional region of the three-dimensional model. The end time of a single operation and the corresponding video frame are then recorded.

[0140] The video segment between the video frame corresponding to the start time of the single operation and the video frame corresponding to the end time of the single operation is taken as the video segment of this single operation.

[0141] Step S23 specifically includes the following steps:

[0142] S231. When the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model, the region where the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model is obtained;

[0143] S232. Obtain the component names and component marker boxes corresponding to the overlapping areas in the video image;

[0144] Further, step S232, obtaining the component names and component marker boxes corresponding to the overlapping areas in the video image, specifically includes the following steps:

[0145] S2321. When the overlapping area falls within a component marking frame, the component marking frame and the component name are used as the component marking frame and component name in step S232;

[0146] When the overlapping area falls within two or more component marking frames, the overlapping area is compared with the overlapping area of ​​the two or more component marking frames respectively, and the component name and component marking frame corresponding to the area with the largest overlapping area are taken as the component name and component marking frame in step S232.

[0147] S233. Determine the first region based on the component marking frame and the three-dimensional area size of the user's hand in step S232;

[0148] Further, step S233 determines the first region based on the component marking frame in step S232 and the three-dimensional region size of the user's hand, specifically including the following steps:

[0149] S2331. Overlay the three-dimensional area of ​​the user's hand with the component marking frame to form a second area;

[0150] S2332. Obtain the maximum width of the second region in the horizontal direction and the maximum height in the vertical direction, and set the first region on the video image according to the maximum width and maximum height; the first region completely covers the second region.

[0151] S234. Adjust the shooting angle and focal length of the 3D camera according to the first area determined in step S233.

[0152] Furthermore, step S234 adjusts the shooting angle and focal length of the 3D camera according to the first region determined in step S233, specifically including the following steps:

[0153] S2341. Obtain the center position of the first region;

[0154] S2342. Adjust the angle of the 3D camera to move the center position of the first area to the center position of the video image;

[0155] S2343. Calculate the area ratio A of the component marking frame in the first region to the area of ​​the first region; divide the area of ​​the first region by the area of ​​the preset region to obtain the ratio B of the area of ​​the first region to the area of ​​the preset region.

[0156] When the area ratio A is greater than the preset threshold T1, the focal length of the 3D camera is adjusted so that the width of the first region is equal to the width of the preset region.

[0157] When the area ratio A is less than the preset threshold T1, the product C of the ratio A and the proportion B is calculated. When the product C is greater than the preset threshold T2, the focal length of the 3D camera is adjusted so that the width of the first region is equal to the width of the preset region.

[0158] When the area ratio A is less than the preset threshold T1, the product C of the ratio A and the proportion B is calculated. When the product C is less than the preset threshold T2, the focal length of the 3D camera is adjusted so that the area ratio of the component marking box in the first region to the area ratio of the preset region is equal to the preset threshold T2.

[0159] The preset threshold T1 is a value between 80% and 100%.

[0160] The preset threshold T2 is a value between 70% and 80%.

[0161] The preset area is a pre-set area size, and the preset area includes a preset width and a preset height;

[0162] S24. Repeat steps S22 to S23 until the user completes all operation procedures and obtains video clips of multiple single operation actions;

[0163] S25. The video segments of the multiple single operation actions are segmented to form multiple segmented video segments, which serve as the multiple single operation actions in step S2.

[0164] Further, step S25 involves segmenting the video clips of the multiple single-operation actions to form multiple segmented video clips, specifically including:

[0165] S251. Obtain one video segment from multiple single-operation video clips, obtain multiple video frame images in the video segment, and obtain the name of the component corresponding to the position where the three-dimensional region of the hand and the three-dimensional region of the three-dimensional model overlap in each video frame image;

[0166] S252. Traverse the component names in the multiple video frame images in step S251. When the component name changes, segment the video segment; divide the video segment into two or more video segments; and record the start time and end time of the operation of the segmented video segments.

[0167] If the name of the component remains unchanged, the video segment will not be segmented.

[0168] S253. For video segments with multiple single operation actions, repeat steps S251 to S252 until all video segments with single operation actions have been processed, forming multiple segmented video segments.

[0169] The video clip includes operation video data, component names, operation start time, and operation end time;

[0170] Each segmented video fragment corresponds to a single operation in step S2.

[0171] S26. Combine the multiple video segments from step S25 in chronological order to form the operation path in step S2.

[0172] Furthermore, step S26 combines the multiple video segments from step S25 in chronological order to form the operation path in step S2, specifically including the following steps:

[0173] S261. Combine multiple video segments according to the order of their start and end times to form an operation path;

[0174] S262. Label each video segment in the operation path formed in step S261 with numbers in ascending order of time.

[0175] For example, if there are 6 video clips in the operation path, the 6 video clips are numbered 1, 2, 3, 4, 5, 6 in chronological order.

[0176] S3. Calculate the fault risk level of the user's operation based on the user's multiple single operation actions and operation paths on the device;

[0177] Furthermore, step S3 calculates the fault risk level of the user's operation based on the user's multiple single operation actions and operation paths on the device, specifically including the following steps:

[0178] S31. Obtain the video segment corresponding to a single operation action and the name of the component corresponding to the single operation action, and obtain the guiding operation action corresponding to the component name;

[0179] The guided operation actions are pre-stored video clips of guided operation actions;

[0180] S32. Compare the video clips corresponding to the single operation action obtained in step S31 with the video clips corresponding to the guided operation action to obtain a type of operation risk value;

[0181] Further, step S32 compares the video segment corresponding to the single operation action obtained in step S31 with the guidance operation video segment corresponding to the guidance operation action to obtain a type of operation risk value, specifically including the following steps:

[0182] S321. Obtain the sequence of user hand actions and operation durations for the user's hand on the operation points of the components in the instruction operation video clip in chronological order;

[0183] S322. Based on the operation point location obtained in step S321, query the user's hand operation sequence and operation duration corresponding to the operation point location in the video segment corresponding to the single operation action obtained in step S31;

[0184] S323. Calculate the similarity of the user's hand operation sequence in step S321 according to the pose similarity comparison algorithm to obtain the operation similarity D;

[0185] The pose similarity comparison algorithm uses conventional algorithms, such as Pr-VIPE.

[0186] S324. Calculate the similarity between the operation duration in step S321 and the operation duration in step S322 to obtain the operation duration similarity I;

[0187] Furthermore, the method for calculating the operation duration similarity I in step S324 is as follows:

[0188] Compare the operation time obtained in step S321 with the operation time obtained in step S322, and use the ratio of the minimum value to the maximum value as the operation time similarity I.

[0189] S325. Based on the similarity value D of the operation action and the similarity value I of the operation duration, calculate the value of a class of operation risk.

[0190] Further, step S325 calculates a type of operational risk value based on the similarity value D of the operation action and the similarity value I of the operation duration, specifically including:

[0191] F1=δ1D+δ2I

[0192] Where F1 is a type of operational risk value; δ1 and δ2 are the first weight and the second weight, respectively; where δ1>δ2.

[0193] S33. For all single operation actions, execute steps S31 to S32 to obtain the operation risk value of one type for all single operation actions;

[0194] S34. Based on the operation path and the guided operation path, obtain the Class II operation risk values ​​for all individual operation actions;

[0195] The guided operation path is a pre-stored sequence of guided operation actions arranged in chronological order;

[0196] Each guided operation action in the guided operation path is labeled with a numerical number in ascending order according to the chronological sequence.

[0197] Further, step S34 obtains the Class II operational risk values ​​for all individual operational actions based on the operational path and the guided operational path, specifically including:

[0198] S341. Obtain the sequence of operation actions in the operation path, and obtain the following data structure S based on the sequence of operation actions: {S1, J1, S2, ... S...} i J i ,…,J n-1 ,S n};

[0199] Si stores the name of the component corresponding to the i-th operation.

[0200] Ji represents the interval between the i-th operation and the (i+1)-th operation.

[0201] The number of operations in the operation path is n;

[0202] Obtain the sequence of guided operation actions in the guided operation path, and obtain the following data structure ZS based on the sequence of operation actions: {ZS1, ZJ1, ZS2, ... ZS...} i ZJ i ,…,ZJ n-1 ZS n};

[0203] Among them, ZS i The name of the component corresponding to the i-th guided operation action is stored in the middle;

[0204] ZJ i The interval between the i-th guided action and the (i+1)-th guided action;

[0205] The number of guided operation actions in the guided operation path is n;

[0206] S342. Calculate the similarity based on data structure S and data structure ZS to obtain the value of the second type of operational risk.

[0207] Furthermore, step S342 specifically includes the following steps:

[0208] S3421. Set the operation action adjustment coefficient array E{E1, E2, ..., E...} i ,…,E n};

[0209] S3422. Regarding S i When S i With ZSi When the component names are inconsistent, set S i The corresponding adjustment factor E i This is the first adjustment factor value;

[0210] When S i With ZS i The component names are consistent, J i-1 With ZJ i-1 When the values ​​are inconsistent, set S i The corresponding adjustment factor E i This is the second adjustment factor value;

[0211] When S i With ZS i The component names are consistent, J i-1 With ZJ i-1 When the values ​​are the same, set S. i The corresponding adjustment factor E i This is the third adjustment factor value;

[0212] For S1 and ZS1, when the part names of S1 and ZS1 are the same, set the adjustment coefficient E1 corresponding to S1 to the third adjustment coefficient value.

[0213] The first adjustment coefficient value is greater than the second adjustment coefficient value;

[0214] The second adjustment factor value is greater than the third adjustment factor value;

[0215] The first adjustment coefficient value is between 1.3 and 1.5;

[0216] The second adjustment coefficient value is between 1.1 and 1.3;

[0217] The third adjustment coefficient is 1;

[0218] S3423. Adjust the operation action coefficient array E{E1, E2, ..., E i E n} is the second type of operational risk value mentioned in step S342.

[0219] S35. Based on the Class I operational risk values ​​of all single operation actions obtained in step S33 and the Class II operational risk values ​​obtained in step S34, obtain the fault risk level of all single operation actions.

[0220] Further, step S35 obtains the fault risk level of all single operation actions based on the Class I operation risk values ​​obtained in step S33 and the Class II operation risk values ​​obtained in step S34, specifically including:

[0221] S351. Multiply the value of a single operation risk by the operation adjustment coefficient corresponding to the single operation, and use the resulting multiplication value as the fault risk level corresponding to the single operation.

[0222] S352. For all single operation actions, repeat step S351 to obtain the fault risk level of each single operation action.

[0223] S4. When equipment malfunctions, locate the fault cause node in the operation action fault tree according to the equipment fault type;

[0224] Furthermore, in step S4, when a device malfunctions, the cause node of the malfunction is located in the operation action fault tree according to the type of device malfunction, specifically including:

[0225] S41. Construct a fault tree for operation actions based on the user's historical operation data of the device;

[0226] The fault tree includes multiple subtrees;

[0227] Each subtree includes a root node, intermediate nodes, and leaf nodes;

[0228] The root node of the subtree stores the device fault type, while the intermediate nodes and leaf nodes are fault cause nodes; the fault cause nodes store the erroneous operation type for the component.

[0229] The root node is directed to the intermediate node by a directed path, and the intermediate node is directed to the leaf node by a directed path.

[0230] The directed path is a path with a correlation value; the correlation value is the correlation between two nodes.

[0231] S42. When equipment malfunctions, locate the fault cause node in the operation action fault tree according to the equipment fault type;

[0232] Furthermore, step S42, which locates the fault cause node, specifically includes the following steps:

[0233] S421. Based on the equipment fault type, locate the root node of one of the multiple subtrees in the fault tree, and take the root node as the current node;

[0234] S422. When the current node points to multiple lower-level nodes, obtain these multiple lower-level nodes as the fault cause nodes;

[0235] When the current node points to a single subordinate node, the subordinate node is taken as the current node, and step S422 is repeated until the current node points to multiple subordinate nodes, and these multiple subordinate nodes are taken as the fault cause node; when the current node has no subordinate nodes after repeating step S422, the current node is taken as the fault cause node.

[0236] S5. Based on the fault risk level calculation results in step S3 and the fault cause nodes in step S4, determine the cause of equipment failure.

[0237] Furthermore, step S5 determines the cause of equipment assembly misoperation based on the fault risk level calculation result in step S3 and the fault cause node in step S4, specifically including the following steps:

[0238] S51. Obtain the fault risk level calculation result in step S3, and sort the fault risk level results in descending order to obtain the component names and video clips corresponding to the first three fault risk level results in the sorting results.

[0239] S52. Compare the component names corresponding to the first three fault risk level fault results obtained in step S51 with the component names corresponding to the fault cause nodes obtained in step S4. If the component name is among the component names corresponding to the first three fault risk level fault results obtained in step S51 and the component names corresponding to the fault cause nodes obtained in step S4, then provide feedback to the user as the cause of the equipment assembly misoperation.

[0240] Optionally, based on the above implementation method, a standard assembly operation video database can be used as a benchmark to compare with non-standard assembly misoperation data. If abnormal data is found, an early warning can be issued to avoid abnormalities caused by assembly misoperation and ensure assembly quality.

[0241] The beneficial effects of this invention are as follows:

[0242] 1. In this invention, the user's operation is described based on a single operation action and operation path, and the fault risk level is calculated based on the two factors of operation action and operation path, which improves the accuracy of the user's diagnosis of equipment faults caused by equipment operation errors.

[0243] 2. In this invention, a video operation segment is obtained when the user's hand overlaps with the component area; and the video operation segment is segmented according to the operation on different components in the video operation segment to form multiple segmented video segments, which provides accurate data support for the comparative analysis of subsequent operation actions and further improves the accuracy of identifying user operation errors.

[0244] 3. In this invention, when the user's hand overlaps with the component area, the user's hand area and the component area are adjusted to the middle area of ​​the video image, and the angle and focal length of the camera are adjusted according to the size of the component and the preset area size, so that the video image can clearly and completely record the user's hand operation, thereby improving the accuracy of identifying user operation errors.

[0245] 4. This invention obtains a first-class operation risk value by comparing and analyzing a single operation action with the guided operation action, obtains a second-class operation risk value by comparing and analyzing the operation path with the guided operation path, and obtains the fault risk level based on the first-class and second-class operation risk values. The above algorithm comprehensively considers many factors such as the standardization of operation actions, the duration, and the interval between operation actions, thereby improving the accuracy of identifying user operation errors.

[0246] 5. This invention constructs an operation action fault tree based on the user's historical operation data, and locates the fault cause node based on the operation action fault tree, thereby improving the accuracy of fault cause diagnosis and assembly misoperation diagnosis;

[0247] 6. This invention combines the fault risk level with the fault tree method to obtain the final cause of equipment failure, thereby improving the accuracy of equipment failure cause diagnosis and assembly misoperation diagnosis.

[0248] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting assembly misoperation based on video data, characterized in that, Includes the following steps: S1. Acquire user operation video data of the device, wherein the operation video data is video data of the current user operating the device according to a predetermined operation procedure; S2. Obtain multiple single operation actions and operation paths of the user on the device from the operation video data; The operation path is a sequence of multiple single operation actions performed by the user on the device; The single operation action refers to the process by which a user performs a single operation on the device; S3. Calculate the fault risk level of the user's operation based on the user's multiple single operation actions and operation paths on the device; S4. When equipment malfunctions, locate the fault cause node in the operation action fault tree according to the equipment fault type; S5. Based on the fault risk level calculation results in step S3 and the fault cause nodes in step S4, determine the cause of equipment assembly misoperation. Step S1, acquiring user operation video data on the device, specifically includes: A 3D camera is used to acquire video data of user interactions with the device. Step S2, which obtains multiple single-action actions and operation paths of the user on the device from the operation video data, specifically includes the following steps: S21. Based on the three-dimensional coordinate information of each point of the device in the video image, perform three-dimensional modeling of the device to form a three-dimensional model of the device; The three-dimensional model of the device includes the three-dimensional coordinate information of each point in the device in the video image; The three-dimensional model of the device includes the three-dimensional position information of each component in the video image; The three-dimensional position information of each component includes the component name and component marker box; The component marking box is the area in the video image where the component is marked with a rectangular frame; S22. Identify the three-dimensional region of the user's hand in the video image, and dynamically track the three-dimensional region of the hand; S23. When the three-dimensional region of the hand overlaps with the three-dimensional region of the three-dimensional model, record the start time of a single operation and the video frame corresponding to the start time of the single operation. The hand's movements are continuously tracked until the three-dimensional region of the hand does not overlap with the three-dimensional region of the three-dimensional model. The end time of a single operation and the corresponding video frame are then recorded. The video segment between the video frame corresponding to the start time of the single operation and the video frame corresponding to the end time of the single operation is taken as the video segment of this single operation. S24. Repeat steps S22 to S23 until the user completes all operation procedures and obtains video clips of multiple single operation actions; S25. The video segments of the multiple single operation actions are segmented to form multiple segmented video segments, which serve as the multiple single operation actions in step S2. S26. Combine the multiple video clips from step S25 in chronological order to form the operation path in step S2; Step S3 calculates the fault risk level of the user's operation based on the user's multiple single operation actions and operation paths on the device, specifically including the following steps: S31. Obtain the video segment corresponding to a single operation action and the name of the component corresponding to the single operation action, and obtain the guiding operation action corresponding to the component name; The guided operation actions are pre-stored guided operation video clips; S32. Compare the video clips corresponding to the single operation action obtained in step S31 with the video clips corresponding to the guided operation action to obtain a type of operation risk value; S33. For all single operation actions, execute steps S31 to S32 to obtain the operation risk value of one type for all single operation actions; S34. Based on the operation path and the guided operation path, obtain the Class II operation risk values ​​for all individual operation actions; The guided operation path is a pre-stored sequence of guided operation actions arranged in chronological order; Each guided operation action in the guided operation path is labeled with a numerical number in ascending order according to the chronological sequence. S35. Based on the Class I operational risk values ​​of all single operation actions obtained in step S33 and the Class II operational risk values ​​obtained in step S34, obtain the fault risk level of all single operation actions.

2. The assembly misoperation detection method based on video data according to claim 1, characterized in that, Step S32 compares the video segment corresponding to the single operation action obtained in step S31 with the guidance operation video segment corresponding to the guidance operation action to obtain a type of operation risk value, which specifically includes the following steps: S321. Obtain the sequence of user hand actions and operation durations for the user's hand on the operation points of the components in the instruction operation video clip in chronological order; S322. Based on the operation point location obtained in step S321, query the user's hand operation sequence and operation duration corresponding to the operation point location in the video segment corresponding to the single operation action obtained in step S31; S323. Calculate the similarity of the user's hand operation sequence in step S321 according to the pose similarity comparison algorithm to obtain the operation similarity D; S324. Calculate the similarity between the operation duration in step S321 and the operation duration in step S322 to obtain the operation duration similarity I; S325. Based on the similarity value D of the operation action and the similarity value I of the operation duration, calculate the value of a class of operation risk.

3. The assembly misoperation detection method based on video data according to claim 2, characterized in that, Step S35 obtains the fault risk level of all single operation actions based on the Class I operation risk values ​​obtained in step S33 and the Class II operation risk values ​​obtained in step S34, specifically including: S351. Multiply the value of a single operation risk by the operation adjustment coefficient corresponding to the single operation, and use the resulting multiplication value as the fault risk level corresponding to the single operation. S352. For all single operation actions, repeat step S351 to obtain the fault risk level of each single operation action.

4. The assembly misoperation detection method based on video data according to claim 3, characterized in that, When a device malfunctions, step S4 involves locating the cause node in the operation action fault tree based on the device malfunction type. Specifically, this includes: S41. Construct a fault tree for operation actions based on the user's historical operation data of the device; The fault tree includes multiple subtrees; Each subtree includes a root node, intermediate nodes, and leaf nodes; The root node of the subtree stores the device fault type, while the intermediate nodes and leaf nodes are fault cause nodes; the fault cause nodes store the erroneous operation type for the component. The root node is directed to the intermediate node by a directed path, and the intermediate node is directed to the leaf node by a directed path. The directed path is a path with a correlation value; the correlation value is the correlation between two nodes. S42. When equipment malfunctions, locate the fault cause node in the operation action fault tree according to the equipment fault type.

5. The assembly misoperation detection method based on video data according to claim 4, characterized in that, Step S42, which locates the fault cause node, specifically includes the following steps: S421. Based on the equipment fault type, locate the root node of one of the multiple subtrees in the fault tree, and take the root node as the current node; S422. When the current node points to multiple lower-level nodes, obtain these multiple lower-level nodes as the fault cause nodes; When the current node points to a single subordinate node, the subordinate node is taken as the current node, and step S422 is repeated until the current node points to multiple subordinate nodes, and these multiple subordinate nodes are taken as the fault cause nodes. When step S422 is executed repeatedly, if the current node has no subordinate nodes, the current node is taken as the fault cause node.

6. The assembly misoperation detection method based on video data according to claim 5, characterized in that, Step S5 determines the cause of equipment assembly misoperation based on the fault risk level calculation result in step S3 and the fault cause node in step S4, specifically including the following steps: S51. Obtain the fault risk level calculation result in step S3, and sort the fault risk level results in descending order to obtain the component names and video clips corresponding to the first three fault risk level results in the sorting results. S52. Compare the component names corresponding to the first three fault risk level fault results obtained in step S51 with the component names corresponding to the fault cause nodes obtained in step S4. If the component name is among the component names corresponding to the first three fault risk level fault results obtained in step S51 and the component names corresponding to the fault cause nodes obtained in step S4, then provide feedback to the user as the cause of the equipment malfunction, along with the video clip.

Images