Material testing and assembly methods, devices, equipment, media, and program products

Abstract

The present application discloses a material detection and assembly method, device, equipment, medium, and program product, which relate to the field of material production technology. The material detection method includes target tracking of an assembly target used to assemble a material to be assembled in a video to be detected, so that a single assembly target can be used to adapt to multiple types of materials to be assembled. In addition, the start assembly moment when the assembly target in the video to be detected carries the material to be assembled to the assembly starting position can be identified, and the moment when the material assembly is completed, that is, the first moment and the second moment, can be identified, and the assembly displacement of the assembly target along the assembly direction during this period can be obtained to verify whether the material to be assembled is in place through the assembly displacement of the assembly target. The technical problem of high product defective rate is solved, and the automation of material assembly detection is achieved to improve the product pass rate, while taking into account the technical effect of the generalizability of material assembly detection.

Description

Technical Field

[0001] The present application relates to the field of material production technology, and in particular to a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In precision manufacturing, products are typically assembled from various components. For example, in the case of a server, the CPU (Central Processing Unit), memory, accelerator card, fan, and power cables are installed on the motherboard. However, high product defect rates are still common, and even those that pass factory inspection can still experience short circuits or structural damage after powering on. Summary of the Invention

[0003] The present application provides a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of high product defect rate in related technologies.

[0004] The present application provides a material detection method, which includes: obtaining a video to be detected shot by a camera component; tracking an assembly target in the video to be detected, and obtaining assembly process information formed by changes in position information of the assembly target when the assembly target assembles the material to be assembled; parsing the assembly process information to obtain a first moment and a second moment; wherein the first moment is the moment when the assembly target and the material to be assembled arrive at the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled; obtaining position data along the assembly direction detected by the detection component at the first moment and the second moment to obtain the assembly displacement of the assembly target along the assembly direction; judging whether the assembly displacement matches the preset displacement; and determining that the material to be assembled is in place in response to the assembly displacement matching the preset displacement.

[0005] The present application also provides a material assembly method, which includes: controlling an assembly target to carry a material to be assembled to an assembly station and performing material assembly on the material to be assembled; using the material detection method as described above to determine whether the material to be assembled is assembled in place; in response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0006] The present application also provides a material assembly device, which includes: a camera component, a detection component and a control component; the camera component is used to be set at the assembly station to shoot the video to be detected; the detection component is used to detect the position data of the assembly target along the assembly direction; the control component is connected to the camera component to implement the material detection method as described above; or, to implement the material assembly method as described above.

[0007] The present application also provides an electronic device, which includes: a memory and a processor; the memory is used to store computer programs; the processor is used to implement the steps of the above-mentioned material detection method when executing the computer program; or, implement the steps of the above-mentioned material assembly method.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the material detection method described above are implemented; or, the steps of the material assembly method described above are implemented.

[0009] The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned material detection method; or, it implements the steps of the above-mentioned material assembly method.

[0010] Through this application, since the assembly target used to assemble the materials to be assembled in the video to be inspected is followed, multiple types of materials to be assembled can be adapted through a single assembly target. In addition, the start assembly moment when the assembly target in the video to be inspected arrives at the assembly starting position with the materials to be assembled can be identified, and the moment when the material assembly is completed can also be identified, and the assembly displacement of the assembly target along the assembly direction during this period can be obtained to verify whether the materials to be assembled are assembled in place through the assembly displacement of the assembly target. Therefore, the technical problem of high product defective rate can be solved, and the automation of material assembly detection can be achieved to improve the product pass rate, while taking into account the technical effect of the generalizability of material assembly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 This is a schematic structural diagram of an embodiment of the material assembly device of the present application;

[0013] Figure 2 This is a schematic structural diagram of another embodiment of the material assembly device of the present application;

[0014] Figure 3 This is a flow chart of an embodiment of the material detection method of the present application;

[0015] Figure 4 This is a flow chart of another embodiment of the material detection method of the present application;

[0016] Figure 5 This is a flow chart of an embodiment of target key point detection in this application;

[0017] Figure 6 This is a structural diagram of an embodiment of the key point of this application;

[0018] Figure 7 This is a flow chart of an embodiment of feature extraction performed by the first convolution unit of the present application;

[0019] Figure 8 This is a flow chart of an embodiment of feature extraction performed by the second convolution unit of the present application;

[0020] Figure 9 A schematic diagram of a flow chart of an embodiment of feature extraction performed by a preset feature extraction unit of this application;

[0021] Figure 10 A schematic diagram of a flow chart of an embodiment of determination by a determination unit of the present application;

[0022] Figure 11 This is a flow chart of an embodiment of the material assembly method of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the material detection method and / or the material assembly method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0027] The embodiment of the present application provides a material assembly device, which is described in detail below in conjunction with its specific structure.

[0028] See also Figure 1 , Figure 1 This is a structural diagram of an embodiment of the material assembly device of the present application.

[0029] In one embodiment, the material assembly device may include a camera component 10 , a detection component 20 , and a control component.

[0030] The camera assembly 10 is used to be arranged at an assembly station to shoot a video to be inspected.

[0031] The detection component 20 is used to detect position data of the assembly target along the assembly direction.

[0032] The control component is connected to the camera component 10 to implement a material detection method; or, to implement a material assembly method.

[0033] The material detection method may at least include obtaining a video to be detected shot by the camera component 10; tracking the assembly target in the video to be detected, and obtaining assembly process information formed by changes in position information of the assembly target when the assembly target assembles the material to be assembled; parsing the assembly process information to obtain a first moment and a second moment; wherein the first moment is the moment when the assembly target and the material to be assembled arrive at the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled; obtaining position data along the assembly direction detected by the detection component 20 at the first moment and the second moment to obtain the assembly displacement of the assembly target along the assembly direction; judging whether the assembly displacement matches the preset displacement; and determining that the material to be assembled is in place in response to the assembly displacement matching the preset displacement.

[0034] The material assembly method may at least include controlling an assembly target to carry the material to be assembled to an assembly station and assembling the material to be assembled; using the material detection method as described above to determine whether the material to be assembled is assembled in place; and in response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0035] Please refer to Figure 1 as well as Figure 2 , Figure 2 This is a structural diagram of another embodiment of the material assembly device of the present application.

[0036] In one embodiment, the detection component 20 includes a signal sending component and a signal receiving component. The signal sending component periodically sends a linear signal to the assembly target at preset time intervals, and the signal receiving component receives the linear signal reflected by the assembly target.

[0037] Furthermore, the signal sending element may include a first sending element 211 and a second sending element 221 .

[0038] The signal receiving element may include a first receiving element 212 , a second receiving element 213 , a third receiving element 222 and a fourth receiving element 223 .

[0039] The first receiving element 212 and the second receiving element 213 are used to obtain the linear signal sent by the first sending element 211 .

[0040] The third receiving element 222 and the fourth receiving element 223 are used to obtain the linear signal sent by the second sending element 221 .

[0041] Thus, in this embodiment, two transmitters and four receivers can be used to calculate distance and speed respectively, effectively avoiding the error of a single sensor, thereby improving the accuracy of assembly target position sensing and the accuracy of material placement detection.

[0042] For the description of the features in the embodiments corresponding to the material assembly device, please refer to the relevant description of the embodiments corresponding to the material detection method and the material assembly method below, which will not be repeated here.

[0043] The embodiments of the present application provide a material detection method, which is described in detail below in conjunction with the execution process of the material detection method.

[0044] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of the material detection method of the present application.

[0045] S101: Obtain the video to be detected shot by the camera component.

[0046] In this embodiment, the video to be inspected is a video of an assembly target to be inspected to see whether the assembly material is in place. The camera assembly is a device for capturing the video to be inspected.

[0047] S102: Following the assembly target in the video to be inspected, and obtaining assembly process information formed by position information changes when the assembly target assembles the material to be assembled.

[0048] In this embodiment, the assembly target present in the video to be inspected can be monitored and followed, so that the motion trajectory of the assembly target in assembling the material to be assembled can be obtained, that is, the assembly process information formed by the change of its position information can be obtained, so as to use the assembly process information to verify whether the assembly target is assembled with the material to be assembled in place.

[0049] S103: Analyze the assembly process information to obtain a first moment and a second moment; wherein the first moment is the moment when the assembly target and the material to be assembled arrive at the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled.

[0050] In this embodiment, in response to obtaining assembly process information, key moments in the video to be inspected can be parsed and extracted: the first moment and the second moment. The first moment is when the assembly target and the material to be assembled reach the assembly starting position, and the second moment is when the assembly target completes the assembly of the material to be assembled. This reduces the computational complexity of evaluating whether the assembly target has fully assembled the material to be assembled.

[0051] S104: Acquire position data along the assembly direction detected by the detection component at the first moment and the second moment to obtain assembly displacement of the assembly target along the assembly direction.

[0052] In this embodiment, the position data of the assembly target along the assembly direction detected by the detection component at the first moment can be obtained, and the position data of the assembly target along the assembly direction detected by the detection component at the second moment can also be obtained, so as to obtain the assembly displacement of the assembly target along the assembly direction at the second moment compared with the first moment, thereby characterizing the displacement of the material to be assembled along the assembly direction.

[0053] S105: Determine whether the assembly displacement matches the preset displacement.

[0054] In this embodiment, it is possible to judge whether the assembly displacement matches the preset displacement associated with the material to be assembled, so as to verify whether the assembly displacement of the assembly target reaches the preset displacement, thereby realizing the travel movement of the material to be assembled to achieve the preset displacement, thereby indirectly verifying whether the material to be assembled is installed in place.

[0055] S106: In response to the assembly displacement matching the preset displacement, it is determined that the material to be assembled is assembled in place.

[0056] In this embodiment, in response to the assembly displacement of the assembly target matching the preset displacement pre-associated with the material to be assembled, it can be considered that the assembly target moves the preset displacement carrying the material to be assembled, and the material to be assembled can be assembled relatively reliably, so it can be determined that the material to be assembled is assembled in place.

[0057] It can be seen that in this embodiment, the assembly target used to assemble the materials to be assembled in the video to be inspected can be followed. This means that a single assembly target can be used to adapt to multiple types of materials to be assembled, and there is no need to train the recognition capabilities of the multiple types of materials to be assembled separately. The image recognition capabilities of the assembly target can be trained to take into account the detection of multiple types of materials to be assembled. In addition, the start assembly moment when the assembly target in the video to be inspected arrives at the assembly starting position with the materials to be assembled can be identified, and the moment when the material assembly is completed can also be identified, that is, the first moment and the second moment. By obtaining the position data of the first moment and the second moment, the assembly displacement of the assembly target along the assembly direction during this period is obtained, so as to verify whether the materials to be assembled are in place through the assembly displacement of the assembly target. The technical problem of high product defective rate is solved, and the automation of material assembly detection is achieved to improve the product pass rate, while taking into account the technical effect of the generalizability of material assembly detection.

[0058] In other words, the reason why high product defect rates are common in this embodiment is that assembly errors occur during the material assembly process. These errors are often hidden, often involving tiny offsets or angular deviations that are difficult to detect using traditional vision or mechanical touch. For example, if workpiece A is not fully embedded in the slot of workpiece B, directly applying power may cause a short circuit or structural damage, resulting in high repair costs and limitations in detection methods. Relying on static measurement makes it difficult to meet the real-time requirements of high-speed production lines. In addition, complex workpiece internal structures obstruct detection, resulting in blind spots. Environmental interference, shape differences between different parts, and manufacturing tolerances of the same part also increase the difficulty of assembly detection.

[0059] See also Figure 4 , Figure 4 This is a flow chart of another embodiment of the material detection method of the present application.

[0060] S201: Obtain the video to be detected.

[0061] In this embodiment, the video to be detected may include one or more video frames to be detected, and the video frames to be detected in the video to be detected may be extracted sequentially, and assembly target detection may be performed sequentially.

[0062] Furthermore, the video frame to be detected may be pre-processed, for example, the video frame to be detected may be scaled to a preset image size, super-resolution reconstructed, and so on.

[0063] For example, the pre-set image size can be 640*640. During the scaling process of the video frame to be detected, the image can be scaled without changing its aspect ratio. The largest side can be scaled to 640 and the smaller sides can be filled with grayscale to retain the original image features.

[0064] S202: Input the video frame to be detected into the key point detection model to perform target key point detection.

[0065] In this embodiment, the keypoint detection model is used to detect the presence of target keypoints in the video frames to be inspected. If the keypoint detection model determines the presence of a target keypoint, the model performs target tracking on the target keypoint and obtains its position information in each video frame to be inspected.

[0066] That is to say, the key point detection model can be used to detect whether the video frame to be detected contains specific key point information, that is, the target key point of the assembly target.

[0067] For example, the assembly target can be a robotic arm or a human hand, without strict limitations. For example, for a robotic arm, the target key points can be the parts near the material being assembled or the robotic arm itself. For a human hand, the target key points can be the fingers, wrist, elbow, etc.

[0068] In this way, compared with identifying the overall motion trajectory and assembly displacement of the assembly target, identifying the motion trajectory and assembly displacement of the target's key points can more reliably confirm the depth of the assembly target's assembly of the material to be assembled, and can improve the recognition granularity, which is conducive to reducing the risk of misidentification and unreliable detection results, and further improving the reliability of material assembly detection.

[0069] S203: Determine whether there is a target key point.

[0070] In this embodiment, when it is determined that the target key point exists, step S204 may be executed. When it is determined that the target key point does not exist, step S202 may be executed.

[0071] The judgment result output by the key point detection model can be identified to confirm whether the assembly target and the target key points exist in the video frame to be detected for the current assembly target detection.

[0072] S204: Obtain the target coordinates of the target key point in the target coordinate system as its position information.

[0073] In this embodiment, the assembly target is located at the pixel coordinates of the video frame to be inspected, the pixel coordinates are mapped to a three-dimensional target coordinate system to form target coordinates, and the target coordinates are used as the position information of the assembly target in the video frame to be inspected.

[0074] Specifically, a matrix expression for the transformation matrix can be obtained. This matrix expression is derived by fusing the joint matrix, rotation matrix, and translation matrix of the camera assembly. The joint matrix is ​​formed based on the camera intrinsic parameters and the scale factor, while the translation matrix is ​​formed based on the camera focal length. The coordinate information of the inspection component and the assembly station are written into the matrix expression to solve for the transformation matrix. The transformation matrix is ​​then fitted to the pixel coordinates to obtain the target coordinates.

[0075] For example, the pixel coordinates can be expressed as (x, y) and the target coordinates can be expressed as [X, Y]. The following example uses the second sending element as the origin of the target coordinate system.

[0076] According to the installation position of the camera component, the pixel coordinates can be converted to the target coordinates using the coordinates P1 (x1, y1) of the second sending component, the coordinates P2 (x2, y2) of the third receiving component, the coordinates P3 (x3, y3) of the fourth receiving component, and the coordinates P4 (x4, y4), P5 (x5, y5), and P6 (x6, y6) of any three non-collinear points on the assembly station as reference points.

[0077] In other words, a real-world coordinate system can be established as the target coordinate system, using the assembly station as the reference plane and the second sending component as the origin. This design can convert the two-dimensional position information in the video frame to be detected into three-dimensional position information in the real world. At the same time, it can also amplify the motion trajectory of small changes in pixels, thereby improving the recognition reliability of small displacements and, in turn, the reliability of material in-place detection.

[0078] The relationship between pixel coordinates and target coordinates can be established. The target coordinates can be as follows:

[0079] Formula 1-1

[0080] Formula 1-2

[0081] Formula 1-3

[0082] Formula 1-4

[0083] Among them, K represents the joint matrix; R represents the rotation matrix; t represents the camera translation matrix; f x and f y Indicates the focal length of the camera; c x and c y represents the camera intrinsic parameter; r represents the rotation vector.

[0084] In this way, the conversion relationship of formula 1-1 can be simplified to form a conversion expression combining the pixel coordinates of the conversion matrix and the target coordinates, as shown in the following example:

[0085] Formula 1-5

[0086] Formula 1-6

[0087] Where P represents the transformation matrix; Represents a matrix expression; p represents the specific value to be obtained.

[0088] Solve the pixel transformation matrix P, 、 、 、 、 and Substituting into formula 1-5, we can find the specific expression of the transformation matrix P.

[0089] If so, the target coordinate expression of the target key point is specifically expressed as:

[0090] Formula 1-7

[0091] Of course, in an alternative embodiment, the ratio between the target coordinate system and the pixel coordinates can also be other ratios, which does not need to match the real-world coordinate system, or other coordinate origins and reference planes can be selected, which will not be repeated here.

[0092] S205: The signal sending component follows the assembly target.

[0093] In this embodiment, the assembly target can move with the material being assembled. To ensure reliable position detection, the signal transmitter can be controlled to change the linear signal transmission angle according to the assembly target. Optionally, this can be manually adjusted by external personnel, or automatically tracked through software and / or hardware control.

[0094] Specifically, in this embodiment, the following of an assembly target is achieved by using a first transmitting component, a first receiving component, and a second receiving component. The signal transmitting end of the first transmitting component and the signal receiving ends of the first and second receiving components can be collinear, and the relative directions of the three can be perpendicular to the surface of the assembly station.

[0095] In this way, the first angle between the first virtual line and the second virtual line can be calculated; the first virtual line represents the virtual connection line between the first receiving part and the assembly target and the material to be assembled, and the second virtual line represents the virtual connection line between the first receiving part and the second receiving part. In this way, the angle of the second angle complementary to the first angle can be calculated, and the second angle is the angle between the first virtual line and the third virtual line, and the third virtual line represents the virtual connection line between the first receiving part and the first sending part. In this way, the third angle associated with the second angle can be solved using the law of cosines. The third angle is the angle between the ideal virtual connection line between the first sending part and the assembly target and the third virtual line, so that the angle of the first sending part can be adjusted according to the calculated third angle.

[0096] Two signal receiving components can be used to associate with one signal sending component to calculate the angle between the transmitter and the point to be detected, facilitate the adjustment of the angle of the signal generator, and deduce the angle of the material relative to the signal generator.

[0097] Similarly, the assembly target following principles of the second sending component, the third receiving component, and the fourth receiving component are similar and will not be described in detail here.

[0098] S206: Calculate the moving speed of the assembly target.

[0099] In this embodiment, the assembly process information includes the instantaneous position of the assembly target in each video frame to be detected. In this way, the moving speed of the assembly target can be calculated based on the instantaneous position between the video frames to be detected.

[0100] Specifically, the distance between the current position of the signal receiver and the instantaneous position of the assembly target when the linear signal is reflected is obtained as the signal transmission distance. The signal transmission distance, the preset time, and the pulse wavelength of the linear signal are used to estimate the signal phase difference between two consecutive linear signals received by the signal receiver. The pulse wavelength and the signal phase difference are fitted to obtain a fitting intermediate value. The ratio of the fitting intermediate value to the duration factor is used as the movement speed of the assembly target between the two consecutive linear signals. The duration factor is obtained by amplifying the preset time.

[0101] The specific calculation method will be explained in detail later.

[0102] S207: Determine whether the assembly start position appears.

[0103] In this embodiment, when it is determined that the assembly start position appears, step S208 may be executed. When it is determined that the assembly start position does not appear, step S202 may be executed.

[0104] The assembly process information is used to calculate the moving speed of the assembly target and the rate of change of the moving speed at each preset time interval. The moment when the moving speed change rate falls below a rate threshold is selected as the first moment to determine that the assembly target has arrived at the assembly starting position.

[0105] Furthermore, in response to the existence of multiple occurrence moments with change rates lower than a rate threshold, the multiple occurrence moments are compared and the occurrence moment that is later than the other occurrence moments among the multiple occurrence moments can be taken as the first moment.

[0106] Thus, using the moment of speed change as the key point in the inspection process simplifies the calculation of the distance traveled by the material to be assembled and the assembly target. Only the position difference between the last speed change and the moment of speed cessation needs to be calculated. This effectively reduces the amount of material in-place detection required, thereby maintaining the reliability of the material inspection method. Furthermore, the prior knowledge that the material to be assembled will be obstructed and its movement speed reduced at the beginning of assembly can be combined to simplify the inspection process decomposition.

[0107] S208: Evaluate the target position of the assembly target at the assembly start position.

[0108] In this embodiment, in response to determining that the assembly target is located at the assembly starting position, the position information of the current position can be obtained as the target position at the assembly starting position. Alternatively, it can be the target coordinates of the assembly target, or the height position along the assembly direction relative to the assembly station, that is, the target position d dis1 (This will be used as an example in this embodiment) etc.

[0109] S209: Determine whether the assembly of the materials to be assembled is completed.

[0110] In this embodiment, when it is determined that the assembly of the materials to be assembled is completed, step S210 may be executed. When it is determined that the assembly of the materials to be assembled is not completed, step S202 may be executed.

[0111] The moment when the moving speed reaches zero is selected as the second moment to determine whether the assembly target has completed the assembly of the materials to be assembled.

[0112] S210: Evaluate the target position of the assembly target when the assembly of the materials to be assembled is completed.

[0113] In this embodiment, the target position d when the assembly target completes the assembly of the material to be assembled is obtained. dis2 .

[0114] S211: Acquire the assembly displacement of the assembly target along the assembly direction.

[0115] In this embodiment, the target position d can be obtained dis2 and the target position d dis1The difference between them is taken as the assembly displacement, and the specific calculation formula can be shown as follows:

[0116] Formula 2-1

[0117] Where ∆d represents the assembly displacement.

[0118] S212: Use assembly displacement to evaluate whether the materials to be assembled are assembled in place.

[0119] In this embodiment, when the assembly displacement is used to evaluate that the material to be assembled is in place, step S213 may be executed. When the assembly displacement is used to evaluate that the material to be assembled is not in place, step S214 may be executed.

[0120] S213: Determine whether the materials to be assembled are in place.

[0121] In this embodiment, in response to determining that the material to be assembled is in place, it can be considered that the material to be assembled has been reliably assembled, and the assembly of the next material to be assembled can be carried out.

[0122] S214: Determine whether the material to be assembled is not assembled in place.

[0123] In this embodiment, in response to a failure to assemble a material, it can be determined that the current state of the material is likely to result in a defective product. Therefore, a prompt indicating the failure can be sent to promote timely and reliable assembly of the material. Furthermore, when assembling materials using a controllable and schedulable robotic arm, adjustment instructions can be adaptively generated and sent to the robotic arm to achieve automated material assembly adjustment.

[0124] For example, when adjusting the signal sending part to follow the assembly target, the assembly target movement tracking and adjustment can be performed in real time. Whether the assembly target has assembled the material to be assembled can be checked after the assembly target completes the current assembly of the material to be assembled, so as to globally and reliably identify the first moment and the second moment in combination with the video to be detected.

[0125] The following describes specific methods for obtaining the assembly targets at the first moment and the second moment with examples.

[0126] Obtain the distance factor, angle factor, delay factor, and transmission factor. The distance factor is the distance between the signal transmitter and the signal receiver, the angle factor is the angle between the signal transmitter and the assembly direction, the delay factor is the time delay for the linear signal to be transmitted from the signal transmitter to the signal receiver, and the transmission factor is the transmission speed of the linear signal.

[0127] A trigonometric expression of the signal transmitter, the signal receiver, and the assembly target is obtained as a first expression. The first expression combines a first position factor, a second position factor, a distance factor, and a trigonometric function. The first position factor is the distance between the signal transmitter and the assembly target at the target time. The second position factor is the distance between the signal receiver and the assembly target at the target time. The target time is either the first time or the second time.

[0128] A second expression between the first position factor, the second position factor, the delay factor, and the transmission factor is obtained.

[0129] The first position factor at the target time is solved by combining the first expression and the second expression.

[0130] The trigonometric function is fitted with the solved first position factor, and the target position of the assembly target along the assembly direction at the target time is taken as the target position.

[0131] For example, the distance between the first receiving element and the first transmitting element is the distance factor h1, the angle between the first transmitting element and the assembly direction such as the vertical direction is the angle factor θ1, the time delay between the received signal and the transmitted signal is the delay factor ∆t1, and the transmission speed of the pulse signal, i.e., the linear signal, is c.

[0132] According to the triangle formed by the first sending component, the first receiving component and the target key point, the trigonometric function expression of the signal sending component, the signal receiving component and the assembly target can be obtained, which can be specifically expressed as the first expression:

[0133] Formula 3-1

[0134] Among them, l 11 Represents the distance between the first sending component and the target key point, which is equivalent to the first position factor; l 12 It represents the distance between the first receiving part and the target key point, which is equivalent to the second position factor of the first receiving part; θ1 represents the angle at the assembly starting position.

[0135] The second expression for obtaining the first position factor, the second position factor, the delay factor, and the transmission factor can be specifically expressed as follows:

[0136] Formula 3-2

[0137] If so, combining formula 3-1 and formula 3-2, we can solve the first position factor l 11 value.

[0138] The specific calculation method of fitting the trigonometric function and the solved first position factor as the target position of the assembly target along the assembly direction at the target time can be shown as follows:

[0139] Formula 3-3

[0140] Formula 3-4

[0141] Where d1 represents the target position solved at the assembly start position; d2 represents the target position solved when the assembly is completed; θ2 represents the angle when the assembly is completed; l 21 Indicates the first-value factor when the assembly is complete.

[0142] Furthermore, if Figure 2 As shown in the example, the signal transmitter includes a first transmitter and a second transmitter; the signal receiver includes a first receiver, a second receiver, a third receiver, and a fourth receiver. Thus, the target positions corresponding to the first receiver, the second receiver, the third receiver, and the fourth receiver can be evaluated separately and weighted fusion performed to obtain position data. The specific calculation formula can be shown as follows:

[0143] Formula 4-1

[0144] Formula 4-2

[0145] in, , α, β, γ, The four are weight factors, which are hyperparameters and can be set according to actual conditions; d 11 Indicates the target position of the first receiving part at the assembly starting position; d 12 Indicates the target position of the second receiving part at the assembly starting position; d 13 Indicates the target position of the third receiving part at the assembly starting position; d 14 Indicates the target position of the fourth receiving component at the assembly starting position; d 21 Indicates the target position of the first receiving part when the assembly is completed; d 22 Indicates the target position of the second receiving part when the assembly is completed; d 23 Indicates the target position of the third receiving part when assembly is completed; d 24 Indicates the target position of the fourth receiving component when assembly is completed.

[0146] Please refer to Figure 5 as well as Figure 6 , Figure 5 This is a flow chart of an embodiment of target key point detection in this application. Figure 6 This is a structural diagram of an embodiment of the key point of this application.

[0147] In one embodiment, a keypoint detection model includes: a first feature extraction module, a second feature extraction module, a third feature extraction module, a first upsampling module, a first linking unit, a fourth feature extraction module, a second upsampling module, a second linking unit, a fifth feature extraction module, a third linking unit, a sixth feature extraction module, and a determination unit, which are sequentially connected. The determination unit is configured to determine whether an input image contains a target keypoint and output a determination result.

[0148] The output end of the first feature extraction module is also connected to the input end of the second connection unit. The output end of the second feature extraction module is also connected to the input end of the first connection unit. The output end of the fourth feature extraction module is also connected to the input end of the third connection unit. At the same time, Figure 5 The dimensions of the video frames to be detected after they have passed through the modules and units of each keypoint detection model are identified, such as [1,3,640,640], where 1 represents the number of video frames to be detected (i.e., batch size) input to the keypoint detection model, 3 represents the number of channels, and 640 and 640 represent the image size. Dimensions such as 0-P1 / 2 are used to identify dimensionality reduction, where 0-P1 / 2 indicates the first dimensionality reduction.

[0149] As shown in the example of Figure 6, when the assembly target is a human hand, it may include target key point K1, target key point K2, and target key point K3.

[0150] When the key point detection model obtains a video frame to be detected, the first feature extraction module performs a first feature extraction process on it to obtain a first image feature, and transmits the first image feature to the second feature extraction module and the second connection unit.

[0151] The second feature extraction module performs second feature extraction processing on the first image feature to obtain a second image feature, and transmits the second image feature to the third feature extraction module and the first connection unit.

[0152] The third feature extraction module performs a third feature extraction process on the second image feature to obtain a third image feature, and transmits the third image feature to the first upsampling module.

[0153] The first upsampling module performs a first upsampling process on the third image feature to obtain a first intermediate feature, and transmits the first intermediate feature to the first connection unit.

[0154] The first connection unit connects the first intermediate feature and the second image feature to form a first connection feature, and transmits the first connection feature to the fourth feature extraction module.

[0155] The fourth feature extraction module performs a fourth feature extraction process on the first connection feature to obtain a fourth image feature, and transmits the fourth image feature to the second upsampling module and the third connection unit.

[0156] The second upsampling module performs a second upsampling process on the fourth image feature to obtain a second intermediate feature, and transmits the second intermediate feature to the second connection unit.

[0157] The second connection unit connects the first image feature and the second intermediate feature to form a second connection feature, and transmits the second connection feature to the fifth feature extraction module.

[0158] The fifth feature extraction module performs a fifth feature extraction process on the second connection feature to form a fifth image feature, and transmits the fifth image feature to the third connection unit.

[0159] The third connection unit connects the fifth image feature and the fourth image feature to form a third connection feature, and transmits the third connection feature to the sixth feature extraction module.

[0160] The sixth feature extraction module performs a sixth feature extraction process on the third connection feature to form a sixth image feature, and transmits the sixth image feature to the determination unit.

[0161] The determination unit analyzes the sixth image feature to determine whether the input video frame to be detected contains the target key point.

[0162] Furthermore, in this embodiment, two convolution units can be pre-constructed, namely, a first convolution unit Conv and a second convolution unit C3f.

[0163] like Figure 7 As shown in the example, Figure 7 This is a flow chart of an embodiment of feature extraction performed by the first convolution unit of the present application.

[0164] The first convolution unit Conv may include a two-dimensional convolution layer Conv2d, a normalization layer BN, and an activation function SiLU.

[0165] like Figure 8 As shown in the example, Figure 8 This is a flow chart of an embodiment of feature extraction performed by the second convolution unit of the present application.

[0166] The second convolution unit C3f may include a convolution layer Conv1, a data segmentation layer split, a preset feature extraction unit B1, a contact function contact and a convolution layer Conv1.

[0167] Split can perform binary channel splitting and then merging, preserving both the original features and the features after B1. C3f can maintain the input and output feature shapes, facilitating model building.

[0168] In addition, at least one of the first contact unit, the second contact unit, and the third contact unit may also be a contact function contact, etc.

[0169] Further, Figure 5 The parameters of the first convolution unit Conv and the second convolution unit C3f are also identified. Such as k, s, p, c, n=6xd, etc. Among them, k represents the convolution kernel size, s represents compensation, p represents padding, and c represents the number of output channels. In m=6xd, d represents the hyperparameter and can be defaulted to 1. "x" is the multiplication sign, and m is applied to Figure 8 In the "(m-1)xB1" upstream of the contact function contact.

[0170] like Figure 9 As shown in the example, Figure 9 This is a flow chart of an embodiment of feature extraction performed by the preset feature extraction unit of this application.

[0171] The preset feature extraction unit B1 may include a convolution layer Conv1 and a convolution layer Conv2.

[0172] See also Figure 10 , Figure 10 This is a flow chart of an embodiment of determination performed by the determination unit of the present application.

[0173] The detection unit detect can include a separable convolution layer DWConv, a two-dimensional convolution layer Conv2d, and an output layer. The output layer can include a prediction score Score, a classification feature vector Cls, and an output Point, which correspond to confidence, classification, and target key point location, respectively.

[0174] Generally speaking, the detection of target key points can include the following steps.

[0175] The input end of the key point detection model can input the image to be detected, that is, the video frame to be detected. After passing through the Conv layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map F1 is obtained; after passing through the Conv layer, a new feature map F1 is obtained; after passing through the C3f layer, a new feature map F2 is obtained; after passing through the Conv layer, a new feature map F2 is obtained; after passing through the C3f layer, a new feature map F4 is obtained; upsampling is performed to obtain a new feature map F3; the new feature map F3 is merged with the feature map F2; after passing through the C3f layer, a new feature map F4 is obtained; upsampling F4 is performed to obtain a new feature map; the new feature map is merged with the feature map F1; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; the new feature map is merged with the feature map F4; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map is obtained; after passing through the Detect layer, the detection point position, classification and confidence are obtained. Among them, the marks of feature graphs F1 to F4 are not reflected in the drawings and are only used to distinguish the feature graphs in this section.

[0176] Furthermore, the model loss function of the key point detection model can include detection point position loss, classification loss and confidence loss. The specific calculation is shown in the following example:

[0177] Formula 5-1

[0178] Formula 5-2

[0179] Formula 5-3

[0180] Formula 5-4

[0181] Among them, Loss represents the model loss function; ε, ζ, τ are weight factors, which can be hyperparameters, etc. p Indicates the detection point loss; (x i ,y i ) is the coordinate of the i-th target key point predicted by the key point detection model; (x igt ,y igt ) is the real point coordinate of the target key point; n represents the number of target key points; loss c represents the classification loss; y ij represents the true classification, c ij Indicates predicted classification; loss s represents the confidence loss; s ij Whether the category exists, which can be 0 or 1; p ijThe probability to strive for for this class prediction.

[0182] In summary, the material in-place detection of the present application can adopt non-direct contact detection, which can effectively reduce the risk of damaging the materials to be assembled during the in-place detection process. In addition, by detecting the target key points of the assembly target (such as the wrist key points, etc.), it is possible to avoid directly identifying and detecting various types of materials to be assembled, which can enhance the versatility of material in-place detection and enable different materials to be assembled to be detected using the same key point detection model, without having to train a detection model for each type of material to be assembled. It can also simplify the conversion process from the camera coordinate system to the world coordinate system, and can use the conversion matrix to pre-calibrate points for calculation, which can reduce the interference of the camera's internal parameters and help simplify the deployment and use of the material detection method. At the same time, it can also reduce the number of detection heads such as the judgment unit, and only retain one detection head, which can improve the detection efficiency of the key point detection model and is conducive to the deployment of the key point detection model in small servers such as edge micro servers.

[0183] A depthwise separable convolutional layer can be used in the detection head to further reduce the number of parameters in the key point detection model. At the same time, the material assembly device can use two transmitters and four receivers to calculate distance and speed, respectively, effectively avoiding the errors of a single sensor, thereby improving the accuracy of sensing the assembly target position and the accuracy of material in-place detection. Furthermore, using the moment of speed change as the key point in the detection process can simplify the calculation of the moving distance of the material to be assembled and the assembly target. Only the position difference between the last speed change and the speed stop need to be calculated, which can effectively reduce the amount of material in-place detection required, thereby helping to maintain the reliable operation of the material detection method.

[0184] The embodiment of the present application provides a material assembly method, which is described in detail below in conjunction with the execution flow of the material assembly method.

[0185] See also Figure 11 , Figure 11 This is a flow chart of an embodiment of the material assembly method of the present application.

[0186] S301: Control the assembly target to carry the materials to be assembled to the assembly station, and assemble the materials to be assembled.

[0187] S302: Using a material detection method to determine whether the material to be assembled is assembled in place.

[0188] S303: In response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, or a combination of software and hardware.

[0190] An embodiment of the present application also provides an electronic device.

[0191] The electronic device may include a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to perform the steps of any of the aforementioned material detection methods or material assembly method embodiments. That is, the memory is configured to store the computer program. When the processor executes the computer program, it implements the steps of the material detection method described above; or, alternatively, the steps of the material assembly method described above.

[0192] Embodiments of the present application further provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute, when executed, the steps of any of the aforementioned material detection methods or material assembly method embodiments. Specifically, when the computer program is executed by a processor, the steps of the aforementioned material detection method or the aforementioned material assembly method are implemented.

[0193] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0194] Embodiments of the present application also provide a computer program product. The computer program product may include a computer program that, when executed by a processor, implements the steps described in any of the aforementioned material detection methods or material assembly method embodiments. Specifically, when executed by a processor, the computer program implements the steps described in the aforementioned material detection method; or, alternatively, the steps described in the aforementioned material assembly method.

[0195] Embodiments of the present application also provide another computer program product. The computer program product may include a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-mentioned material detection method or material assembly method embodiments.

[0196] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] The above is a detailed introduction to a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A material detection method, characterized in that: The material detection method comprises: Obtain the video to be detected taken by the camera component; Performing target tracking on the assembly target in the video to be detected, and obtaining assembly process information formed by changes in position information of the assembly target when assembling the material to be assembled; Parsing the assembly process information to obtain a first moment and a second moment; wherein the first moment is the moment when the assembly target and the material to be assembled arrive at the assembly starting position, and the second moment is the moment when the assembly target completes assembly of the material to be assembled; Acquire position data along the assembly direction detected by the detection component at the first moment and the second moment to obtain an assembly displacement of the assembly target along the assembly direction; Determining whether the assembly displacement matches a preset displacement; In response to the assembly displacement matching the preset displacement, determining that the material to be assembled is assembled in place; The parsing of the assembly process information to obtain the first moment and the second moment includes: using the assembly process information to calculate the moving speed of the assembly target and the rate of change of the moving speed at each preset time interval, so as to select the occurrence times of the first moment and the second moment; The detection component includes a signal sending component and a signal receiving component; and the calculation of the moving speed of the assembly target using the assembly process information includes: Obtaining the distance between the current position of the signal receiving component and the instantaneous position of the assembly target when the linear signal is reflected as the signal transmission distance; Using the signal transmission distance, the preset time, and the pulse wavelength of the linear signal, the signal phase difference between two adjacent linear signals received by the signal receiving component is evaluated; The pulse wavelength and the signal phase difference are fitted to obtain a fitting intermediate value; the ratio of the fitting intermediate value to the time factor is used as the moving speed of the assembly target between the two adjacent linear signals; wherein the time factor is obtained by magnifying the preset time.

2. The material detection method according to claim 1, characterized in that: The analyzing the assembly process information to obtain the first moment and the second moment includes: Selecting the moment when the change rate is lower than the moving speed threshold as the first moment, and determining that the assembly object has arrived at the assembly starting position; The moment when the moving speed reaches zero is selected as the second moment, and it is determined that the assembly target completes the assembly of the material to be assembled.

3. The material detection method according to claim 2, characterized in that: The selecting the moment when the change rate is lower than the moving speed threshold as the first moment includes: In response to the existence of a plurality of occurrence moments at which the rate of change is lower than the rate threshold, comparing the plurality of occurrence moments; The occurrence time that is later than the other occurrence times among the multiple occurrence times is used as the first time.

4. The material detection method according to claim 2, characterized in that: The assembly process information includes the instantaneous position of the assembly target in each video frame to be detected; The signal sending component periodically sends a linear signal to the assembly target at intervals of the preset time, and the signal receiving component receives the linear signal reflected by the assembly target.

5. The material detection method according to claim 1, characterized in that: The detection component includes a signal sending component and a signal receiving component; the acquisition of position data along the assembly direction detected by the detection component at the first moment and the second moment includes: Obtaining a distance factor, an angle factor, a delay factor, and a transmission factor; wherein the distance factor is the distance between the signal transmitter and the signal receiver, the angle factor is the angle between the signal transmitter and the assembly direction, the delay factor is the time delay for a linear signal to be transmitted from the signal transmitter to the signal receiver, and the transmission factor is the transmission speed of the linear signal; Obtaining a trigonometric function expression of the signal sending component, the signal receiving component, and the assembly target as a first expression; wherein the first expression integrates a first position factor, a second position factor, a distance factor, and a trigonometric function, the first position factor being the distance between the signal sending component and the assembly target at a target moment, the second position factor being the distance between the signal receiving component and the assembly target at the target moment, and the target moment being the first moment or the second moment; Obtaining a second expression between the first position factor, the second position factor, the delay factor, and the transmission factor; Solving the first position factor of the target time by combining the first expression and the second expression; The trigonometric function is fitted with the solved first position factor to obtain a target position of the assembly target along the assembly direction at the target moment.

6. The material detection method according to claim 5, characterized in that: The signal sending element includes a first sending element and a second sending element; the signal receiving element includes a first receiving element, a second receiving element, a third receiving element and a fourth receiving element; the first receiving element and the second receiving element are used to obtain the linear signal sent by the first sending element; the third receiving element and the fourth receiving element are used to obtain the linear signal sent by the second sending element; The acquiring of position data along the assembly direction detected by the detection component at the first moment and the second moment includes: The target positions corresponding to the first receiving element, the second receiving element, the third receiving element, and the fourth receiving element are evaluated respectively, and weighted fusion is performed to obtain the position data.

7. The material detection method according to claim 1, characterized in that: The assembly process information obtained by obtaining the position information change when the assembly target assembles the material to be assembled includes: Extracting a video frame to be detected from the video to be detected; Positioning the assembly target at the pixel coordinates of the video frame to be detected; The pixel coordinates are mapped to a three-dimensional target coordinate system to form target coordinates, and the target coordinates are used as position information of the assembly target in the video frame to be detected.

8. The material detection method according to claim 7, characterized in that: Mapping the pixel coordinates to a three-dimensional target coordinate system to form target coordinates includes: Obtaining a matrix expression for a transformation matrix; wherein the matrix expression is obtained by fusing a joint matrix, a rotation matrix, and a translation matrix of the camera assembly; wherein the joint matrix is ​​formed based on camera intrinsic parameters and a scale factor, and the translation matrix is ​​formed based on the camera focal length; Writing the coordinate information of the detection component and the coordinate information of the assembly station into the matrix expression to solve the transformation matrix; The target coordinates are obtained by fitting the transformation matrix and the pixel coordinates.

9. The material detection method according to claim 1, characterized in that: The target tracking of the assembly target in the video to be detected includes: Extracting a video frame to be detected from the video to be detected; Inputting the video frame to be detected into a key point detection model to detect whether the video frame to be detected contains the target key point of the assembly target; In response to the key point detection model determining that the target key point exists, target tracking is performed on the target key point, and position information of the target key point in each of the to-be-detected video frames is obtained.

10. The material detection method according to claim 9, characterized in that: The key point detection model includes: a first feature extraction module, a second feature extraction module, a third feature extraction module, a first upsampling module, a first linking unit, a fourth feature extraction module, a second upsampling module, a second linking unit, a fifth feature extraction module, a third linking unit, a sixth feature extraction module, and a determination unit connected in sequence; the determination unit is used to determine whether the target key point is in the input image and output a determination result; The output end of the first feature extraction module is also connected to the input end of the second connection unit; the output end of the second feature extraction module is also connected to the input end of the first connection unit; the output end of the fourth feature extraction module is also connected to the input end of the third connection unit.

11. A material assembly method, characterized in that: The material assembly method comprises: Control the assembly target to carry the materials to be assembled to the assembly station and assemble the materials to be assembled; Determining whether the material to be assembled is in place by using the material detection method according to any one of claims 1 to 10; In response to the material to be assembled not being assembled in place, the assembly target is controlled to adjust the material to be assembled until the material to be assembled is assembled in place.

12. A material assembly device, characterized in that: The material assembly device includes: A camera assembly is provided at the assembly station to shoot the video to be inspected; A detection component, used for detecting position data of an assembly target along an assembly direction; A control component, connected to the camera component, is used to implement the material detection method according to any one of claims 1 to 10; or, to implement the material assembly method according to claim 11.

13. An electronic device, characterized in that: The electronic device comprises: memory for storing computer programs; A processor, configured to implement the steps of the material detection method according to any one of claims 1 to 10 when executing the computer program; or implement the steps of the material assembly method according to claim 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the material detection method according to any one of claims 1 to 10 are implemented; or the steps of the material assembly method according to claim 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the material detection method according to any one of claims 1 to 10 are implemented; or the steps of the material assembly method according to claim 11 are implemented.

Images