Material detection and assembly method, device, equipment, medium and program product

Abstract

The invention discloses a material detection and assembly method, device and equipment, a medium and a program product, and relates to the technical field of material production. The material detection method comprises the steps of performing target following on an assembly target used for assembling a to-be-assembled material in a to-be-detected video, and if a single assembly target can be adapted to various types of to-be-assembled materials. Moreover, the assembly starting moment when the assembly target in the to-be-detected video carries the to-be-assembled material to arrive at the assembly starting position can be identified, the moment when material assembly is completed, namely the first moment and the second moment, can be identified, and the assembly displacement of the assembly target in the assembly direction in the period is obtained. And checking whether the to-be-assembled material is assembled in place or not through the assembly displacement of the assembly target. The technical problem that the reject ratio of products is high is solved, and the technical effects that material assembly in-place detection automation is achieved, the first pass yield of the products can be improved, and meanwhile generalization of material assembly detection can be considered are achieved.

Description

Technical Field

[0001] The present application relates to the technical field of material production, and particularly relates to a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the scenario of precision manufacturing, products are usually assembled from different components as materials. For example, taking a server as an example, a CPU (Central Processing Unit), memory, acceleration card, fan, and component power supply line can be installed on the motherboard of the server. However, there is still a problem of a relatively high defective rate of products, and problems such as short circuits or structural damage after power-on are likely to occur even through factory inspections. Summary of the Invention

[0003] The present application provides a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of a relatively high defective rate of products in the related art.

[0004] The present application provides a material detection method, which includes: obtaining a video to be detected captured by a camera component; performing target tracking on an assembly target in the video to be detected, and obtaining assembly process information formed by changes in the position information of the assembly target when assembling a material to be assembled; parsing the assembly process information to obtain a first moment and a second moment; wherein the first moment is the moment when the assembly target and the material to be assembled reach the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled; obtaining position data along the assembly direction detected by the detection component at the first moment and the second moment to obtain the assembly displacement of the assembly target along the assembly direction; determining whether the assembly displacement matches a preset displacement; and in response to the assembly displacement matching the preset displacement, determining that the material to be assembled is assembled in place.

[0005] The present application further provides a material assembly method, which includes: controlling the assembly target to carry the material to be assembled to move to the assembly station and performing material assembly on the material to be assembled; using the material detection method as described above to determine whether the material to be assembled is assembled in place; and in response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0006] The present application further provides a material assembly device, which includes: a camera component, a detection component, and a control component; the camera component is used to be arranged at the assembly station to capture a video to be detected; the detection component is used to detect the position data of the assembly target along the assembly direction; the control component is connected to the camera component and is used to implement the material detection method as described above; or, implement the material assembly method as described above.

[0007] The present application also provides an electronic device, which includes: a memory and a processor; the memory is used to store a computer program; the processor is used to implement the steps of the above-mentioned material detection method when executing the computer program; or, implement the steps of the above-mentioned material assembly method.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored. Wherein, when the computer program is executed by a processor, it implements the steps of the above-mentioned material detection method; or, implements the steps of the above-mentioned material assembly method.

[0009] The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned material detection method; or, implements the steps of the above-mentioned material assembly method.

[0010] Through the present application, since target tracking is performed on the assembly target for assembling the to-be-assembled material in the to-be-detected video, thus multiple types of to-be-assembled materials can be adapted through a single assembly target. Moreover, it is possible to identify the start assembly moment when the assembly target in the to-be-detected video arrives at the assembly starting position while carrying the to-be-assembled material, and also identify the moment when the material assembly is completed, and obtain the assembly displacement of the assembly target along the assembly direction during this period, so as to verify whether the to-be-assembled material is assembled in place through the assembly displacement of the assembly target. Therefore, the technical problem of a relatively high product defect rate can be solved, achieving the automation of detecting whether the material is assembled in place to improve the product first-pass yield, and at the same time being able to take into account the generalizability of material assembly detection. Description of the Drawings

[0011] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0012] Figure 1 It is a schematic structural diagram of an embodiment of the material assembly device of the present application; Figure 2 It is a schematic structural diagram of another embodiment of the material assembly device of the present application; Figure 3 It is a schematic flowchart of an embodiment of the material detection method of the present application; Figure 4 It is a schematic flowchart of another embodiment of the material detection method of the present application; Figure 5 It is a schematic flowchart of an embodiment of the target key point detection of the present application; Figure 6 It is a schematic structural diagram of an embodiment of the first key point of the present application; Figure 7 It is a schematic flowchart of an embodiment of feature extraction by the first convolutional unit of the present application; Figure 8 It is a schematic flowchart of an embodiment of feature extraction by the second convolutional unit of the present application; Figure 9 It is a schematic flowchart of an embodiment of feature extraction by the preset feature extraction unit of the present application; Figure 10 It is a schematic flowchart of an embodiment of determination by the determination unit of the present application; Figure 11 It is a schematic flowchart of an embodiment of the material assembly method of the present application. Detailed implementation manners

[0013] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0014] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0015] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0016] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the material detection method and / or the material assembly method depends, the specific application environment architecture or specific hardware architecture will be described herein.

[0017] Embodiments of the present application provide a material assembly device. The material assembly device will be described in detail below in combination with the specific structure of the material assembly device.

[0018] Please refer to Figure 1 , Figure 1 It is a schematic structural diagram of an embodiment of the material assembly device of the present application.

[0019] In one embodiment, the material assembly device may include a camera assembly 10, a detection assembly 20, and a control assembly.

[0020] The camera assembly 10 is configured to be disposed at the assembly station to capture a video to be detected.

[0021] The detection assembly 20 is configured to detect the position data of the assembly target along the assembly direction.

[0022] The control assembly is connected to the camera assembly 10 and is configured to implement a material detection method; or, implement a material assembly method.

[0023] The material detection method may at least include obtaining the video to be detected captured by the camera assembly 10; performing target tracking on the assembly target in the video to be detected, and obtaining the assembly process information formed by the change of the position information of the assembly target when assembling the material to be assembled; parsing the assembly process information to obtain a first moment and a second moment; wherein, the first moment is the moment when the assembly target and the material to be assembled reach the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled; obtaining the position data along the assembly direction detected by the detection assembly 20 at the first moment and the second moment to obtain the assembly displacement of the assembly target along the assembly direction; determining whether the assembly displacement matches a preset displacement; in response to the assembly displacement matching the preset displacement, determining that the material to be assembled is assembled in place.

[0024] The material assembly method may at least include controlling the assembly target to carry the material to be assembled to move to the assembly station, and performing material assembly on the material to be assembled; using the above-mentioned material detection method to determine whether the material to be assembled is assembled in place; in response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0025] Please refer to Figure 1 and Figure 2 , Figure 2 , which is a schematic structural diagram of another embodiment of the material assembly device of the present application.

[0026] In one embodiment, the detection assembly 20 includes a signal transmitter and a signal receiver. The signal transmitter periodically sends a linear signal to the assembly target at preset time intervals, and the signal receiver receives the linear signal reflected by the assembly target.

[0027] Further, the signal transmitter may include a first transmitter 211 and a second transmitter 221.

[0028] The signal receiver may include a first receiver 212, a second receiver 213, a third receiver 222, and a fourth receiver 223.

[0029] The first receiving member 212 and the second receiving member 213 are used to obtain the linear signal emitted by the first transmitting member 211.

[0030] The third receiving member 222 and the fourth receiving member 223 are used to obtain the linear signal emitted by the second transmitting member 221.

[0031] Thus, in this embodiment, two transmitters and four receivers can be used to calculate the distance and speed respectively, effectively avoiding the error of a single sensor, thereby improving the sensing accuracy of the assembly target position and also improving the accuracy of the material in-place detection determination.

[0032] For the description of the features in the corresponding embodiment of the material assembly device, reference can be made to the relevant descriptions in the corresponding embodiments of the following material detection method and material assembly method, which will not be elaborated here one by one.

[0033] The embodiment of the present application provides a material detection method. The material detection method will be described in detail below in combination with the execution process of the material detection method.

[0034] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of an embodiment of the material detection method of the present application.

[0035] S101: Obtain the video to be detected captured by the camera assembly.

[0036] In this embodiment, the video to be detected represents a video of whether the assembly target to be detected has assembled the material to be assembled in place. The camera assembly is a device for capturing the video to be detected.

[0037] S102: Perform target tracking on the assembly target in the video to be detected, and obtain the assembly process information formed by the change of its position information when the assembly target assembles the material to be assembled.

[0038] In this embodiment, the assembly target existing in the video to be detected can be monitored and target-tracked to obtain the movement trajectory of the assembly target assembling the material to be assembled, that is, the assembly process information formed by the change of its position information can be obtained, so as to use the assembly process information to verify whether the assembly target has assembled the material to be assembled in place.

[0039] S103: Analyze the assembly process information to obtain the first moment and the second moment; wherein, the first moment is the moment when the assembly target and the material to be assembled reach the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled.

[0040] In this embodiment, in response to obtaining assembly process information, key moments in the video to be detected can be parsed and extracted, namely the first moment and the second moment. The first moment is the moment when the assembly target and the material to be assembled arrive at the assembly start position, and the second moment is the moment when the assembly target completes the assembly of the material to be assembled. In this way, the amount of computation required to evaluate whether the assembly target has assembled the material to be assembled can be reduced.

[0041] S104: Acquire position data along the assembly direction detected by the detection component at the first moment and the second moment to acquire assembly displacement of the assembly target along the assembly direction.

[0042] In this embodiment, the position data of the assembly target along the assembly direction detected by the detection component at the first moment can be acquired, and the position data of the assembly target along the assembly direction detected by the detection component at the second moment can also be acquired, so that the assembly displacement of the assembly target along the assembly direction at the second moment compared to the first moment can be acquired, thereby characterizing the displacement of the material to be assembled along the assembly direction.

[0043] S105: Determine whether the assembly displacement matches the preset displacement.

[0044] In this embodiment, it is possible to judge whether the assembly displacement matches the preset displacement associated with the material to be assembled, so as to verify whether the assembly displacement of the assembly target reaches the preset displacement, thereby realizing the travel movement of the material to be assembled to achieve the preset displacement, thereby indirectly verifying whether the material to be assembled is installed in place.

[0045] S106: In response to the assembly displacement matching the preset displacement, determining that the material to be assembled is assembled in place.

[0046] In this embodiment, in response to the assembly displacement of the assembly target matching the preset displacement pre-associated with the material to be assembled, it can be considered that the assembly target moves the preset displacement carrying the material to be assembled, and the material to be assembled can be assembled relatively reliably, so it can be determined that the material to be assembled is assembled in place.

[0047] It can be seen that in this embodiment, the assembly target for assembling the to-be-assembled material in the video to be detected can be target-followed. Thus, multiple types of to-be-assembled materials can be adapted through a single assembly target, and there is no need to separately train the recognition ability for multiple types of to-be-assembled materials. Training the image recognition ability for the assembly target can take into account the detection of multiple types of to-be-assembled materials. Moreover, the start assembly moment when the assembly target in the video to be detected carries the to-be-assembled material to reach the assembly starting position can be recognized, and the moment when the material assembly is completed will also be recognized, that is, the first moment and the second moment. Through the position data of the first moment and the second moment, the assembly displacement of the assembly target along the assembly direction during this period is obtained, so as to verify whether the to-be-assembled material is assembled in place through the assembly displacement of the assembly target. This solves the technical problem of a relatively high product defect rate and achieves the technical effects of realizing the automation of in-place detection of material assembly to improve the product first-pass rate and at the same time being able to take into account the generalizability of material assembly detection.

[0048] That is to say, in this embodiment, it is considered that the reason for the relatively high product defect rate is that there are assembly errors in the material assembly process, and the assembly errors are hidden, usually with small offsets or angular deviations that are difficult to identify by traditional vision or mechanical touch. For example, if workpiece A is not fully inserted into the card slot of workpiece B, directly powering on may cause a short circuit or structural damage, resulting in high maintenance costs and limitations of detection means. Relying on static measurement is difficult to meet the real-time requirements of high-speed production lines, and the internal structure of complex workpieces blocks the detection blind area. The interference of the environment, the shape differences of different components, and the manufacturing tolerances of the same component will also increase the difficulty of assembly detection.

[0049] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of another embodiment of the material detection method of this application.

[0050] S201: Obtain the video to be detected.

[0051] In this embodiment, the video to be detected may include one or more video frames to be detected. The video frames to be detected in the video to be detected can be extracted in sequence and the assembly target detection can be performed in sequence.

[0052] Furthermore, the video frames to be detected can be preprocessed. For example, the video frames to be detected can be scaled to a preset picture size, super-resolution reconstruction, etc.

[0053] For example, the preset picture size can be 640*640. During the scaling process of the video frames to be detected, the aspect ratio of the picture can be kept unchanged during scaling, the maximum side is scaled to 640, and the smaller side is filled with gray scale to retain the original image features.

[0054] S202: Input the video frame to be detected into the key point detection model for target key point detection.

[0055] In this embodiment, input the video frame to be detected into the key point detection model to detect whether there are target key points of the assembly target in the video frame to be detected. In response to the key point detection model determining that there are target key points, perform target tracking on the target key points, and obtain the position information of the target key points in each video frame to be detected.

[0056] That is to say, it is possible to detect whether the video frame to be detected contains specific key point information, that is, the target key points of the assembly target, through the key point detection model.

[0057] For example, the assembly target can be a robotic arm or a human hand, etc., and there is no strict limitation here. For example, for a robotic arm, components near the material to be assembled held by the robotic arm, the robotic arm body, etc. can be selected as target key points, and for a human hand, fingers, wrists, elbows, etc. can be selected as target key points.

[0058] In this way, compared with identifying the overall motion trajectory and assembly displacement of the assembly target, identifying the motion trajectory and assembly displacement of the target key points can more reliably confirm the depth of the assembly target assembling the material to be assembled, can improve the recognition granularity, which is beneficial to reducing the risk of misrecognition and unreliable detection results, and further can improve the reliability of the in-place detection of material assembly.

[0059] S203: Determine whether there are target key points.

[0060] In this embodiment, when it is determined that there are target key points, step S204 can be executed. When it is determined that there are no target key points, step S202 can be executed.

[0061] It is possible to identify the determination result output by the key point detection model to confirm whether there is an assembly target and target key points in the video frame to be detected for current assembly target detection.

[0062] S204: Obtain the target coordinates of the target key points in the target coordinate system as its position information.

[0063] In this embodiment, locate the pixel coordinates of the assembly target in the video frame to be detected. Map the pixel coordinates to a three-dimensional target coordinate system to form target coordinates, and use the target coordinates as the position information of the assembly target in the video frame to be detected.

[0064] Specifically, the matrix expression of the transformation matrix can be obtained. Among them, the matrix expression is obtained by fusing the joint matrix, rotation matrix, and translation matrix of the fusion camera component. Among them, the joint matrix is formed based on the camera internal parameters and the scale factor, and the translation matrix is formed based on the camera focal length. Write the coordinate information of the detection component and the coordinate information of the assembly station into the matrix expression to solve the transformation matrix. Fit the transformation matrix and the pixel coordinates to obtain the target coordinates.

[0065] For example, the pixel coordinates can be represented as (x, y), and the target coordinates can be represented as [X, Y]. The following takes the second sending part as the origin of the target coordinate system for example.

[0066] According to the installation position of the camera component, the coordinates P1(x1, y1) of the second sending part, the coordinates P2(x2, y2) of the third receiving part, the coordinates P3(x3, y3) of the fourth receiving part, and the coordinates P4(x4, y4), P5(x5, y5), P6(x6, y6) of any three non-collinear points on the assembly station can be used as reference points to perform the conversion from pixel coordinates to target coordinates.

[0067] That is to say, the assembly station can be used as the reference plane and the second sending part as the origin to establish the real-world coordinate system as the target coordinate system. With such a design, the two-dimensional position information in the video frame to be detected can be converted into the three-dimensional position information in the real world. At the same time, the movement trajectory and other small changes in the pixels can also be magnified, which is beneficial to improving the recognition reliability of small displacements, and thus beneficial to improving the reliability of material arrival detection.

[0068] The relationship between the pixel coordinates and the target coordinates can be constructed. The target coordinates can be as exemplified by the following formula: Formula 1-1 Formula 1-2 Formula 1-3 Formula 1-4 Among them, K represents the joint matrix; R represents the rotation matrix; t represents the translation matrix of the camera; f x and f y represent the focal length of the camera; c x and c y represent the camera internal parameters; r represents the rotation vector.

[0069] In this way, the conversion relationship of Formula 1-1 can be simplified to form the conversion expression of the pixel coordinates and the target coordinates combined with the transformation matrix, as exemplified by the following: Formula 1-5 Formula 1-6 Among them, P represents the conversion matrix; represents a matrix expression; p represents the specific value to be obtained currently.

[0070] Solve the pixel conversion matrix P, and substitute , , , , and into Formula 1-5, and the specific expression of the conversion matrix P can be obtained.

[0071] Thus, the target coordinate expression of the target key point is specifically expressed as: Formula 1-7 Of course, in an alternative embodiment, the ratio between the target coordinate system and the pixel coordinate can also be other ratios, without the need to match the real-world coordinate system, or other coordinate origins and reference planes can be selected, which will not be elaborated here.

[0072] S205: The signal transmitter follows the assembly target.

[0073] In this embodiment, when the assembly target assembles the material to be assembled, it can carry the material to be assembled and move. In order to perform reliable position detection, the signal transmitter can be controlled to change the linear signal transmission angle according to the assembly target. Optionally, it can be manually adjusted by external staff, or it can be automatically followed through software and / or hardware control.

[0074] Specifically, in this embodiment, taking the realization of the assembly target following through the first transmitter, the first receiver, and the second receiver as an example. It can be made that the signal transmission end of the first transmitter, the signal receiving ends of the first receiver and the second receiver are collinear, and the relative directions of the three are perpendicular to the surface of the assembly station.

[0075] Thus, the first included angle between the first virtual connection line and the second virtual connection line can be calculated; the first virtual connection line represents the virtual connection line between the first receiver and the assembly target and the material to be assembled, and the second virtual connection line represents the virtual connection line between the first receiver and the second receiver. Thus, the angle of the second included angle complementary to the first included angle can be calculated, and the second included angle is the included angle between the first virtual connection line and the third virtual connection line, and the third virtual connection line represents the virtual connection line between the first receiver and the first transmitter. Thus, the third included angle associated with the second included angle can be solved by the cosine theorem, and the third included angle is the included angle between the ideal virtual connection line between the first transmitter and the assembly target and the third virtual connection line, so that the angle of the first transmitter can be adjusted according to the calculated third included angle.

[0076] Two signal receivers can be used to associate with a signal transmitter to calculate the angle between the transmitter and the point to be detected, facilitating the adjustment of the angle of the signal generator and inferring the angle of the material relative to the signal generator.

[0077] Similarly, the assembly target following principle of the second transmitter, the third receiver, and the fourth receiver is similar, and will not be elaborated here.

[0078] S206: Calculate the moving speed of the assembly target.

[0079] In this embodiment, the assembly process information includes the instantaneous position of the assembly target in each video frame to be detected. Thus, the moving speed of the assembly target can be calculated based on the instantaneous positions between the video frames to be detected.

[0080] Specifically, obtain the distance between the current position of the signal receiver and the instantaneous position when the assembly target reflects the linear signal as the signal transmission distance. Use the signal transmission distance, the preset time, and the pulse wavelength of the linear signal to evaluate the signal phase difference between the signal receiver receiving two adjacent linear signals. Fit the pulse wavelength and the signal phase difference to obtain a fitting intermediate value. Take the ratio of the fitting intermediate value to the duration factor as the moving speed of the assembly target between two adjacent linear signals. Among them, the duration factor is obtained by magnifying the preset time.

[0081] The specific calculation method will be elaborated in detail later.

[0082] S207: Determine whether the assembly starting position appears.

[0083] In this embodiment, when it is determined that the assembly starting position appears, step S208 can be executed. When it is determined that the assembly starting position does not appear, step S202 can be executed.

[0084] Use the assembly process information to calculate the moving speed of the assembly target and the change rate of the moving speed at each preset time interval. Select the occurrence moment of the moving speed with a change rate lower than the rate threshold as the first moment, and determine that the assembly target reaches the assembly starting position.

[0085] Furthermore, in response to there being multiple occurrence moments with a change rate lower than the rate threshold, compare the multiple occurrence moments. The occurrence moment later than other occurrence moments among the multiple occurrence moments can be used as the first moment.

[0086] Thus, the moment when the speed changes is a key point in the detection process, which can simplify the calculation of the moving distance of the material to be assembled and the assembly target. It is only necessary to calculate the position difference at the last speed change and when the speed stops, which can effectively reduce the operation amount of material in-place detection and is thus conducive to maintaining the reliable operation of the material detection method. Moreover, it is possible to simplify the decomposition of the detection process by combining the prior knowledge that the movement speed of the material to be assembled will be blocked and reduced at the start of assembly.

[0087] S208: Evaluate the target position of the assembly target at the assembly starting position.

[0088] In this embodiment, in response to determining that the assembly target is at the assembly starting position, the position information of its current position can be obtained as the target position at the assembly starting position. Optionally, it can be the target coordinates of the assembly target or the height position in the assembly direction relative to the assembly station, i.e., the target position d dis1 (This will be used as an example in this embodiment) and so on.

[0089] S209: Determine whether the assembly of the material to be assembled is completed.

[0090] In this embodiment, when it is determined that the assembly of the material to be assembled is completed, step S210 can be executed. When it is determined that the assembly of the material to be assembled is not completed, step S202 can be executed.

[0091] Select the occurrence moment when the moving speed is zero as the second moment, and determine that the assembly target has completed the assembly of the material to be assembled.

[0092] S210: Evaluate the target position of the assembly target when it has completed the assembly of the material to be assembled.

[0093] In this embodiment, obtain the target position d of the assembly target when it has completed the assembly of the material to be assembled dis2 .

[0094] S211: Obtain the assembly displacement of the assembly target in the assembly direction.

[0095] In this embodiment, the target position d dis2 and the target position d dis1 can be obtained, and the difference between them is used as the assembly displacement. The specific calculation formula can be as follows: Equation 2-1 where ∆d represents the assembly displacement.

[0096] S212: Evaluate whether the material to be assembled is assembled in place using the assembly displacement.

[0097] In this embodiment, when it is determined that the material to be assembled is assembled in place by using the assembly displacement, step S213 can be executed. When it is determined that the material to be assembled is not assembled in place by using the assembly displacement, step S214 can be executed.

[0098] S213: Determine that the material to be assembled is assembled in place.

[0099] In this embodiment, in response to determining that the material to be assembled is assembled in place, it can be considered that the material to be assembled has been reliably assembled, and the assembly of the next material to be assembled can be carried out, etc.

[0100] S214: Determine that the material to be assembled is not assembled in place.

[0101] In this embodiment, in response to the material to be assembled not being assembled in place, it can be considered that the current state of the material to be assembled is likely to result in defective products. Therefore, a prompt message indicating that it is not assembled in place can be sent to promote the timely and reliable assembly of the material to be assembled. Further, when the material is assembled by a controllable and schedulable robotic arm, an adjustment instruction can also be adaptively formed and sent to the robotic arm to achieve the automation of material assembly adjustment.

[0102] For example, when the adjustment signal sender follows the assembly target, it can move and adjust following the assembly target in real time. Whether the assembly target assembles the material to be assembled in place can be determined after the assembly target completes the current assembly of the material to be assembled, so as to globally and reliably identify the first moment and the second moment in combination with the video to be detected.

[0103] The following gives examples to illustrate the specific acquisition methods of the assembly target at the first moment and the second moment.

[0104] Obtain a distance factor, an angle factor, a delay factor, and a transmission factor. Among them, the distance factor is the distance between the signal sender and the signal receiver, the angle factor is the angle between the signal sender and the assembly direction, the delay factor is the time delay of the linear signal transmitted from the signal sender to the signal receiver, and the transmission factor is the transmission speed of the linear signal.

[0105] Obtain the trigonometric function expressions of the signal sender, the signal receiver, and the assembly target as the first expression. Among them, the first expression combines a first position factor, a second position factor, a distance factor, and trigonometric functions. The first position factor is the distance between the signal sender and the assembly target at the target moment, the second position factor is the distance between the signal receiver and the assembly target at the target moment, and the target moment is the first moment or the second moment.

[0106] Obtain the second expression between the first position factor, the second position factor, the delay factor, and the transmission factor.

[0107] Solve for the first position factor at the target time by combining the first expression and the second expression.

[0108] Fit a trigonometric function to the solved first position factor as the target position of the assembly target along the assembly direction at the target time.

[0109] For example, the distance between the first receiving part and the first transmitting part is the distance factor h1, the angle between the first transmitting part and the assembly direction such as the vertical direction is the angle factor θ1, the time delay between the received signal and the transmitted signal is the delay factor ∆t1, and the transmission speed of the pulse signal, i.e., the linear signal, is c.

[0110] Based on the triangle formed by the first transmitting part, the first receiving part, and the target key point, trigonometric function expressions for the signal transmitting part, the signal receiving part, and the assembly target can be obtained. The first expression can be specifically represented as: Equation 3-1 where l 11 represents the distance between the first transmitting part and the target key point, which is equivalent to the first position factor; l 12 represents the distance between the first receiving part and the target key point, which is equivalent to the second position factor of the first receiving part; θ1 represents the angle at the start of assembly.

[0111] The second expression for obtaining the first position factor and the second position factor in relation to the delay factor and the transmission factor can be specifically represented as: Equation 3-2 Thus, by combining Equation 3-1 and Equation 3-2, the first position factor l 11 can be solved for its value.

[0112] The specific calculation method of fitting a trigonometric function to the solved first position factor as the target position of the assembly target along the assembly direction at the target time can be exemplified as follows: Equation 3-3 Equation 3-4 where d1 represents the target position solved at the start of assembly; d2 represents the target position solved when assembly is completed; θ2 represents the angle when assembly is completed; l 21 represents the first position value factor when assembly is completed.

[0113] Furthermore, as Figure 2As shown in the examples, the signal transmitter includes a first transmitter and a second transmitter; the signal receiver includes a first receiver, a second receiver, a third receiver, and a fourth receiver. Thus, the target positions corresponding to the first receiver, the second receiver, the third receiver, and the fourth receiver can be evaluated respectively, and weighted fusion is performed to obtain position data. The specific calculation formula can be as shown in the examples below: Equation 4-1 Equation 4-2 Among them, , α, β, γ, These four are weight factors, which are hyperparameters and can be set according to the actual situation; d 11 represents the target position solved by the first receiver at the assembly starting position; d 12 represents the target position solved by the second receiver at the assembly starting position; d 13 represents the target position solved by the third receiver at the assembly starting position; d 14 represents the target position solved by the fourth receiver at the assembly starting position; d 21 represents the target position solved by the first receiver when the assembly is completed; d 22 represents the target position solved by the second receiver when the assembly is completed; d 23 represents the target position solved by the third receiver when the assembly is completed; d 24 represents the target position solved by the fourth receiver when the assembly is completed.

[0114] Please refer to Figure 5 and Figure 6 , Figure 5 which is a schematic flowchart of an embodiment of the target key point detection of the present application, Figure 6 and

[0115]

[0116] Figure 5 In one embodiment, the key point detection model includes: a first feature extraction module, a second feature extraction module, a third feature extraction module, a first upsampling module, a first connection unit, a fourth feature extraction module, a second upsampling module, a second connection unit, a fifth feature extraction module, a third connection unit, a sixth feature extraction module, and a determination unit connected in sequence. The determination unit is used to determine whether there are target key points in the input image and output a determination result. The output end of the first feature extraction module is also connected to the input end of the second connection unit. The output end of the second feature extraction module is also connected to the input end of the first connection unit. The output end of the fourth feature extraction module is also connected to the input end of the third connection unit. At the same time, Figure 5The dimensions of the video frame to be detected after passing through various key-point detection model modules, units, etc. are identified, such as [1, 3, 640, 640]. Among them, 1 represents the number of video frames to be detected input into the key-point detection model (i.e., batch size), 3 represents the number of channels, and 640 and 640 are the image sizes. Dimensions such as 0 - P1 / 2 are used to indicate dimensionality reduction, where 0 - P1 / 2 represents the first dimensionality reduction.

[0117] As shown in the example in 6, when the assembly target is a human hand, it may include target key point K1, target key point K2, and target key point K3.

[0118] When the key-point detection model obtains the video frame to be detected, the first feature extraction module performs first feature extraction processing on it to obtain the first image feature, and transmits the first image feature to the second feature extraction module and the second connection unit.

[0119] The second feature extraction module performs second feature extraction processing on the first image feature to obtain the second image feature, and transmits the second image feature to the third feature extraction module and the first connection unit.

[0120] The third feature extraction module performs third feature extraction processing on the second image feature to obtain the third image feature, and transmits the third image feature to the first upsampling module.

[0121] The first upsampling module performs first upsampling processing on the third image feature to obtain the first intermediate feature, and transmits the first intermediate feature to the first connection unit.

[0122] The first connection unit connects the first intermediate feature and the second image feature to form the first connection feature, and transmits the first connection feature to the fourth feature extraction module.

[0123] The fourth feature extraction module performs fourth feature extraction processing on the first connection feature to obtain the fourth image feature, and transmits the fourth image feature to the second upsampling module and the third connection unit.

[0124] The second upsampling module performs second upsampling processing on the fourth image feature to obtain the second intermediate feature, and transmits the second intermediate feature to the second connection unit.

[0125] The second connection unit connects the first image feature and the second intermediate feature to form the second connection feature, and transmits the second connection feature to the fifth feature extraction module.

[0126] The fifth feature extraction module performs fifth feature extraction processing on the second connection feature to form the fifth image feature, and transmits the fifth image feature to the third connection unit.

[0127] The third connection unit connects the fifth image feature and the fourth image feature to form a third connection feature, and transmits the third connection feature to the sixth feature extraction module.

[0128] The sixth feature extraction module performs a sixth feature extraction process on the third connection feature to form a sixth image feature, and transmits the sixth image feature to the determination unit.

[0129] The determination unit analyzes the sixth image feature to determine whether the input video frame to be detected contains the target key points.

[0130] Furthermore, in this embodiment, two convolutional units can be pre-constructed, namely the first convolutional unit Conv and the second convolutional unit C3f.

[0131] As Figure 7 exemplarily shown in Figure 7 is a schematic flowchart of an embodiment of feature extraction by the first convolutional unit of this application.

[0132] The first convolutional unit Conv may include a two-dimensional convolutional layer Conv2d, a normalization layer BN, and an activation function SiLU.

[0133] As Figure 8 exemplarily shown in Figure 8 is a schematic flowchart of an embodiment of feature extraction by the second convolutional unit of this application.

[0134] The second convolutional unit C3f may include a convolutional layer Conv1, a data splitting layer split, a preset feature extraction unit B1, a connection function contact, and a convolutional layer Conv1.

[0135] Among them, split can perform channel binary division and then merge, that is, it retains both the original features and the features after passing through B1. C3f can keep the feature shapes of the input and output unchanged to facilitate the construction of the model.

[0136] In addition, at least one of the first connection unit, the second connection unit, and the third connection unit may also be a connection function contact, etc.

[0137] Furthermore, Figure 5 also identifies the parameters of each first convolutional unit Conv and second convolutional unit C3f. Such as k, s, p, c, n = 6xd, etc. Among them, k represents the convolutional kernel size, s represents the padding, p represents the filling, and c represents the number of output channels. In m = 6xd, d represents a hyperparameter that can be defaulted to 1, "x" is the multiplication sign, and m is applied to Figure 8 "(m - 1)xB1" upstream of the connection function contact in

[0138] As Figure 9As exemplified in Figure 9 is a schematic flowchart of an embodiment of feature extraction by the preset feature extraction unit of the present application.

[0139] The preset feature extraction unit B1 may include a convolutional layer Conv1 and a convolutional layer Conv2.

[0140] Please refer to Figure 10 , Figure 10 is a schematic flowchart of an embodiment of determination by the determination unit of the present application.

[0141] The determination unit detect may include a depthwise separable convolutional layer DWConv, a two-dimensional convolutional layer Conv2d, and an output layer. The output layer may include a prediction score Score, a classification feature vector Cls, and an output Point, which respectively correspond to the confidence level, classification, and target key point position.

[0142] Generally speaking, the detection of target key points may specifically include the following steps.

[0143] The input end of the key point detection model may input an image to be detected, that is, a video frame to be detected. After passing through the Conv layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map F1 is obtained; F1 passes through the Conv layer to obtain a new feature map; after passing through the C3f layer, a new feature map F2 is obtained; F2 passes through the Conv layer to obtain a new feature map; after passing through the C3f layer, a new feature map is obtained; upsampling is performed to obtain a new feature map F3; the new feature map F3 and the feature map F2 are merged; after passing through the C3f layer, a new feature map F4 is obtained; F4 is upsampled to obtain a new feature map; the new feature map and the feature map F1 are merged; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; the new feature map and the feature map F4 are merged; after passing through the C3f layer, a new feature map is obtained; after passing through the Conv layer, a new feature map is obtained; after passing through the C3f layer, a new feature map is obtained; after passing through the Detect layer, the detection point position, classification, and confidence level are obtained. Among them, the markings of the feature maps F1 to F4 are not shown in the drawings and are only used to distinguish the feature maps in this paragraph.

[0144] Furthermore, the model loss function of the key point detection model may include a detection point position loss, a classification loss, and a confidence loss, and the specific calculation is as exemplified below: Equation 5-1 Equation 5-2 Equation 5-3 Formula 5-4 where Loss represents the model loss function; ε, ζ, τ are weight factors, which can be hyperparameters, etc.; loss p represents the detection point loss; (x i , y i ) is the coordinate of the i-th target key point predicted by the key point detection model; (x igt , y igt ) is the true point coordinate of the target key point; n represents the number of target key points; loss c represents the classification loss; y ij represents the true classification, c ij represents the predicted classification; loss s represents the confidence loss; s ij is whether the category exists, with two values of 0 and 1; p ij is the probability of the correct prediction of the category.

[0145] In summary, the material arrival detection of this application can adopt non-direct contact detection, which can effectively reduce the risk of damaging the materials to be assembled during the arrival detection process. Moreover, by detecting the key points of the assembly target (such as the wrist key points, etc.), avoiding directly identifying and detecting various materials to be assembled, it can enhance the versatility of the material arrival detection, enabling different materials to be assembled to be detected using the same key point detection model, and eliminating the need to train a detection model for each type of material to be assembled separately. It can also simplify the conversion process from the camera coordinate system to the world coordinate system, and can calculate using the pre-calibrated points of the conversion matrix, reducing the interference of the camera internal parameters, which is beneficial to simplifying the deployment and use of the material detection method. At the same time, it can also reduce the number of detection heads such as the determination unit, and only one detection head can be retained, which can improve the detection efficiency of the key point detection model and is beneficial to deploying the key point detection model in small servers such as edge micro-servers.

[0146] In the detection head part, a depthwise separable convolutional layer can be adopted to further reduce the number of parameters of the key point detection model. At the same time, the material assembly device can adopt two transmitters and four receivers to calculate the distance and speed respectively, effectively avoiding the error of a single sensor, thereby improving the sensing accuracy of the assembly target position and the accuracy of the material arrival detection determination. Moreover, by using the moment when the speed changes as the key point in the detection process, it can simplify the calculation of the moving distance of the materials to be assembled and the assembly target, and only the position difference between the last speed change and the speed stop needs to be calculated, which can effectively reduce the operation amount of the material arrival detection and is thus beneficial to maintaining the reliable operation of the material detection method.

[0147] Embodiments of the present application provide a material assembly method. The material assembly method will be described in detail below in combination with the execution process of the material assembly method.

[0148] Please refer to Figure 11 , Figure 11 , which is a schematic flowchart of an embodiment of the material assembly method of the present application.

[0149] S301: Control the assembly target to move the material to be assembled to the assembly station and perform material assembly on the material to be assembled.

[0150] S302: Use the material detection method to determine whether the material to be assembled is assembled in place.

[0151] S303: In response to the material to be assembled not being assembled in place, control the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

[0152] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, or a combination of software and hardware.

[0153] Embodiments of the present application also provide an electronic device.

[0154] The electronic device may include a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above embodiments of the material detection method or the material assembly method. That is, the memory is used to store the computer program. The processor is used to implement the steps of the material detection method as described above when executing the computer program; or, implement the steps of the material assembly method as described above.

[0155] Embodiments of the present application also provide a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above embodiments of the material detection method or the material assembly method when running. That is, the computer program implements the steps of the material detection method as described above when executed by the processor; or, implements the steps of the material assembly method as described above.

[0156] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs and other various media that can store computer programs.

[0157] Embodiments of the present application further provide a computer program product. The computer program product may include a computer program, and when the computer program is executed by a processor, the steps in any of the above-described material detection method or material assembly method embodiments are implemented. That is, when the computer program is executed by a processor, the steps of the material detection method described above are implemented; or, the steps of the material assembly method described above are implemented.

[0158] Embodiments of the present application further provide another computer program product. The computer program product may include a non-volatile computer-readable storage medium that stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-described material detection method or material assembly method embodiments are implemented.

[0159] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0160] The above has introduced in detail a material detection method, a material assembly method, a material assembly device, an electronic device, a computer-readable storage medium, and a computer program product provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A material detection method, characterized in that, The material detection method includes: Obtaining a to-be-detected video captured by a camera component; Performing target tracking on an assembly target in the to-be-detected video, and obtaining assembly process information formed by changes in its position information when the assembly target assembles a to-be-assembled material; Analyzing the assembly process information to obtain a first moment and a second moment; wherein, the first moment is the moment when the assembly target and the to-be-assembled material reach the assembly starting position, and the second moment is the moment when the assembly target completes the assembly of the to-be-assembled material; Obtaining position data in the assembly direction detected by a detection component at the first moment and the second moment, so as to obtain the assembly displacement of the assembly target in the assembly direction; Judging whether the assembly displacement matches a preset displacement; In response to the assembly displacement matching the preset displacement, determining that the to-be-assembled material is assembled in place.

2. The material detection method according to claim 1, wherein The analyzing the assembly process information to obtain a first moment and a second moment includes: Calculating the moving speed of the assembly target and the change rate of the moving speed at each preset time interval by using the assembly process information; Selecting the occurrence moment of the moving speed whose change rate is lower than a rate threshold as the first moment, and determining that the assembly target reaches the assembly starting position; Selecting the occurrence moment when the moving speed is zero as the second moment, and determining that the assembly target completes the assembly of the to-be-assembled material.

3. The material detection method according to claim 2, characterized in that, The selecting the occurrence moment of the moving speed whose change rate is lower than a rate threshold as the first moment includes: In response to there being multiple occurrence moments where the change rate is lower than the rate threshold, comparing the multiple occurrence moments; Taking the occurrence moment among the multiple occurrence moments that is later than other occurrence moments as the first moment.

4. The material detection method according to claim 2, characterized in that The assembly process information includes the instant position of the assembly target in each to-be-detected video frame; The detection component includes a signal sender and a signal receiver. The signal sender periodically sends a linear signal to the assembly target at intervals of the preset time, and the signal receiver receives the linear signal reflected by the assembly target; The calculating the moving speed of the assembly target by using the assembly process information includes: Obtaining the distance between the current position of the signal receiver and the instant position when the assembly target reflects the linear signal, as the signal transmission distance; Evaluating the signal phase difference between two adjacent linear signals received by the signal receiver by using the signal transmission distance, the preset time, and the pulse wavelength of the linear signal; Fitting the pulse wavelength and the signal phase difference to obtain a fitting intermediate value; taking the ratio of the fitting intermediate value to a duration factor as the moving speed of the assembly target between two adjacent linear signals; wherein, the duration factor is obtained by magnifying the preset time.

5. The material detection method according to claim 1, characterized in that The detection component includes a signal sender and a signal receiver; the obtaining the position data in the assembly direction detected by the detection component at the first moment and the second moment includes: Obtain the distance factor, angle factor, delay factor, and transmission factor; wherein, the distance factor is the distance between the signal sender and the signal receiver, the angle factor is the angle between the signal sender and the assembly direction, the delay factor is the time delay for the linear signal to be transmitted from the signal sender to the signal receiver, and the transmission factor is the transmission speed of the linear signal; Obtain the trigonometric function expression of the signal sender, the signal receiver, and the assembly target as the first expression; wherein, the first expression combines the first position factor, the second position factor, the distance factor, and the trigonometric function, the first position factor is the distance between the signal sender and the assembly target at the target moment, the second position factor is the distance between the signal receiver and the assembly target at the target moment, and the target moment is the first moment or the second moment; Obtain the second expression between the first position factor and the second position factor and the delay factor and the transmission factor; Combine the first expression and the second expression to solve the first position factor at the target moment; Fit the trigonometric function and the solved first position factor as the target position of the assembly target along the assembly direction at the target moment.

6. The material detection method according to claim 5, wherein The signal sender includes a first sender and a second sender; the signal receiver includes a first receiver, a second receiver, a third receiver, and a fourth receiver; the first receiver and the second receiver are used to obtain the linear signal emitted by the first sender; the third receiver and the fourth receiver are used to obtain the linear signal emitted by the second sender; The obtaining of the position data along the assembly direction detected by the detection component at the first moment and the second moment includes: Respectively evaluate the target positions corresponding to the first receiver, the second receiver, the third receiver, and the fourth receiver, and perform weighted fusion to obtain the position data.

7. The material detection method according to claim 1, characterized in that The obtaining of the assembly process information formed by the change of the position information of the assembly target when assembling the material to be assembled includes: Extract the video frames to be detected in the video to be detected; Locate the pixel coordinates of the assembly target in the video frame to be detected; Map the pixel coordinates to a three-dimensional target coordinate system to form target coordinates, and use the target coordinates as the position information of the assembly target in the video frame to be detected.

8. The material detection method according to claim 7, wherein The mapping of the pixel coordinates to a three-dimensional target coordinate system to form target coordinates includes: Obtain the matrix expression of the transformation matrix; wherein, the matrix expression is obtained by combining the joint matrix, rotation matrix, and translation matrix of the imaging component; wherein, the joint matrix is formed based on the camera internal parameters and the scale factor, and the translation matrix is formed based on the camera focal length; Write the coordinate information of the detection component and the coordinate information of the assembly station into the matrix expression to solve the transformation matrix; Fit the transformation matrix and the pixel coordinates to obtain the target coordinates.

9. The material detection method according to claim 1, wherein The target tracking of the assembly target in the video to be detected includes: Extract the video frames to be detected in the video to be detected; Input the video frames to be detected into a key point detection model to detect whether there are target key points of the assembly target in the video frames to be detected; In response to the key point detection model determining the existence of the target key points, perform target tracking on the target key points and obtain the position information of the target key points in each of the video frames to be detected.

10. The material detection method according to claim 9, characterized in that, The key point detection model includes: a first feature extraction module, a second feature extraction module, a third feature extraction module, a first upsampling module, a first connection unit, a fourth feature extraction module, a second upsampling module, a second connection unit, a fifth feature extraction module, a third connection unit, a sixth feature extraction module, and a determination unit connected in sequence; the determination unit is used to determine whether there are the target key points in the input image and output a determination result; The output end of the first feature extraction module is further connected to the input end of the second connection unit; the output end of the second feature extraction module is further connected to the input end of the first connection unit; the output end of the fourth feature extraction module is further connected to the input end of the third connection unit.

11. A material assembly method, characterized in that, The material assembly method includes: Controlling the assembly target to carry the material to be assembled and move it to the assembly station, and performing material assembly on the material to be assembled; Using the material detection method according to any one of claims 1 to 10 to determine whether the material to be assembled is assembled in place; In response to the material to be assembled not being assembled in place, controlling the assembly target to adjust the material to be assembled until the material to be assembled is assembled in place.

12. A material assembly device, characterized in that, The material assembly device includes: A camera assembly for being arranged at the assembly station to capture a video to be detected; A detection assembly for detecting the position data of the assembly target along the assembly direction; A control assembly connected to the camera assembly for implementing the material detection method according to any one of claims 1 to 10; or, implementing the material assembly method according to claim 11.

13. An electronic device, characterized in that, The electronic device includes: A memory for storing a computer program; A processor for implementing the steps of the material detection method according to any one of claims 1 to 10 when executing the computer program; or, implementing the steps of the material assembly method according to claim 11.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the material detection method according to any one of claims 1 to 10; or, implements the steps of the material assembly method according to claim 11.

15. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the material detection method according to any one of claims 1 to 10; or, implements the steps of the material assembly method according to claim 11.

Images