A vision-based robotic arm assembly control method

Abstract

The present invention relates to the technical field of robotic arm control, and discloses a robotic arm assembly control method based on vision migration, the method comprising: utilizing the robotic arm's vision system to acquire an image of a device to be assembled, and automatically preprocessing the acquired image of the device to be assembled; inputting the preprocessed image of the device to be assembled into an offline product registration model to obtain a registration fit; if the registration fit is greater than a specified threshold, assembly can be performed; otherwise, the preprocessed image of the device to be assembled is input into a real-time product registration robust control model for correction; and controlling the robotic arm to perform assembly processing of the device to be assembled. The present invention determines the assembly rotation angle of the device to be assembled at an initial position based on the image's registration fit, controls the robotic arm to move along an assembly path based on an obstacle avoidance algorithm, reaches the assembly position, migrates the rotation angle based on the initial position vision to the final assembly position, controls the robotic arm to rotate, and realizes assembly control based on robotic arm vision.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arm control, and in particular to a robotic arm assembly control method based on visual migration. Background Art

[0002] Product assembly using robotic arms has been widely used in industrial manufacturing. Existing autonomous robotic arm assembly technologies are primarily divided into two types: programmable logic controller (PLC) control and visual control. PLC control has simple logic and is suitable for simple assembly scenarios, but is inadequate for product assembly in complex scenarios. Visual control technology, on the other hand, can process large amounts of scene information and is suitable for product assembly in complex environments. However, it suffers from poor robustness, namely, poor scene adaptability and an inability to achieve adaptive assembly control under multiple different environmental conditions. To address this issue, the present invention proposes a robotic arm assembly control method using visual transfer, focusing on robust robotic arm assembly control under complex and changing conditions. Summary of the Invention

[0003] In view of this, the present invention provides a robotic arm assembly control method of visual migration, the purpose of which is: 1) dividing the image of the device to be assembled into several sub-images based on the image edges divided after preprocessing, and calculating the convolution feature maps corresponding to the sub-images and the template registration image to obtain the semantic similarity between the two, and selecting the sub-image with the largest semantic similarity, extracting several local feature maps, and thus determining the position area of ​​the device to be assembled in the image of the device to be assembled based on the cosine similarity between the local feature map and the convolution feature map of the device to be assembled in the template registration image, converting the device to be assembled into a three-dimensional image representation, and randomly rotating and combining the represented device to be assembled and the assembly position to obtain different combination results, calculating the registration fit between the different combination results and the template registration image, selecting the largest combination result and the corresponding registration fit for output, and if the registration fit is greater than a specified threshold, instructing the robotic arm to calculate the rotation angle in the combination result, and controlling the robotic arm to rotate After arriving at the assembly position, the device is rotated and assembled according to the recorded rotation angle. At the same time, a real-time correction scheme for the image of the device to be assembled is proposed. By using the correction code to correct the image of the device to be assembled, a larger range of corrections is made to the middle area of ​​the image, and the possibility of the device to be assembled in the middle area is greater, thereby improving the recognition efficiency of the device to be assembled; 2) The connecting rod between adjacent joints of the robot arm is used as the reference unit of the obstacle avoidance algorithm. By calculating the virtual force conditions of the connecting rod and the obstacle between adjacent joint nodes, it is determined whether there is a collision between the connecting rod and the obstacle at each path node in the assembly path, and the path nodes with collision are corrected to obtain an assembly path that avoids collision of the robot arm. The robot arm is controlled to move along the assembly path to reach the assembly position, and the robot arm is controlled to rotate the device to be assembled according to the recorded rotation angle. The rotation angle based on the initial position vision is migrated to the final assembly position, realizing assembly control based on robot arm vision.

[0004] To achieve the above objectives, the present invention provides a vision-based robotic arm assembly control method comprising the following steps:

[0005] S1: Using the vision system of the robotic arm to obtain an image of the device to be assembled, and automatically preprocessing the obtained image of the device to be assembled;

[0006] S2: Build an offline product registration model, which takes the template registration image and the image of the device to be assembled as input.

[0007] The output is the registration fit;

[0008] S3: Build a real-time product registration robust control model, which takes the image of the device to be assembled as input and outputs the corrected image of the device to be registered;

[0009] S4: Input the preprocessed image of the device to be assembled into the offline product registration model to obtain the registration consistency. If the registration consistency is greater than a specified threshold, assembly can be performed. Otherwise, input the preprocessed image of the device to be assembled into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, and input the corrected image of the device to be assembled into the offline product registration model.

[0010] S5: According to the image of the device to be assembled that can be assembled, the robot arm is controlled to assemble the device to be assembled.

[0011] As a further improvement method of the present invention:

[0012] Optionally, obtaining an image of the device to be assembled in step S1 includes:

[0013] The robot's visual system is used to obtain an image of the device to be assembled. The visual system of the robot is a camera on the robot. The image of the device to be assembled represents a scene image of the area where the device to be assembled is located. The robot's visual system automatically performs grayscale, stretching, and edge detection preprocessing on the obtained image of the device to be assembled. The grayscale, stretching, and edge detection process is as follows:

[0014] S11: Calculating the maximum value of the RGB color components of each pixel in the collected image of the device to be assembled, and using the maximum value as the grayscale value of the pixel to obtain a grayscale image of the image of the device to be assembled;

[0015] S12: Stretch the grayscale value of each pixel in the grayscale image:

[0016]

[0017] in:

[0018] g(i, j) represents the grayscale value of pixel (i, j) in the grayscale image. Pixel (i, j) represents the pixel in the i-th row and j-th column of the grayscale image. The number of pixel rows in the grayscale image is M, and the number of pixel columns is N.

[0019] MAX g Indicates the maximum grayscale value in the grayscale image, MIN g Indicates the minimum grayscale value in the grayscale image;

[0020] g′(i, j) represents the grayscale value of pixel (i, j) after grayscale stretching;

[0021] S13: Construct a Gaussian filter template of 3×3 pixels to filter the grayscale image after grayscale stretching, wherein the constructed Gaussian filter template is G σ , σ represents the scale of the Gaussian filter template, which is set to 2. The Gaussian filter template is used to perform convolution processing on each pixel in the grayscale image after grayscale stretching to obtain the grayscale image after Gaussian filtering:

[0022] g″(i, j)=g′(i, j)*G σ

[0023] in:

[0024] g″(i, j) represents the Gaussian filtering result of pixel (i, j), and the Gaussian filtering results of all pixels constitute the grayscale image after Gaussian filtering;

[0025] In the embodiment of the present invention, the construction process of the Gaussian filter template is as follows:

[0026] Set the initial Gaussian filter template:

[0027]

[0028] in:

[0029] G σ (i′, j′) represents the element value of the element in the i′th row and j′th column of the initial Gaussian filter template with a scale of σ;

[0030] Normalize the element values ​​of each element in the initial Gaussian filter template so that the sum of all elements is 1, and obtain the Gaussian filter template G with a scale of σ σ ;

[0031] S14: Calculate the gradient value of each pixel in the grayscale image after Gaussian filtering:

[0032]

[0033] in:

[0034] grad(i, j) represents the gradient value of pixel (i, j) in the grayscale image after Gaussian filtering;

[0035] Mark pixels whose gradient values ​​are greater than a preset threshold as edge pixels;

[0036] In the embodiment of the present invention, if the pixel (i+1, j) does not exist in the grayscale image after Gaussian filtering, the Gaussian filtering result of the pixel is marked as 0.

[0037] Optionally, constructing an offline product registration model in step S2 includes:

[0038] Constructing an offline product registration model, wherein the offline product registration model takes a template registration image and an image of the device to be assembled as input and outputs a registration fit; wherein the template registration image is a pre-processed display image of the device to be assembled after successful assembly;

[0039] The offline product registration model includes a device identification module, a combination registration module, and a registration fit calculation module. The device identification module is used to identify the device from the device image. The combination registration module is used to randomly combine the device and the assembly position to obtain different combination results. The registration fit calculation module is used to calculate the registration fit of different combination results with the template registration image, and the largest combination result and the corresponding registration fit are selected for output.

[0040] The process of registering the image of the equipment to be assembled based on the offline registration model is as follows:

[0041] S21: Divide the image of the device to be assembled into several sub-images according to the image edges;

[0042] S22: Inputting the sub-image of the device to be assembled image and the template registration image into the device to be assembled recognition module, the device to be assembled recognition module uses two convolution layers to perform convolution processing on the sub-image and the template registration image respectively to obtain an initial convolution feature map of the sub-image and the template registration image, wherein the convolution kernel size of the convolution layer is 5×5 pixels and the stride is 2;

[0043] S23: Calculate the inner product result of the initial convolution feature map corresponding to the sub-image and the initial convolution feature map corresponding to the template registration image as the semantic similarity between the sub-image and the template registration image;

[0044] S24: Selecting the initial convolution feature map corresponding to the sub-image with the highest semantic similarity, and sliding the initial convolution feature map of the sub-image using a sliding window to extract a local feature map, wherein the size of the sliding window is the size of the device to be assembled in the template registration image; calculating the cosine similarity between the local feature map and the initial convolution feature map corresponding to the device to be assembled region in the template registration image, and selecting the local feature map with the highest cosine similarity as the device to be assembled feature map, and the device to be assembled image region corresponding to the local feature map is the identified device to be assembled;

[0045] S25: The combined registration module converts the pixel coordinates of the device to be assembled identified in the image of the device to be assembled into the world coordinate system to obtain a three-dimensional image representation of the device to be assembled:

[0046]

[0047] in:

[0048] d1 and d2 represent the lengths of the unit pixel in the image captured by the visual system in the horizontal and vertical directions respectively;

[0049] x0, y0 represent the number of pixels in the horizontal and vertical directions that are different between the center pixel coordinates of the image captured by the visual system and the pixel coordinates of the image origin;

[0050] f represents the focal length of the robot's vision system;

[0051] K1 and K2 represent the external parameters of the visual system, which are the position and rotation direction parameters of the visual system respectively;

[0052] Z represents the coordinate value of the pixel coordinate (x, y) of the device to be assembled on the Z axis in the camera coordinate system;

[0053] (X * , Y * , Z * ) represents the mapping result of the pixel coordinates (x, y) of the device to be assembled in the world coordinate system;

[0054] In an embodiment of the present invention, the method for obtaining the external parameters of the visual system is:

[0055] Capture a reference image with known 3D world coordinates, substitute the pixel coordinates of the reference image into the coordinate transformation formula, and select the extrinsic parameters that minimize the error between the transformed world coordinates and the known 3D world coordinates as the extrinsic parameters of the vision system;

[0056] S26: randomly rotating the device to be assembled represented by the three-dimensional image to obtain multiple rotation results, and randomly combining the edges of the rotation results with the assembly positions to obtain multiple combination results;

[0057] S27: The registration fit calculation module calculates the curvature and position coordinates of the combined edge curve in each combination result, and calculates the curvature and position coordinates of the edge curve of the successful assembly result in the template registration image, and obtains the registration fit of each combination result:

[0058]

[0059] in:

[0060] β s (m) represents the curvature of the sth edge curve in the mth combination, α s Indicates the curvature of the edge curve that is closest to the position coordinates of the successful assembly result in the template registration image and the s-th edge curve in the m-th combination;

[0061] R m It represents the registration accuracy of the mth combination.

[0062] Optionally, the step S3 of constructing a real-time product registration robust control model includes:

[0063] Constructing a robust control model for real-time product registration, where the robust control model takes the image of the device to be assembled as input and produces the corrected image of the device to be registered as output;

[0064] The control process based on the real-time product registration robust control model is as follows:

[0065] S31: Selecting the sub-image with the highest semantic similarity in the image of the device to be assembled as the image to be corrected;

[0066] S32: Construct an image correction code, wherein the image correction code is in the form of a sliding window, and the sliding window size is the number of pixels in each row of pixels in the image to be corrected. The constructed image correction code is:

[0067]

[0068] in:

[0069] h n (q) represents the correction code of the qth pixel in the nth row, Q n Represents the total number of pixels in the nth row of pixels;

[0070] S33: Correct each row of pixels in the image to be corrected based on the image correction code to obtain a corrected image. The correction result of the qth pixel in the nth row in the image to be corrected is:

[0071] g″′ n，q =ωg″ n，q +(1-ω)h n (q)

[0072] in:

[0073] g″ n，q represents the Gaussian filtering result of the qth pixel in the nth row of the image to be corrected, ω represents the correction coefficient, which is set to 0.82, g″′ n，q Indicates the correction result of the qth pixel in the nth row of the image to be corrected;

[0074] S34: Adding the corrected image to the image of the device to be assembled as the corrected image of the device to be assembled.

[0075] Optionally, in step S4, the image of the device to be assembled is input into an offline product registration model to obtain a registration fit, and if the registration fit is greater than a specified threshold, assembly can be performed, including:

[0076] The image of the device to be assembled is input into the offline product registration model to obtain the registration consistency of different registration combination results. If the maximum registration consistency is greater than the specified threshold, assembly can be performed, and the image of the device to be assembled that can be assembled is obtained, and the rotation angle corresponding to the maximum registration consistency is recorded.

[0077] Optionally, in step S4, inputting the image of the device to be assembled whose registration degree of coincidence is lower than a specified threshold into the real-time product registration robust control model to obtain a corrected image of the device to be assembled includes:

[0078] The image of the device to be assembled whose registration degree is lower than the specified threshold is input into the real-time product registration robust control model for correction to obtain the corrected image of the device to be assembled. The corrected image of the device to be assembled is then input into the offline product registration model to obtain the registration degrees of different registration combination results. If the maximum registration degree is greater than the specified threshold, assembly can be carried out to obtain the image of the device to be assembled, and the rotation angle corresponding to the maximum registration degree is recorded. Otherwise, correction is performed again.

[0079] Optionally, in step S5, based on the image of the device to be assembled that can be assembled, controlling the robot arm to perform assembly processing of the device to be assembled includes:

[0080] Based on the image of the device to be assembled and the rotation angle corresponding to the maximum registration fit, the rotation angle of the device to be assembled is obtained when the device to be assembled is assembled, the robot arm records the rotation angle, and grabs the device to be assembled;

[0081] The robot arm uses the vision system to obtain a map of the equipment to be assembled and the assembly location, uses an obstacle avoidance algorithm to generate the assembly path for the robot arm, controls the robot arm to move along the assembly path until it reaches the assembly location, and controls the robot arm to rotate the equipment to be assembled according to the recorded rotation angle, completing assembly control based on the robot arm's vision.

[0082] The assembly path generation process based on the obstacle avoidance algorithm is as follows:

[0083] S51: Initialize several assembly paths, where the kth assembly path is in the form of:

[0084] L k =[L k (1), L k (2), ..., L k (u), ...L k (U)]

[0085] L k (u)=[θ k，u (1), θ k，u (2), ..., θ k，u (v)] T

[0086] in:

[0087] L k (u) represents the u-th path node position in the k-th assembly path, and U represents the total number of path nodes. In the embodiment of the present invention, the first path node position is the initial position of the robot arm, and the last path node position is the assembly position;

[0088] θ k，u (v) represents the angle of the vth joint node in the robot arm when the robot arm is at the uth path node position in the kth assembly path, where v represents the total number of joint nodes in the robot arm;

[0089] S52: Initialize the obstacle avoidance detection vector of the k-th assembly path with dimension U: [0, 0, ..., 0];

[0090] S53: Construct the virtual force function between the connecting rods and obstacles between adjacent joint nodes in the robot arm:

[0091]

[0092] in:

[0093] F k，u (a, a+1) represents the virtual force between the link and the obstacle between the a-th joint node and the a+1-th joint node at the u-th path node position in the k-th assembly path, d(r a，a+1 , rnext ) represents the connecting rod r a，a+1 The distance to the nearest obstacle next, δ represents a very small positive number, r next Indicates the radius of the nearest obstacle next;

[0094] If there is F k，u If (a, a+1) is greater than the force threshold, it indicates that there is a collision at the u-th path node in the k-th assembly path, and the u-th value in the obstacle avoidance detection vector is set to 1; in this embodiment of the present invention, a, a+1∈[1, v];

[0095] S54: traverse the obstacle avoidance detection vectors of the k-th assembly path, and for the path nodes with a value of 1, randomly select path nodes of the corresponding order from the remaining assembly paths to replace them, until the obstacle avoidance detection vectors in each assembly path are all 0;

[0096] S55: Calculate the moving time of each assembly path, select the assembly path with the shortest moving time as output, and control the robot arm to move along the assembly path to reach the assembly position.

[0097] In order to solve the above problem, the present invention provides an electronic device, comprising:

[0098] a memory storing at least one instruction;

[0099] Communication interfaces to enable electronic equipment to communicate; and

[0100] The processor executes the instructions stored in the memory to implement the above-mentioned vision migration robot assembly control method.

[0101] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned visual migration robot assembly control method.

[0102] Compared with the existing technology, the present invention proposes a robot arm assembly control method based on vision transfer, which has the following advantages:

[0103] First, this scheme proposes a method for determining an assembly scheme and constructs an offline product registration model, wherein the offline product registration model takes a template registration image and an image of a device to be assembled as input and takes a registration degree of fit as output; wherein the template registration image is a pre-processed display image of the device to be assembled after successful assembly; the offline product registration model includes a device to be assembled recognition module, a combined registration module and a registration degree of fit calculation module, wherein the device to be assembled recognition module is used to identify the device to be assembled from the image of the device to be assembled, the combined registration module is used to randomly combine the device to be assembled with the assembly position to obtain different combination results, and the registration degree of fit calculation module is used to calculate the registration degree of fit between different combination results and the template registration image, and the largest combination result and the corresponding registration degree of fit are selected for output. This scheme divides the image of the device to be assembled into several sub-images based on the image edges divided after preprocessing, and calculates the convolution feature maps corresponding to the sub-images and the template registration image to obtain the semantic similarity between the two. The sub-image with the largest semantic similarity is selected and several local feature maps are extracted. Based on the cosine similarity between the local feature maps and the convolution feature maps of the device to be assembled in the template registration image, the position area of ​​the device to be assembled in the device to be assembled image is determined. The device to be assembled is converted into a three-dimensional image representation, and the represented device to be assembled and the assembly position are randomly rotated and combined to obtain different combination results. The registration consistency of different combination results with the template registration image is calculated, and the largest combination result and the corresponding registration consistency are selected and output. If the registration consistency is greater than a specified threshold, the robot arm is instructed to calculate the rotation angle in the combination result and control the robot arm to rotate and assemble according to the recorded rotation angle after reaching the assembly position. At the same time, a real-time correction scheme for the image of the device to be assembled is proposed. By using the correction code to correct the image of the device to be assembled, a larger correction is performed on the central area of ​​the image, and the central area is more likely to contain the device to be assembled, thereby improving the recognition efficiency of the device to be assembled.

[0104] At the same time, this solution proposes an assembly path determination algorithm. The robot arm uses the vision system to obtain a map of the equipment to be assembled and the assembly location. The obstacle avoidance algorithm is used to generate the assembly path of the robot arm. The robot arm is controlled to move along the assembly path to reach the assembly location. The robot arm is also controlled to rotate the equipment to be assembled according to the recorded rotation angle to complete the assembly control based on the robot arm vision. The assembly path generation process based on the obstacle avoidance algorithm is as follows: Initialize several assembly paths, where the kth assembly path is in the form of:

[0105] L k =[L k (1), L k (2), ..., L k (u), ...L k (U)]

[0106] Lk (u)=[θ k，u (1), θ k，u (2), ..., θ k，u (v)] T

[0107] Where: L k (u) represents the position of the u-th path node in the k-th assembly path, and U represents the total number of path nodes; θ k，u (v) represents the angle of the vth joint node in the robot arm when the robot arm is at the uth path node position in the kth assembly path, where v represents the total number of joint nodes in the robot arm. Initialize the obstacle avoidance detection vector of the kth assembly path with dimension U: [0, 0, ..., 0]. Construct the virtual force function between the connecting rods and obstacles between adjacent joint nodes in the robot arm:

[0108]

[0109] Among them: F k，u (a, a+1) represents the virtual force between the link and the obstacle between the a-th joint node and the a+1-th joint node at the u-th path node position in the k-th assembly path, d(r a，a+1 , r next ) represents the connecting rod r a，a+1 The distance to the nearest obstacle next, δ represents a very small positive number, r next Indicates the radius of the nearest obstacle next; if there is F k，u If (a, a+1) is greater than the force threshold, it indicates that there is a collision at the u-th path node in the k-th assembly path, and the u-th value in the obstacle avoidance detection vector is set to 1. The obstacle avoidance detection vector of the k-th assembly path is traversed, and for the path nodes with a value of 1, the path nodes of the corresponding order are randomly selected from the remaining assembly paths to replace them, until the obstacle avoidance detection vectors in each assembly path are all 0. The movement time of each assembly path is calculated, and the assembly path with the shortest movement time is selected as the output. The robot is controlled to move along the assembly path to reach the assembly position. This scheme uses the connecting rods between adjacent joints of the robot as the reference unit of the obstacle avoidance algorithm. By calculating the virtual force between the connecting rods and obstacles between adjacent joint nodes, it is determined whether there is a collision between the connecting rods and obstacles at each path node in the assembly path. The path nodes with collisions are corrected to obtain an assembly path that avoids collisions of the robot arm. The robot arm is controlled to move along the assembly path to reach the assembly position, and the robot arm is controlled to rotate the assembly equipment according to the recorded rotation angle. The rotation angle based on the initial position vision is transferred to the final assembly position, realizing assembly control based on robot arm vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1A schematic flow chart of a robotic arm assembly control method using vision transfer provided by one embodiment of the present invention;

[0111] Figure 2 A schematic structural diagram of an electronic device for implementing a robotic arm assembly control method for implementing visual migration provided by an embodiment of the present invention.

[0112] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0113] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0114] The embodiment of the present application provides a method for controlling the assembly of a robotic arm with visual migration. The execution subject of the method for controlling the assembly of a robotic arm with visual migration includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for controlling the assembly of a robotic arm with visual migration can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0115] Example 1:

[0116] S1: Utilize the vision system of the robotic arm to obtain an image of the device to be assembled, and automatically pre-process the obtained image of the device to be assembled.

[0117] The step S1 of obtaining an image of the device to be assembled includes:

[0118] The robot's visual system is used to obtain an image of the device to be assembled. The visual system of the robot is a camera on the robot. The image of the device to be assembled represents a scene image of the area where the device to be assembled is located. The robot's visual system automatically performs grayscale, stretching, and edge detection preprocessing on the obtained image of the device to be assembled. The grayscale, stretching, and edge detection process is as follows:

[0119] S11: Calculating the maximum value of the RGB color components of each pixel in the collected image of the device to be assembled, and using the maximum value as the grayscale value of the pixel to obtain a grayscale image of the image of the device to be assembled;

[0120] S12: Stretch the grayscale value of each pixel in the grayscale image:

[0121]

[0122] in:

[0123] g(i, j) represents the grayscale value of pixel (i, j) in the grayscale image. Pixel (i, j) represents the pixel in the i-th row and j-th column of the grayscale image. The number of pixel rows in the grayscale image is M, and the number of pixel columns is N.

[0124] MAX g Indicates the maximum grayscale value in the grayscale image, MIN g Indicates the minimum grayscale value in the grayscale image;

[0125] g′(i, j) represents the grayscale value of pixel (i, j) after grayscale stretching;

[0126] S13: Construct a Gaussian filter template of 3×3 pixels to filter the grayscale image after grayscale stretching, wherein the constructed Gaussian filter template is G σ , σ represents the scale of the Gaussian filter template, which is set to 2. The Gaussian filter template is used to perform convolution processing on each pixel in the grayscale image after grayscale stretching to obtain the grayscale image after Gaussian filtering:

[0127] g″(i, j)=g′(i, j)*G σ

[0128] in:

[0129] g″(i, j) represents the Gaussian filtering result of pixel (i, j), and the Gaussian filtering results of all pixels constitute the grayscale image after Gaussian filtering;

[0130] In the embodiment of the present invention, the construction process of the Gaussian filter template is as follows:

[0131] Set the initial Gaussian filter template:

[0132]

[0133] in:

[0134] G σ (i′, j′) represents the element value of the element in the i′th row and j′th column of the initial Gaussian filter template with a scale of σ;

[0135] Normalize the element values ​​of each element in the initial Gaussian filter template so that the sum of all elements is 1, and obtain the Gaussian filter template G with a scale of σ σ ;

[0136] S14: Calculate the gradient value of each pixel in the grayscale image after Gaussian filtering:

[0137]

[0138] in:

[0139] grad(i, j) represents the gradient value of pixel (i, j) in the grayscale image after Gaussian filtering;

[0140] Mark pixels whose gradient values ​​are greater than a preset threshold as edge pixels;

[0141] In the embodiment of the present invention, if the pixel (i+1, j) does not exist in the grayscale image after Gaussian filtering, the Gaussian filtering result of the pixel is marked as 0.

[0142] S2: Build an offline product registration model, which takes the template registration image and the image of the device to be assembled as input and uses the registration fit as output.

[0143] The offline product registration model is constructed in step S2, including:

[0144] Constructing an offline product registration model, wherein the offline product registration model takes a template registration image and an image of the device to be assembled as input and outputs a registration fit; wherein the template registration image is a pre-processed display image of the device to be assembled after successful assembly;

[0145] The offline product registration model includes a device identification module, a combination registration module, and a registration fit calculation module. The device identification module is used to identify the device from the device image. The combination registration module is used to randomly combine the device and the assembly position to obtain different combination results. The registration fit calculation module is used to calculate the registration fit of different combination results with the template registration image, and the largest combination result and the corresponding registration fit are selected for output.

[0146] The process of registering the image of the equipment to be assembled based on the offline registration model is as follows:

[0147] S21: Divide the image of the device to be assembled into several sub-images according to the image edges;

[0148] S22: Inputting the sub-image of the device to be assembled image and the template registration image into the device to be assembled recognition module, the device to be assembled recognition module uses two convolution layers to perform convolution processing on the sub-image and the template registration image respectively to obtain an initial convolution feature map of the sub-image and the template registration image, wherein the convolution kernel size of the convolution layer is 5×5 pixels and the stride is 2;

[0149] S23: Calculate the inner product result of the initial convolution feature map corresponding to the sub-image and the initial convolution feature map corresponding to the template registration image as the semantic similarity between the sub-image and the template registration image;

[0150] S24: Selecting the initial convolution feature map corresponding to the sub-image with the highest semantic similarity, and sliding the initial convolution feature map of the sub-image using a sliding window to extract a local feature map, wherein the size of the sliding window is the size of the device to be assembled in the template registration image; calculating the cosine similarity between the local feature map and the initial convolution feature map corresponding to the device to be assembled region in the template registration image, and selecting the local feature map with the highest cosine similarity as the device to be assembled feature map, and the device to be assembled image region corresponding to the local feature map is the identified device to be assembled;

[0151] S25: The combined registration module converts the pixel coordinates of the device to be assembled identified in the image of the device to be assembled into the world coordinate system to obtain a three-dimensional image representation of the device to be assembled:

[0152]

[0153] in:

[0154] d1 and d2 represent the lengths of the unit pixel in the image captured by the visual system in the horizontal and vertical directions respectively;

[0155] x0, y0 represent the number of pixels in the horizontal and vertical directions that are different between the center pixel coordinates of the image captured by the visual system and the pixel coordinates of the image origin;

[0156] f represents the focal length of the robot's vision system;

[0157] K1 and K2 represent the external parameters of the visual system, which are the position and rotation direction parameters of the visual system respectively;

[0158] Z represents the coordinate value of the pixel coordinate (x, y) of the device to be assembled on the Z axis in the camera coordinate system;

[0159] (X * , Y * , Z * ) represents the mapping result of the pixel coordinates (x, y) of the device to be assembled in the world coordinate system;

[0160] S26: randomly rotating the device to be assembled represented by the three-dimensional image to obtain multiple rotation results, and randomly combining the edges of the rotation results with the assembly positions to obtain multiple combination results;

[0161] S27: The registration fit calculation module calculates the curvature and position coordinates of the combined edge curve in each combination result, and calculates the curvature and position coordinates of the edge curve of the successful assembly result in the template registration image, and obtains the registration fit of each combination result:

[0162]

[0163] in:

[0164] β s (m) represents the curvature of the sth edge curve in the mth combination, α s Indicates the curvature of the edge curve that is closest to the position coordinates of the successful assembly result in the template registration image and the s-th edge curve in the m-th combination;

[0165] R m It represents the registration accuracy of the mth combination.

[0166] S3: Construct a real-time product registration robust control model, which takes the image of the device to be assembled as input and the corrected image of the device to be registered as output.

[0167] The S3 step constructs a real-time product registration robust control model, including:

[0168] Constructing a robust control model for real-time product registration, where the robust control model takes the image of the device to be assembled as input and produces the corrected image of the device to be registered as output;

[0169] The control process based on the real-time product registration robust control model is as follows:

[0170] S31: Selecting the sub-image with the highest semantic similarity in the image of the device to be assembled as the image to be corrected;

[0171] S32: Construct an image correction code, wherein the image correction code is in the form of a sliding window, and the sliding window size is the number of pixels in each row of pixels in the image to be corrected. The constructed image correction code is:

[0172]

[0173] in:

[0174] h n (q) represents the correction code of the qth pixel in the nth row, Q n Represents the total number of pixels in the nth row of pixels;

[0175] S33: Correct each row of pixels in the image to be corrected based on the image correction code to obtain a corrected image. The correction result of the qth pixel in the nth row in the image to be corrected is:

[0176] g″′ n，q =ωg″ n，q +(1-ω)h n (q)

[0177] in:

[0178] g″ n，qrepresents the Gaussian filtering result of the qth pixel in the nth row of the image to be corrected, ω represents the correction coefficient, which is set to 0.82, g″′ n，q Indicates the correction result of the qth pixel in the nth row of the image to be corrected;

[0179] S34: Adding the corrected image to the image of the device to be assembled as the corrected image of the device to be assembled.

[0180] S4: The preprocessed image of the device to be assembled is input into the offline product registration model to obtain the registration fit. If the registration fit is greater than the specified threshold, assembly can be performed. Otherwise, the preprocessed image of the device to be assembled is input into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, and the corrected image of the device to be assembled is input into the offline product registration model.

[0181] In step S4, the image of the device to be assembled is input into the offline product registration model to obtain the registration fit. If the registration fit is greater than a specified threshold, assembly can be performed, including:

[0182] The image of the device to be assembled is input into the offline product registration model to obtain the registration consistency of different registration combination results. If the maximum registration consistency is greater than the specified threshold, assembly can be performed, and the image of the device to be assembled that can be assembled is obtained, and the rotation angle corresponding to the maximum registration consistency is recorded.

[0183] In step S4, the image of the device to be assembled whose registration degree of coincidence is lower than a specified threshold is input into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, including:

[0184] The image of the device to be assembled whose registration degree is lower than the specified threshold is input into the real-time product registration robust control model for correction to obtain the corrected image of the device to be assembled. The corrected image of the device to be assembled is then input into the offline product registration model to obtain the registration degrees of different registration combination results. If the maximum registration degree is greater than the specified threshold, assembly can be carried out to obtain the image of the device to be assembled, and the rotation angle corresponding to the maximum registration degree is recorded. Otherwise, correction is performed again.

[0185] S5: According to the image of the device to be assembled that can be assembled, the robot arm is controlled to assemble the device to be assembled.

[0186] In step S5, based on the image of the device to be assembled, the robot arm is controlled to assemble the device to be assembled, including:

[0187] Based on the image of the device to be assembled and the rotation angle corresponding to the maximum registration fit, the rotation angle of the device to be assembled is obtained when the device to be assembled is assembled, the robot arm records the rotation angle, and grabs the device to be assembled;

[0188] The robot arm uses the vision system to obtain a map of the equipment to be assembled and the assembly location, uses an obstacle avoidance algorithm to generate the assembly path for the robot arm, controls the robot arm to move along the assembly path until it reaches the assembly location, and controls the robot arm to rotate the equipment to be assembled according to the recorded rotation angle, completing assembly control based on the robot arm's vision.

[0189] The assembly path generation process based on the obstacle avoidance algorithm is as follows:

[0190] S51: Initialize several assembly paths, where the kth assembly path is in the form of:

[0191] L k =[L k (1), L k (2), ..., L k (u), ...L k (U)]

[0192] L k (u)=[θ k，u (1), θ k，u (2), ..., θ k，u (v)] T

[0193] in:

[0194] L k (u) represents the u-th path node position in the k-th assembly path, and U represents the total number of path nodes. In the embodiment of the present invention, the first path node position is the initial position of the robot arm, and the last path node position is the assembly position;

[0195] θ k，u (v) represents the angle of the vth joint node in the robot arm when the robot arm is at the uth path node position in the kth assembly path, where v represents the total number of joint nodes in the robot arm;

[0196] S52: Initialize the obstacle avoidance detection vector of the k-th assembly path with dimension U: [0, 0, ..., 0];

[0197] S53: Construct the virtual force function between the connecting rods and obstacles between adjacent joint nodes in the robot arm:

[0198]

[0199] in:

[0200] F k，u (a, a+1) represents the virtual force between the link and the obstacle between the a-th joint node and the a+1-th joint node at the u-th path node position in the k-th assembly path, d(r a，a+1 , r next ) represents the connecting rod r a，a+1 The distance to the nearest obstacle next, δ represents a very small positive number, r next Indicates the radius of the nearest obstacle next;

[0201] If there is F k，u If (a, a+1) is greater than the force threshold, it means that there is a collision at the u-th path node in the k-th assembly path, and the u-th value in the obstacle avoidance detection vector is set to 1;

[0202] S54: traverse the obstacle avoidance detection vectors of the k-th assembly path, and for the path nodes with a value of 1, randomly select path nodes of the corresponding order from the remaining assembly paths to replace them, until the obstacle avoidance detection vectors in each assembly path are all 0;

[0203] S55: Calculate the moving time of each assembly path, select the assembly path with the shortest moving time as output, and control the robot arm to move along the assembly path to reach the assembly position.

[0204] Example 2:

[0205] like Figure 2 , which is a structural diagram of an electronic device for a robotic arm assembly control method for realizing visual migration provided by an embodiment of the present invention.

[0206] The electronic device 1 may include a processor 10 , a memory 11 , a communication interface 13 and a bus, and may further include a computer program stored in the memory 11 and executable on the processor 10 , such as a program 12 .

[0207] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the program 12, etc., but can also be used to temporarily store data that has been output or is to be output.

[0208] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and circuits. It executes or runs programs or modules stored in the memory 11 (such as the program 12 for implementing robotic arm assembly control) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0209] The communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices, and to realize connection and communication between internal components of the electronic devices.

[0210] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11 and at least one processor 10, etc.

[0211] Figure 2 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 2 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0212] For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for powering the various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device 1 may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0213] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0214] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0215] The program 12 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When executed in the processor 10, it can achieve the following:

[0216] Utilize the vision system of the robotic arm to obtain images of the equipment to be assembled, and automatically pre-process the obtained images of the equipment to be assembled;

[0217] Build an offline product registration model;

[0218] Build a robust control model for real-time product registration;

[0219] The pre-processed image of the device to be assembled is input into the offline product registration model to obtain the registration consistency. If the registration consistency is greater than a specified threshold, assembly can be performed. Otherwise, the pre-processed image of the device to be assembled is input into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, and the corrected image of the device to be assembled is input into the offline product registration model.

[0220] According to the image of the equipment to be assembled, the robot arm is controlled to assemble the equipment to be assembled.

[0221] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0222] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0223] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0224] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vision-based robotic arm assembly control method, characterized in that: The method comprises: S1: Using the vision system of the robotic arm to obtain an image of the device to be assembled, and automatically preprocessing the obtained image of the device to be assembled; S2: Build an offline product registration model, where the model takes the template registration image and the image of the device to be assembled as input and outputs the registration consistency; S3: Build a real-time robust control model for product registration, where the model takes the image of the device to be assembled as input and outputs the corrected image of the device to be assembled; S4: Input the preprocessed image of the device to be assembled into the offline product registration model to obtain the registration consistency. If the registration consistency is greater than a specified threshold, assembly can be performed. Otherwise, input the preprocessed image of the device to be assembled into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, and input the corrected image of the device to be assembled into the offline product registration model. S5: According to the image of the device to be assembled that can be assembled, the robot arm is controlled to assemble the device to be assembled.

2. The method for controlling assembly of a robotic arm using vision transfer according to claim 1, wherein: The step S1 of obtaining an image of the device to be assembled includes: The robot's visual system is used to obtain an image of the device to be assembled. The visual system of the robot is a camera on the robot. The image of the device to be assembled represents a scene image of the area where the device to be assembled is located. The robot's visual system automatically performs grayscale, stretching, and edge detection preprocessing on the obtained image of the device to be assembled. The grayscale, stretching, and edge detection process is as follows: S11: Calculating the maximum value of the RGB color component of each pixel in the collected image of the device to be assembled, and using the maximum value as the grayscale value of the pixel point of the calculated image to obtain a grayscale image of the image of the device to be assembled; S12: Stretch the grayscale value of each pixel in the grayscale image: ； in: Represents pixels in a grayscale image Grayscale value, pixel Represents the pixel in the i-th row and j-th column of the grayscale image. The number of pixel rows in the grayscale image is M, and the number of pixel columns is N. Represents the maximum grayscale value in the grayscale image, Indicates the minimum grayscale value in the grayscale image; Indicates the pixel after grayscale stretching Gray value of S13: Build The pixel-sized Gaussian filter template is used to filter the grayscale image after grayscale stretching, where the constructed Gaussian filter template is , Represents the scale of the Gaussian filter template, set it to 2, and use the Gaussian filter template to perform convolution processing on each pixel in the grayscale image after grayscale stretching to obtain the grayscale image after Gaussian filtering: ； in: Represents pixels The Gaussian filtering results of all pixels are used to form a grayscale image after Gaussian filtering; S14: Calculate the gradient value of each pixel in the grayscale image after Gaussian filtering: ； in: Represents the pixel in the grayscale image after Gaussian filtering The gradient value of Pixels with gradient values ​​greater than a preset threshold are marked as edge pixels.

3. The method for controlling assembly of a robotic arm using vision transfer according to claim 1, wherein: The offline product registration model is constructed in step S2, including: Constructing an offline product registration model, wherein the offline product registration model takes a template registration image and an image of the device to be assembled as input and outputs a registration fit; wherein the template registration image is a pre-processed display image of the device to be assembled after successful assembly; The offline product registration model includes a device identification module, a combination registration module, and a registration fit calculation module. The device identification module is used to identify the device from the device image. The combination registration module is used to randomly combine the device and the assembly position to obtain different combination results. The registration fit calculation module is used to calculate the registration fit of different combination results with the template registration image, and the largest combination result and the corresponding registration fit are selected for output. The process of registering the image of the equipment to be assembled based on the offline registration model is as follows: S21: Divide the image of the device to be assembled into several sub-images according to the image edges; S22: Input the sub-image of the device to be assembled image and the template registration image into the device to be assembled recognition module. The device to be assembled recognition module uses two convolution layers to perform convolution processing on the sub-image and the template registration image respectively to obtain the initial convolution feature map of the sub-image and the template registration image, where the convolution kernel size of the convolution layer is Pixels, step size is 2; S23: Calculate the inner product result of the initial convolution feature map corresponding to the sub-image and the initial convolution feature map corresponding to the template registration image as the semantic similarity between the sub-image and the template registration image; S24: Selecting the initial convolution feature map corresponding to the sub-image with the highest semantic similarity, and sliding the initial convolution feature map of the sub-image using a sliding window to extract a local feature map, wherein the size of the sliding window is the size of the device to be assembled in the template registration image; calculating the cosine similarity between the local feature map and the initial convolution feature map corresponding to the device to be assembled region in the template registration image, and selecting the local feature map with the highest cosine similarity as the device to be assembled feature map, and the device to be assembled image region corresponding to the local feature map is the identified device to be assembled; S25: The combined registration module converts the pixel coordinates of the device to be assembled identified in the image of the device to be assembled into the world coordinate system to obtain a three-dimensional image representation of the device to be assembled: ； in: Respectively represent the lengths of the unit pixel in the horizontal and vertical directions in the image captured by the visual system; Respectively represent the number of pixels in the horizontal and vertical directions that are different between the center pixel coordinates of the image captured by the visual system and the pixel coordinates of the image origin; represents the focal length of the robot's vision system; Represents the external parameters of the visual system, which are the position and rotation direction parameters of the visual system; Z represents the pixel coordinate of the device to be assembled The coordinate value of the Z axis in the camera coordinate system; Indicates the pixel coordinates of the device to be assembled Mapping results in the world coordinate system; S26: randomly rotating the device to be assembled represented by the three-dimensional image to obtain multiple rotation results, and randomly combining the edges of the rotation results with the assembly positions to obtain multiple combination results; S27: The registration fit calculation module calculates the curvature and position coordinates of the combined edge curve in each combination result, and calculates the curvature and position coordinates of the edge curve of the successful assembly result in the template registration image, and obtains the registration fit of each combination result: ； in: represents the curvature of the sth edge curve in the mth combination, Indicates the curvature of the edge curve that is closest to the position coordinates of the successful assembly result in the template registration image and the s-th edge curve in the m-th combination; It represents the registration accuracy of the mth combination.

4. The method for controlling assembly of a robotic arm using vision transfer according to claim 3, wherein: The S3 step constructs a real-time product registration robust control model, including: Constructing a robust control model for real-time product registration, where the robust control model takes the image of the device to be assembled as input and produces a corrected image of the device to be assembled as output; The control process based on the real-time product registration robust control model is as follows: S31: Selecting the sub-image with the highest semantic similarity in the image of the device to be assembled as the image to be corrected; S32: Construct an image correction code, wherein the image correction code is in the form of a sliding window, and the sliding window size is the number of pixels in each row of pixels in the image to be corrected. The constructed image correction code is: ； in: represents the correction code of the qth pixel in the nth row, Represents the total number of pixels in the nth row of pixels; S33: Correct each row of pixels in the image to be corrected based on the image correction code to obtain a corrected image. The correction result of the qth pixel in the nth row in the image to be corrected is: ； in: Represents the Gaussian filtering result of the qth pixel in the nth row of the image to be corrected, Indicates the correction factor, set it to 0.82, Indicates the correction result of the qth pixel in the nth row of the image to be corrected; S34: Adding the corrected image to the image of the device to be assembled as the corrected image of the device to be assembled.

5. The method for controlling assembly of a robotic arm using vision transfer according to claim 1, wherein: In step S4, the image of the device to be assembled is input into the offline product registration model to obtain the registration fit. If the registration fit is greater than a specified threshold, assembly can be performed, including: The image of the device to be assembled is input into the offline product registration model to obtain the registration consistency of different registration combination results. If the maximum registration consistency is greater than the specified threshold, assembly can be performed, and the image of the device to be assembled that can be assembled is obtained, and the rotation angle corresponding to the maximum registration consistency is recorded.

6. The method for controlling assembly of a robotic arm using vision transfer according to claim 5, wherein: In step S4, the image of the device to be assembled whose registration degree of coincidence is lower than a specified threshold is input into the real-time product registration robust control model to obtain a corrected image of the device to be assembled, including: The image of the device to be assembled whose registration degree is lower than the specified threshold is input into the real-time product registration robust control model for correction to obtain the corrected image of the device to be assembled. The corrected image of the device to be assembled is then input into the offline product registration model to obtain the registration degrees of different registration combination results. If the maximum registration degree is greater than the specified threshold, assembly can be carried out to obtain the image of the device to be assembled, and the rotation angle corresponding to the maximum registration degree is recorded. Otherwise, correction is performed again.

7. The method for controlling assembly of a robotic arm using vision transfer according to claim 6, wherein: In step S5, based on the image of the device to be assembled, the robot arm is controlled to assemble the device to be assembled, including: Based on the image of the device to be assembled and the rotation angle corresponding to the maximum registration fit, the rotation angle of the device to be assembled is obtained when the device to be assembled is assembled, the robot arm records the rotation angle, and grabs the device to be assembled; The robot arm uses the vision system to obtain a map of the equipment to be assembled and the assembly location, uses an obstacle avoidance algorithm to generate the assembly path for the robot arm, controls the robot arm to move along the assembly path until it reaches the assembly location, and controls the robot arm to rotate the equipment to be assembled according to the recorded rotation angle, completing assembly control based on the robot arm's vision. The assembly path generation process based on the obstacle avoidance algorithm is as follows: S51: Initialize several assembly paths, where the kth assembly path is in the form of: ； ； in: represents the position of the u-th path node in the k-th assembly path, and U represents the total number of path nodes; represents the angle of the vth joint node in the robot arm when the robot arm is at the uth path node position in the kth assembly path, where v represents the total number of joint nodes in the robot arm; S52: Initialize the obstacle avoidance detection vector of the k-th assembly path with dimension U: ; S53: Construct the virtual force function between the connecting rods and obstacles between adjacent joint nodes in the robot arm: ； in: The position of the u-th path node in the k-th assembly path is represented by The joint node and the The virtual force conditions of the connecting rods and obstacles between the joint nodes, Indicates connecting rod The distance to the nearest obstacle next, Represents a very small positive number, Indicates the radius of the nearest obstacle next; If exists If it is greater than the force threshold, it means that there is a collision at the u-th path node in the k-th assembly path, and the u-th value in the obstacle avoidance detection vector is set to 1; S54: traverse the obstacle avoidance detection vectors of the k-th assembly path, and for the path nodes with a value of 1, randomly select path nodes of the corresponding order from the remaining assembly paths to replace them, until the obstacle avoidance detection vectors in each assembly path are all 0; S55: Calculate the moving time of each assembly path, select the assembly path with the shortest moving time as output, and control the robot arm to move along the assembly path to reach the assembly position.

Images