Memory assembly method and system, electronic equipment, storage medium and product

Abstract

The invention discloses a memory assembly method and system, electronic equipment, a storage medium and a product, and relates to the field of server manufacturing, and the method comprises the steps: under the condition that a target carrier where a to-be-installed memory bank is located is recognized, grabbing the to-be-installed memory bank in the target carrier through a mechanical arm, obtaining a memory bank grabbing image, and storing the memory bank grabbing image in a storage medium; constructing an image pyramid of the memory bank capture image, performing feature extraction operation on the image pyramid to obtain feature point information of the to-be-installed memory bank so as to determine the actual posture of the memory bank, and adjusting the tail end posture of the mechanical arm according to the actual posture of the to-be-installed memory bank and a preset memory bank posture. According to the technical scheme, the mechanical arm can dynamically correct the posture of the to-be-installed memory bank, the problem that in the related technology, due to posture deviation possibly occurring after the memory bank is grabbed, the precision of memory assembly is low is solved, the precision and reliability of memory assembly are improved, the assembly failure rate is reduced to a certain degree, and the production efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of server manufacturing, and in particular to memory assembly methods, systems, electronic devices, storage media, and products. Background Art

[0002] In the memory assembly process of the related art, the positioning and grabbing of the memory bar often rely on a fixed mechanical structure and viewing angle. This limits the flexibility and compatibility of the assembly, especially when faced with the diverse layouts of different server motherboards and the slight deflection of the memory bar, and the adaptability to the assembly environment is limited, especially after the memory bar is grabbed, the deflection may be caused by factors such as the slight vibration of the mechanical arm, resulting in the problem of low precision of memory assembly in the related art. Summary of the invention

[0003] The present application provides a memory assembly method, system, electronic device, storage medium and product to at least solve the problem of low precision of memory assembly in the related art.

[0004] The present application provides a memory assembly method, which is applied to a memory assembly system. The memory assembly system includes a robot and a first visual sensor. The robot is configured with a mechanical arm. The method includes: when a target carrier where a memory bar to be installed is located is identified, the mechanical arm is used to grab the memory bar to be installed in the target carrier, and a memory bar grab image is obtained by the first visual sensor, wherein the memory bar grab image is an image when the mechanical arm grabs the memory bar to be installed; an image pyramid of the memory bar grab image is constructed, wherein the image pyramid includes a group of down-sampled images obtained by performing a group of down-sampling operations of different granularities on the memory bar grab image; a feature extraction operation is performed on the image pyramid to obtain feature point information of the memory bar to be installed; an actual posture of the memory bar to be installed is determined according to the feature point information of the memory bar to be installed; based on the actual posture and a preset memory bar posture, the end posture of the mechanical arm is adjusted to adjust the posture of the memory bar to be installed; the mechanical arm after the end posture adjustment is performed, and the memory bar to be installed is installed into a designated memory slot corresponding to the memory bar to be installed.

[0005] The present application also provides a memory assembly system, including a control device, a robot and a first visual sensor, wherein the robot is configured with a robotic arm; wherein the control device is used to control the robotic arm to grab the memory bar to be installed in the target carrier when identifying the target carrier where the memory bar to be installed is located, and control the first visual sensor to obtain a captured image of the memory bar, wherein the captured image of the memory bar is an image when the robotic arm grabs the memory bar to be installed; construct an image pyramid of the captured image of the memory bar, wherein the image pyramid includes a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the captured image of the memory bar; perform a feature extraction operation on the image pyramid to obtain the memory bar to be installed. feature point information of a memory stick; determining the actual posture of the memory stick to be installed according to the feature point information of the memory stick to be installed; adjusting the end posture of the robotic arm based on the actual posture and a preset posture of the memory stick to be installed, so as to adjust the posture of the memory stick to be installed; controlling the robotic arm after the end posture adjustment to install the memory stick to be installed into the designated memory slot corresponding to the memory stick to be installed; the robotic arm is used to, in response to the control of the control device, grab the memory stick to be installed in the target carrier, adjust the end posture of the robotic arm, and install the memory stick to be installed into the designated memory slot; the first visual sensor is used to, in response to the control of the control device, obtain a grabbed image of the memory stick.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned memory assembly methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned memory assembly methods are implemented.

[0008] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned memory assembly methods when executed by a processor.

[0009] Through the present application, when the target carrier where the memory stick to be installed is located is identified, the memory stick to be installed in the target carrier is grabbed by a robotic arm, and the captured image of the memory stick is obtained by a first visual sensor. By constructing an image pyramid of the captured image of the memory stick, feature detection is allowed at different image resolutions, thereby improving the accuracy of the feature extraction operation, thereby obtaining more accurate feature point information of the memory stick to be installed, so as to determine the actual posture of the memory stick, and adjust the end posture of the robotic arm according to the actual posture of the memory stick to be installed and the preset posture of the memory stick, so that the robotic arm can dynamically correct the posture of the memory stick to be installed, thereby solving the problem of low precision of memory assembly caused by the posture deviation that may occur in the memory stick after grabbing in the related technology, improving the accuracy and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A flowchart of an optional memory assembly method provided according to an embodiment of the present application.

[0012] Figure 2 A schematic diagram of an optional memory assembly method provided according to an embodiment of the present application.

[0013] Figure 3 A schematic diagram of characteristic points of an optional memory stick provided according to an embodiment of the present application.

[0014] Figure 4 A schematic diagram of an optional memory assembly provided according to an embodiment of the present application.

[0015] Figure 5 A structural block diagram of an optional memory assembly system provided according to an embodiment of the present application.

[0016] Figure 6 A structural block diagram of another optional memory assembly system provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0019] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0020] In the related art, during the memory assembly process, the positioning and grasping of the memory bar often rely on a fixed mechanical structure and viewing angle. This limits the flexibility and compatibility of the assembly, especially when facing the diverse layouts of different server motherboards and the slight deflection of the memory bar, the adaptability to the assembly environment is limited, especially after the memory bar is grasped, the deflection may be caused by factors such as the slight vibration of the mechanical arm, which leads to the problem of relatively low precision of memory assembly in the related art. To solve the above problem, the embodiment of the present application provides a memory assembly method, in the case of identifying the target carrier where the memory bar to be installed is located, the mechanical arm grasps the memory bar to be installed in the target carrier, constructs an image pyramid of the memory bar grasped image, and performs feature extraction operation on the image pyramid to obtain feature point information of the memory bar to be installed, thereby determining the actual posture of the memory bar, so as to adjust the end posture of the mechanical arm, so that the mechanical arm can dynamically correct the posture of the memory bar to be installed, solves the problem of relatively low precision of memory assembly caused by the posture deviation that may occur in the memory bar after grasping in the related art, and improves the accuracy and reliability of memory assembly.

[0021] The embodiment of the present application provides a memory assembly method. Taking the memory assembly method in the embodiment executed by a control device as an example, the memory assembly method is applied to a memory assembly system. The memory assembly system includes a robot and a first visual sensor. The robot is equipped with a mechanical arm. Figure 1 A flowchart of an optional memory assembly method provided according to an embodiment of the present application is shown in FIG. Figure 1 As shown, the process of the method may include steps S102 to S112.

[0022] Step S102, when the target carrier where the memory stick to be installed is located is identified, the memory stick to be installed in the target carrier is grabbed by a robotic arm, and a memory stick grabbing image is obtained by a first visual sensor, wherein the memory stick grabbing image is an image when the robotic arm grabs the memory stick to be installed.

[0023] It should be noted that the memory assembly system may include a robot and a first visual sensor, and the robot may be equipped with a robotic arm. Among them, the robot may adopt a composite robot, which may refer to an advanced robot system that combines multiple technologies and functions, usually including but not limited to: multi-degree-of-freedom robotic arms, multiple types or modes of end effectors (such as grippers, suction cups, etc.), and highly integrated control systems of visual sensors (such as monocular cameras, stereo cameras, or depth cameras). The composite robot can autonomously perform precise grasping, operation, and assembly tasks in complex environments through a robotic arm and an intelligent perception system, and has a high degree of autonomy, adaptability, and operational accuracy.

[0024] The memory stick to be installed refers to the memory component that has not yet been installed in the memory slot of the server motherboard or other electronic devices. The memory stick to be installed is usually composed of a circuit board, memory particles (storage chips), gold fingers (metal sheets for contacting the motherboard slot), etc., with standard specifications and sizes. The target carrier can refer to a container or platform that carries the memory stick to be installed, which plays a role in positioning and transferring the memory stick in the memory assembly process, such as a tray.

[0025] The robot can move to a workbench where multiple carriers are placed. After identifying the target carrier where the memory stick to be installed is located, the robot can move to the vicinity of the workbench where the target carrier is located, and can accurately move the robotic arm to the top of the area where the target carrier is located, and use the robotic arm to grab the memory stick to be installed. After grabbing the memory stick to be installed, the robotic arm can be photographed by the first visual sensor to obtain a memory stick grab image, that is, an image when the robotic arm of the composite robot grabs the memory stick to be installed. Optionally, the first visual sensor can be an external visual sensor, and a variety of visual sensor technologies such as stereo vision, infrared or laser scanning can be used to obtain a memory stick grab image. The use of visual sensors is not limited to monocular cameras, but can also be infrared cameras or structured light scanning systems to adapt to different lighting conditions or obtain more accurate depth information.

[0026] It should be noted that after grabbing the memory stick to be installed, the end of the robot arm will be deflected to a certain extent from the memory stick to be installed, such as Figure 2As shown, this deflection will seriously affect the success rate of assembly to a certain extent. If the deflection is not taken into account, it may lead to installation failure at the least and damage to components or even the mainboard at the worst. Therefore, it is necessary to adjust the posture of the end of the robot arm, that is, the assembly angle of the gripper, so as to improve the success rate and reliability of assembly.

[0027] Step S104 : constructing an image pyramid of the memory bar captured image, wherein the image pyramid includes a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the memory bar captured image.

[0028] It should be noted that when constructing the image pyramid of the memory bar captured image, the memory bar captured image can be first filtered, and optionally, the memory bar captured image is filtered using a preset filtering algorithm to obtain the filtered memory bar captured image, wherein the preset filtering algorithm may include at least one of the following: bilateral filtering, median filtering, and Gaussian filtering. The purpose of filtering is to remove noise, smooth the image, and provide a better basis for subsequent feature point extraction and edge detection.

[0029] The image pyramid may include a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the filtered memory bar captured image. Specifically, the filtered memory bar captured image is subjected to multiple downsampling operations to generate downsampled images of different granularities, thereby obtaining an image pyramid of the memory bar captured image.

[0030] Specifically, the downsampling operation can be used to reduce the resolution of the image to achieve feature point detection at different scales. Optionally, the downsampling operation on the filtered memory bar captured image may include: merging continuous pixel blocks in the filtered memory bar captured image into a new pixel, and the new pixel can be determined by the average value, median value or maximum value of the original pixel block. Specifically, the filtered memory bar captured image is divided into pixel blocks of equal size, and the statistical value (such as the average value, median value or maximum value) of each pixel block is calculated as the value of the new pixel, and each new pixel is copied to the original pixel position to form a downsampled image.

[0031] Of course, in the process of constructing the image pyramid, a Gaussian pyramid can be used, by performing Gaussian blur on the memory bar captured image after filtering, and then performing pixel merging processing. Specifically, the memory bar captured image is processed by a Gaussian filter, and then the Gaussian blurred image is downsampled using pixel merging.

[0032] Of course, the downsampling operation may also use bilinear interpolation, bicubic interpolation or wavelet transform downsampling, which is not limited in this application.

[0033] By capturing images from a filtered memory bar constructed through downsampling operations, the computational complexity of subsequent image processing (such as edge detection, etc.) can be reduced. In addition, by constructing image pyramids of different resolutions, feature detection can be performed at different scales, which can improve the robustness and accuracy of feature detection.

[0034] Step S106, performing a feature extraction operation on the image pyramid to obtain feature point information of the memory stick to be installed.

[0035] It should be noted that the feature points of the memory bar to be installed refer to points with specific meanings in the captured image of the memory bar, for example, the edges and corners of the memory bar to be installed in the captured image of the memory bar. The feature point information of the memory bar to be installed is obtained by extracting feature points from the captured image of the memory bar, and is used to determine the information of the actual posture of the memory bar to be installed.

[0036] Optionally, during the feature extraction operation (ie, feature point extraction), image processing techniques such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features) may be used to identify key points in the image, such as edges and corners of the memory stick to be installed.

[0037] In specific practice, in order to improve the accuracy of actual posture determination, feature point extraction can also take into account standard size and shape characteristics of the memory bar to be installed. For example, the memory bar usually has a standard aspect ratio and a specific edge profile.

[0038] Optionally, the number of feature points of the memory bar to be installed in the captured image of the selected memory bar can be determined according to the size and shape features of the memory bar to be installed. For example, when the size of the memory bar is large or the shape is complex, more feature points may need to be selected to obtain a more accurate posture.

[0039] Step S108: determining the actual posture of the memory module to be installed according to the feature point information of the memory module to be installed.

[0040] It should be noted that the actual posture of the memory bar to be installed may refer to the actual position and direction of the memory bar to be installed in space. Optionally, the actual posture of the memory bar to be installed may refer to the position and rotation angle of the memory bar to be installed relative to the first visual sensor.

[0041] Optionally, there is a corresponding standard size of the memory bar to be installed, and the actual posture of the memory bar to be installed is determined by combining the feature point information of the memory bar to be installed and the standard size of the memory bar to be installed. Specifically, when determining the actual posture of the memory bar, RANSAC (random sample consensus) can be used to remove outliers in the feature points to improve the robustness of the posture estimation. In addition, in order to adapt to different types of memory bars, it may be necessary to store the standard size and shape information of multiple memory bars to ensure the accuracy of the posture estimation.

[0042] Step S110, adjusting the end posture of the robot arm based on the actual posture and the preset posture of the memory stick to adjust the posture of the memory stick to be installed.

[0043] It should be noted that, when actually assembling the memory stick, there are many different types and specifications of memory sticks. Therefore, a series of preset memory stick postures can be pre-stored. The preset memory stick posture corresponding to the memory stick to be installed can be searched to calculate the adjustment posture of the end of the robot arm according to the deviation between the actual posture and the preset memory stick posture, so as to adjust the end posture of the robot arm. The preset memory stick posture corresponding to the memory stick to be installed can be a preset standard posture corresponding to the memory stick to be installed.

[0044] The adjustment posture of the end of the robotic arm may include the rotation and translation adjustment amount that the end of the robotic arm needs to perform to ensure that the memory stick to be installed can be aligned with the corresponding memory slot of the memory stick to be installed in the correct posture.

[0045] Optionally, during the end posture adjustment process of the robot arm, force feedback control can be set to ensure that the memory bar is not damaged while adjusting the posture.

[0046] Step S112, installing the memory stick to be installed into the designated memory slot corresponding to the memory stick to be installed by the robotic arm after the end posture is adjusted.

[0047] It should be noted that the designated memory slot may refer to a specific slot position of a memory module to be installed on the device to be installed. The device to be installed may be a server motherboard.

[0048] Optionally, the robot moves in the adjusted posture to move the memory module to be installed to the top of the designated memory slot and install it, and during the installation process, force feedback control is used to ensure a safe threshold of force during installation. The safe threshold of force can be determined based on multiple factors, such as the physical properties of the memory module and the server motherboard (including the hardness and thickness of the memory module and the strength and toughness of the server motherboard material), the design of the memory module slot, etc.

[0049] In actual practice, during the installation phase of the memory stick to be installed, the robot may adopt a progressive force control strategy, that is, first gently contacting the memory slot, and then adjusting according to the real-time force feedback information and the preset safety threshold to ensure that the memory stick to be installed is safely inserted without damaging the motherboard.

[0050] Through the embodiments provided by the present application, when a target carrier where a memory stick to be installed is located is identified, the memory stick to be installed in the target carrier is grabbed by a robotic arm, and a grabbed image of the memory stick is obtained by a first visual sensor, and an image pyramid of the grabbed image of the memory stick is constructed, allowing feature detection at different image resolutions, thereby improving the accuracy of feature extraction operations, thereby obtaining more accurate feature point information of the memory stick to be installed, and determining the actual posture of the memory stick. According to the actual posture of the memory stick to be installed and the preset posture of the memory stick, the end posture of the robotic arm is adjusted, so that the robotic arm can dynamically correct the posture of the memory stick to be installed, thereby solving the problem of low precision of memory assembly caused by posture deviation that may occur in the memory stick after grabbing in the related art, improving the accuracy and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving production efficiency.

[0051] In an exemplary embodiment, the robot is configured with a second visual sensor; a memory bar to be installed in a target carrier is grabbed by a robotic arm, including: acquiring a first area image by the second visual sensor, wherein the first area image is an image of an area in the target carrier where the memory bar to be installed is located; based on the first area image, identifying the relative position of the memory bar to be installed and the robotic arm and the relative rotation angle between the memory bar to be installed and the robotic arm; based on the relative position and the relative rotation angle, adjusting the posture of the robotic arm, and using the robotic arm with the adjusted posture to grab the memory bar to be installed.

[0052] It should be noted that the robot can be configured with a second visual sensor. The second visual sensor in the robot can refer to an internal visual sensor installed at the end effector of the robot, which can be used to capture the image of the memory stick to be installed when the robotic arm needs to grab it. Compared with the first visual sensor, the second visual sensor can observe the target at a closer distance and provide more detailed visual data. The first area image can be an image taken by the robot's second visual sensor of the target carrier area (Tray or other carrying platform) where the memory stick to be installed is located before grabbing the memory stick to be installed. The first area image may include the environmental background of the memory stick to be installed and the specific position of the memory stick to be installed in the Tray.

[0053] The relative position of the memory bar to be installed and the robot arm may refer to the spatial position relationship between the memory bar to be installed and the end effector of the robot arm, that is, the distance between the two on the x, y, and z axes. The relative rotation angle of the memory bar to be installed and the robot arm may refer to the spatial rotation angle relationship between the memory bar to be installed and the end effector of the robot arm, that is, the rotation angle of the two around the x, y, and z axes.

[0054] Optionally, in the process of grasping the memory bar to be installed by the robot arm, the memory bar to be installed in the target carrier is photographed by the second visual sensor on the robot to obtain a first area image. The first area image is processed by, for example, color threshold segmentation, template matching processing, etc., and the relative position and relative rotation angle between the memory bar to be installed and the end effector of the robot arm are calculated through the first area image after image processing, so as to adjust the posture of the robot arm according to the relative position and relative rotation angle, so as to grasp the memory bar to be installed by using the adjusted robot arm.

[0055] Optionally, in the case where template matching is used for image processing, the first region image is preprocessed, and the preprocessing step may include image processing techniques such as grayscale, edge detection, smoothing (such as Gaussian filtering, median filtering), and binarization. The preprocessed first region image is template matched with the template image, such as correlation matching, normalized cross-correlation (NCC) or a fast template matching algorithm (such as a feature-based matching algorithm). The region most similar to the template image is searched in the preprocessed first region image to obtain the position information of the memory bar to be installed in the first region image, that is, the coordinates of the center of the memory bar or a certain feature point in the image coordinate system. By comparing the similarity or correlation between the template image and the first region image (mainly the long side and short side directions of the memory bar), the relative rotation angle between the memory bar to be installed and the robotic arm is calculated.

[0056] Through this embodiment, the relative position and relative rotation angle are estimated by the second visual sensor, which improves the accuracy of the robot in grasping the memory stick to be installed and reduces the grasping failure rate due to the posture error of the robot arm to a certain extent.

[0057] In an exemplary embodiment, obtaining a first area image by using a second visual sensor includes: determining target area information, wherein the target area information is used to indicate an area in a target carrier where a memory bar to be installed is located; and obtaining a first area image by photographing the memory bar to be installed by using the second visual sensor according to the target area information.

[0058] It should be noted that the target area information may be used to indicate an area in the target carrier where the memory stick to be installed is located.

[0059] Optionally, the second visual sensor in the robot is used to search and locate the target carrier, that is, when the target carrier where the memory bar to be installed is located is identified, the precise position of the memory bar to be installed on the target carrier can be determined by a preset target detection algorithm, such as barcode recognition based on machine vision. Once the position of the memory bar to be installed on the target carrier is determined, the viewing angle and focal length of the second visual sensor are adjusted to ensure that the sensor can clearly capture the image of the memory bar to be installed, that is, the first area image.

[0060] Optionally, the target area information of the memory stick to be installed in the target carrier may be determined based on the preset barcode position of the target carrier and the preset position offset.

[0061] In one example, a barcode on a target carrier is identified by a second visual sensor installed at the end of the robotic arm. Based on the barcode information and the preset position offset algorithm, the specific position of the memory bar in the target carrier can be determined. For example, through barcode recognition, it is known that the coordinates of the barcode on a tray are (100, 150). Based on the preset spatial relationship between the memory bar and the barcode, it can be inferred that the first memory bar is located at (150, 200). In this way, the target area information of the memory bar is determined, and then, by adjusting the second visual sensor at the end of the robotic arm to align it with the area indicated by the target area information, a high-definition image of the memory bar is taken.

[0062] Through this embodiment, the second visual sensor can capture the image of the memory bar to be installed at close range and with high precision, so that the specific position and range of the memory bar on the target carrier (such as a tray) can be more accurately identified. This close-range visual feedback can significantly improve the accuracy of feature point detection; before the robot grabs the memory bar, the first area image obtained by the second visual sensor can provide real-time visual feedback, and can instantly adjust the posture of the robot arm to align the memory bar for grabbing. To a certain extent, it optimizes the assembly process, reduces the preparation time and operational errors before assembly, and improves the assembly quality and success rate. The built-in visual sensor allows the robot to reduce its dependence on fixed external cameras, and the robot arm can move freely in space to adapt to the position and posture of the memory bar in different working scenarios, enhancing the flexibility of memory assembly. This flexibility enables the assembly system to more efficiently handle memory bars of multiple models and sizes.

[0063] In an exemplary embodiment, the relative position of the memory bar to be installed and the robotic arm and the relative rotation angle between the memory bar to be installed and the robotic arm are identified based on the first area image, including: performing edge detection on the first area image to obtain line segment information of the first area image, wherein the line segment information of the first area image is used to indicate the edge of the memory bar to be installed; determining the relative position of the memory bar to be installed and the robotic arm and the relative rotation angle between the memory bar to be installed and the robotic arm based on the line segment information of the first area image and the posture information of the robotic arm.

[0064] It should be noted that the line segment information of the first area image may refer to a set of line segments extracted from the first area image through edge detection processing, which can represent the shape and position of the memory bar to be installed. Each line segment in the line segment set may be a digital representation of the edge of the memory bar to be installed, which is used for subsequent feature point recognition and position and posture calculation. The posture information of the manipulator may include its current position coordinates and plane rotation angle, which is a posture parameter describing the manipulator in three-dimensional space.

[0065] Optionally, the relative position of the memory stick to be installed and the robotic arm refers to the distance and translation vector between the center point of the memory stick to be installed and the center point of the end effector of the robotic arm in the world coordinate system; the relative rotation angle refers to the angle difference between the long side and short side direction of the memory stick to be installed and the long side and short side direction of the end effector of the robotic arm.

[0066] Optionally, edge detection is performed on the first area image to obtain line segment information of the first area image, which may specifically include: preprocessing the first area image, using an edge detection algorithm, such as Canny edge detection or a Sobel operator, to extract line segment information of the edge of the memory bar to be installed from the first area image, such as the long side and short side of the memory bar to be installed, and the straight line segments of each edge.

[0067] Optionally, based on the line segment information of the first area image and the posture information of the robotic arm, the relative position of the memory stick to be installed and the robotic arm and the relative rotation angle between the memory stick to be installed and the robotic arm are determined, including: based on the extracted line segment information of the first area image and the posture information of the robotic arm, through coordinate transformation and posture estimation algorithms, such as PnP (Perspective-n-Point) problem solving, the relative position and translation vector, and the relative rotation angle between the memory stick and the robotic arm are calculated.

[0068] Through this embodiment, by identifying the relative position and rotation angle between the memory stick to be installed and the robotic arm, the robot can accurately adjust the posture of the robotic arm to ensure that it can accurately align with the edge of the memory stick to be installed when grasping it. This can avoid grasping failure or damage caused by posture mismatch to a certain extent, and improve the stability of the memory stick to be installed during the grasping process.

[0069] In an exemplary embodiment, feature point extraction is performed on a captured image of a memory stick to obtain feature point information of the memory stick to be installed, including: performing edge detection on downsampled images in an image pyramid to obtain a set of edge maps; performing image construction based on pixel values ​​of pixel points of edge maps in the set of edge maps to obtain a first intermediate image; performing contour recognition on the first intermediate image, extracting and screening the contour in the first intermediate image to obtain a second intermediate image, wherein the second intermediate image is an image of a region within the contour in the first intermediate image; identifying feature points in the second intermediate image to obtain pixel coordinate information of the feature points in the second intermediate image, wherein the pixel coordinate information of the feature points in the second intermediate image is feature point information of the memory stick to be installed.

[0070] Optionally, edge detection is performed on the downsampled images in the image pyramid to obtain a set of edge maps. The memory bar contour features at different scales can be discovered through the set of edge maps, and the image is constructed based on the pixel values ​​of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image.

[0071] The first intermediate image may be constructed based on the edge map of the downsampled image in the image pyramid, and the contour information of the memory bar to be installed may be obtained by analyzing the pixel values. The second intermediate image may be a regional image within the contour in the first intermediate image, and the second intermediate image may be used for accurate recognition of feature points. Optionally, an edge map is generated from the downsampled image in the image pyramid, and then a set of pixel values ​​corresponding to each pixel point in the edge map is analyzed to construct the first intermediate image, and contour recognition is performed on the first intermediate image, and contours in the first intermediate image are extracted and screened to obtain the second intermediate image. Optionally, in the process of extracting and screening contours in the first intermediate image, polygonal approximation may be performed on all contours of the first intermediate image, and contours that are approximately quadrilaterals and whose areas meet a specified threshold are screened out, wherein the specified threshold may be determined based on the size information of the memory bar to be installed.

[0072] The feature point information of the memory bar to be installed is obtained by performing a feature extraction operation on the captured image of the memory bar, and is used to determine the actual posture of the memory bar to be installed. Specifically, the feature point information of the memory bar to be installed may include pixel coordinate information of the feature points in the second intermediate image.

[0073] Optionally, feature points of the memory bar to be installed are obtained by performing feature point detection on the second intermediate image. Feature point detection may include but is not limited to corner point detection, edge detection, Radon transform, etc. Feature points of the memory bar to be installed may include but are not limited to corner points or specific edge points.

[0074] In actual practice, other methods such as deep learning, probability projection, etc. can also be used to extract the feature points of the installed memory stick.

[0075] Through this embodiment, through image processing processes such as downsampling and edge detection, a preprocessing basis is provided for feature point extraction, ensuring high accuracy of feature point detection. At the same time, by constructing an image pyramid and edge map for analysis, the processing speed and computational efficiency of feature point extraction are improved to a certain extent.

[0076] In an exemplary embodiment, an image is constructed based on pixel values ​​of pixel points of an edge map in a group of edge maps to obtain a first intermediate image, including: traversing edge maps in a group of edge maps and obtaining pixel values ​​of pixel points of an edge map in a group of edge maps; taking the maximum pixel value of pixel points at the same position in a group of edge maps as the pixel value of pixel points at corresponding positions in the image to be constructed, and constructing the image to obtain the first intermediate image.

[0077] It should be noted that a set of edge maps may refer to a set of images obtained by performing edge detection on each layer of downsampled images in the process of building an image pyramid. A set of edge maps may highlight the outline and edge information of the memory module to be installed.

[0078] Optionally, the image pyramid may include downsampled images at different levels. At each level of the image pyramid, a preset edge detection algorithm, such as a Canny algorithm or a Sobel operator, is used to process the downsampled image in the image pyramid to obtain an edge map of the corresponding level.

[0079] The pixel value of a pixel refers to the grayscale value or color intensity value of each pixel in the image, which reflects the image feature strength of the pixel. The first intermediate image is a comprehensive image obtained by merging the maximum pixel values ​​of the pixels at the same position in a group of edge maps to construct an image. The first intermediate image can focus on showing the most significant edge features of the memory module to be installed.

[0080] In one example, by traversing the pixels of each edge map and recording the pixel value of each pixel, the higher the grayscale value, the more obvious the edge, and the lower the grayscale value, the weaker the edge or not an edge. For example, at a certain pixel position (200, 150) in the edge map, the grayscale values ​​of different layers are recorded as 255, 230, 240, and 250, respectively. Then the one with the largest grayscale value (such as 255) is selected as the pixel value of the corresponding position in the first intermediate image. After this process, the first intermediate image clearly reveals the contour lines of the memory bar, which is convenient for subsequent feature point extraction.

[0081] Through this embodiment, by merging the first intermediate image constructed by the maximum grayscale value in a group of edge maps, the edge features of the memory bar to be installed can be strengthened, so that the feature point extraction of the memory bar to be installed is more accurate. By constructing the first intermediate image instead of processing each edge map separately, the feature point identification of the memory bar to be installed can be completed with less computing resources and time, which improves the efficiency of the memory assembly process to a certain extent.

[0082] In an exemplary embodiment, identifying feature points in the second intermediate image and obtaining pixel coordinate information of the feature points in the second intermediate image includes: performing edge detection on the second intermediate image to obtain an edge image corresponding to the second intermediate image; performing Radon transform on the edge image corresponding to the second intermediate image to obtain a Radon transform result; performing peak detection on the Radon transform result to obtain straight line parameters in the second intermediate image; determining feature points in the second intermediate image based on the straight line parameters in the second intermediate image, and calculating pixel coordinate information of the feature points in the second intermediate image.

[0083] It should be noted that the second intermediate image may refer to an image that contains only the memory bar to be installed after contour recognition and screening. The second intermediate image is contour screened to remove non-memory bar areas in the image and retain the clear contour and internal structure of the memory bar.

[0084] Use edge detection algorithms (such as Canny edge detection, Sobel operator, etc.) to identify the boundaries and feature lines of the memory bar to be installed in the second intermediate image, and obtain the edge image corresponding to the second intermediate image. The Radon transform result may include but is not limited to the radial distance and angle parameters of the edge straight line of the memory bar to be installed, the peak points in the Radon space, and the distribution information of the straight line parameters. Among them, the radial distance represents the vertical distance between the straight line and the center of the image, and the angle represents the angle between the straight line direction and the horizontal axis. Each set of radial distance and angle combinations corresponds to a straight line on the memory bar. The peak point in the Radon space can represent the most obvious straight line feature on the memory bar to be installed. The distribution information of the straight line parameters may include the distribution of different straight line parameters in the Radon space.

[0085] Optionally, performing a Radon transform on the edge image corresponding to the second intermediate image to obtain a Radon transform result may include: mapping the edge lines of the memory bar to be installed into the Radon space, using the radial distance and angle of the straight line as parameters to form a Radon transform result. Specifically, performing a Radon transform on the edge image corresponding to the second intermediate image, projecting the edge lines of the long side and the short side of the memory bar to be installed into the Radon space. Each edge line is represented as a curve in the Radon space, and the peak points of these curves correspond to the straight lines on the edge of the memory bar to be installed, and the radial distance and angle of the straight line on the edge of the memory bar to be installed are the straight line parameters.

[0086] Specifically, the Radon transform of the edge image corresponding to the second intermediate image can be performed using formula (1):

[0087] ; (1)

[0088] in, is the result of Radon transform, is the perpendicular distance from the origin to the line, is the perpendicular line (i.e. the vector perpendicular to the line) and The angle of the positive axis, is the Dirac function, and is the edge in the image coordinate system Axis and The position of the axis.

[0089] The peak value is detected on the Radon transform result, and the local maximum detection algorithm is used to find the peak point. The peak point corresponds to the straight line on the edge of the memory bar. The radial distance and angle can be extracted from it to describe the position and direction of the straight line, thereby obtaining the straight line parameters in the second intermediate image. The straight line equation can be shown as formula (2):

[0090] ; (2)

[0091] in, Represents the coordinate point in the image coordinate system.

[0092] Specifically, since the edge of the memory stick to be installed is a straight line and the feature point can be a corner point, the angle corresponding to the feature point is a right angle. The angle range of the Ladon change can be focused on 0°, 90°, 180°, and 270°, and the sampling density can be appropriately increased.

[0093] Specifically, the coordinates of the corner point can be solved by the equation set formed by the two straight lines in the local area (i.e., formula (3)):

[0094] ; (3)

[0095] in, , ; , The normals of the two lines can be The angle of the positive axis; and Can be corner point coordinates.

[0096] Specifically, for each feature point, the corner point position of the memory stick to be installed can be determined by formula (3), that is, the pixel coordinate information of the feature point in the second intermediate image. In practice, under different forms of gripping states, some corner points may be blocked. Therefore, the unblocked corner points can be defined as feature points to participate in the calculation.

[0097] Through this embodiment, by using Radon transform and performing peak detection on the Radon transform result, the straight line features of the second intermediate image are obtained, and the key feature points on the memory bar can be accurately identified. In addition, through accurate feature point identification and coordinate information calculation, the assembly error caused by inaccurate grasping angle or position can be effectively reduced, thereby improving the assembly quality and success rate.

[0098] In an exemplary embodiment, the feature point information of the memory bar to be installed is pixel coordinate information of the feature point of the memory bar to be installed.

[0099] According to the feature point information of the memory bar to be installed, the actual posture of the memory bar to be installed is determined, including: according to the specification information of the memory bar to be installed and the world coordinate system, the pixel coordinate information of the feature points of the memory bar to be installed is mapped to obtain the three-dimensional coordinate information of the feature points of the memory bar to be installed; according to the camera coordinate system of the first visual sensor, the three-dimensional coordinate information of the feature points of the memory bar to be installed and the pixel coordinate information of the feature points of the memory bar to be installed, the parameter information of the memory bar to be installed is calculated, wherein the parameter information of the memory bar to be installed includes the rotation parameters of the memory bar to be installed relative to the first visual sensor and the translation parameters of the memory bar to be installed relative to the first visual sensor; according to the parameter information of the memory bar to be installed, the actual posture of the memory bar to be installed is calculated.

[0100] It should be noted that the specification information of the memory bar to be installed includes detailed parameters such as the size, shape, and three-dimensional coordinates of key feature points of the memory bar. The specification information of the memory bar to be installed can usually be obtained from the manufacturing data or specification manual of the memory bar to be installed. In specific practice, since the machining accuracy of the memory bar size is ±0.15 mm, the three-dimensional coordinate information of the feature points of the memory bar to be installed can be directly determined based on the size of the memory bar.

[0101] The feature point information of the memory bar to be installed is the pixel coordinate information of the feature point of the memory bar to be installed, that is, the pixel coordinate information of the key feature point in the second intermediate image. For example, the coordinate information of the four corner points (150, 100), (350, 100), (150, 200), (350, 200) is detected from the second intermediate image.

[0102] The three-dimensional coordinate information of the feature point of the memory stick to be installed can be a description of the position of the feature point of the memory stick to be installed in space, usually including x, y and z coordinates, which are used to characterize the position of the feature point in the world coordinate system. Specifically, the pixel coordinate information of the feature point is converted into a three-dimensional coordinate in the world coordinate system through mapping processing.

[0103] The parameter information of the memory bar to be installed includes the rotation parameters of the memory bar to be installed relative to the first visual sensor and the translation parameters of the memory bar to be installed relative to the first visual sensor. The relative rotation parameters and the relative translation parameters describe the change in the posture of the memory bar to be installed relative to the first visual sensor, and are used to represent the three-dimensional posture of the memory bar. Specifically, according to the pixel coordinates of the corner point in the image coordinate system, the camera parameters of the first visual sensor, and the camera coordinate system, the coordinate information of the corner point in the camera coordinate system is determined; by solving the PnP (Perspective-n-Point) problem, the three-dimensional coordinate information of the corner point and the coordinate information of the corner point in the camera coordinate system are used to calculate the rotation parameters and translation parameters of the memory bar to be installed relative to the first visual sensor.

[0104] Specifically, Figure 3 Take this as an example to illustrate: Figure 3 The following is a schematic diagram of characteristic points of an optional memory stick provided according to an embodiment of the present application. Figure 3 The part pointed by the arrow in the figure can be the characteristic point of the memory bar. Figure 3 For example, take corner point A as the origin of the coordinate system and the two orthogonal axes in the plane as Axis and Axis (usually along the long side Axis, short side is axis), The direction of the plane normal vector is The axis defines the coordinate system and uses this coordinate system as the world coordinate system ; Define the first visual sensor as a monocular camera, and the corresponding camera coordinate system is , the origin is located at the optical center of the external monocular camera, The axis is along the optical axis of the external monocular camera. The direction is the same as the xy definition of the image plane; the image plane coordinate system , Axis and The axes are parallel to the horizontal and vertical directions of the image respectively. Then the two-dimensional pixel coordinates of the corner points obtained by corner point detection in the image coordinate system are shown in formula (4):

[0105] ; (4)

[0106] Where N is the number of corner points, ( ) ( ) represents the two-dimensional pixel coordinates of the corner point j in the image coordinate system.

[0107] The coordinate expression of a point in the camera coordinate system of a monocular camera is shown in formula (5):

[0108] ; (5)

[0109] in, It can represent the position of corner point j in the camera coordinate system;

[0110] Then the coordinates projected onto the image plane The coordinates of can be calculated using formula (6):

[0111] ; (6)

[0112] in, is the equivalent focal length in the x-axis direction of the camera coordinate system, is the equivalent focal length in the y-axis direction of the camera coordinate system, is the pixel size in the x-axis direction of the camera coordinate system, is the pixel size in the y-axis direction of the camera coordinate system, is the distance from the origin of the camera coordinate system in the x-axis direction of the image coordinate system, is the distance from the origin of the camera coordinate system in the y-axis direction of the image coordinate system, which can be obtained through calibration.

[0113] The corner point is transformed from the world coordinate system to the camera coordinate system through the rotation matrix and translation vector The specific formula is as shown in formula (7):

[0114] ; (7)

[0115] in, is the rotation matrix; is the translation vector; is the position of corner point j in the world coordinate system.

[0116] The representation of the rotation matrix and translation vector can be referred to as shown in formula (8):

[0117] ; (8)

[0118] in, represents the rotation transformation parameters, , , Represents the translation transformation parameters.

[0119] For each corner point j, two equations (such as formula (9) and formula (10)) can be obtained (ignoring the non-zero factors multiplied on both sides of the equation):

[0120] ; (9)

[0121] ; (10)

[0122] in,( , ) is the focal length of the monocular camera; ( , ) is the pixel size of the monocular camera, usually the physical size of a unit pixel; ( , ) is the pixel coordinate of the projection of the optical center of the monocular camera on the image.

[0123] Among them, the rotation matrix There are 9 elements, the translation vector There are 3 elements, but since the rotation matrix is ​​an orthogonal matrix, it satisfies formula (11):

[0124] ; (11)

[0125] Where I is the identity matrix; for The transposed matrix of The rows and columns of are swapped. Therefore, there are only 6 parameters (3 rotation parameters and 3 translation parameters) that need to be solved, that is, the parameter information of the memory module to be installed.

[0126] In practice, the nonlinear least squares method can be used to solve the posture parameters (i.e., the parameter information of the memory module to be installed), as shown in formula (12):

[0127] ; (12)

[0128] in, and is the projection coordinates of each corner point calculated under the parameter information of the memory bar to be installed. is the error value, and the SVD (Singular Value Decomposition) method can be used to solve The rotation matrix for the minimum value and translation vector t, and then determine the posture parameters obtained by solution.

[0129] According to the posture parameters (parameter information of the memory bar to be installed) obtained by solving, the actual posture of the memory bar relative to the robot operation space (i.e., the end of the robot arm) is determined. The actual posture may include the position coordinates and deflection angle of the memory bar, so that according to the difference between the actual posture and the preset posture of the memory bar, the end posture of the robot arm is adjusted, thereby adjusting the posture of the memory bar to be installed.

[0130] Through this embodiment, the pixel coordinate information of the feature point is converted into three-dimensional coordinate information, and then the relative rotation parameters and translation parameters are calculated, and finally the actual posture of the memory bar is determined, so as to adjust the posture of the end of the robotic arm, thereby improving the assembly success rate and reliability to a certain extent.

[0131] In an exemplary embodiment, before installing the memory module to be installed into the designated memory slot corresponding to the memory module to be installed by moving the robot arm after the posture adjustment of the terminal, the method further includes:

[0132] Acquire a device image, wherein the device image is an image of a target device into which the memory stick to be installed is to be installed; extract slot features from the device image to obtain slot features; determine position information of a specified memory slot based on the slot features, and calculate a slot position of the specified memory slot based on the position information of the specified memory slot, wherein the robot arm after the posture adjustment of the mobile terminal is executed according to the slot position of the specified memory slot.

[0133] It should be noted that the device image may refer to an image of the target device taken by the robotic arm before preparing to install the memory stick to be installed. For example, in the case where the target device is a server, the device image contains various features of the server motherboard, especially the image information of the memory slot. Slot feature extraction refers to identifying specific features of the memory slot from the device image, such as shape, size, position, etc., which will be used to calculate the position information of the slot. The position information of a specified memory slot may include the position coordinates and attitude angle of the specified slot, that is, the three-dimensional coordinates and rotation angle relative to the world coordinate system. The slot position may refer to the precise position coordinates of the specified memory slot relative to the robot or a preset reference point in the world coordinate system.

[0134] In order to further ensure that the memory bar to be installed can be accurately installed in the target device to be installed, it is also necessary to consider identifying the slot features of the target device and calculating the actual position and posture information of the slot. Specifically, after detecting and adjusting the posture of the memory bar, the robot end is adjusted to the specified position to ensure that the device image of the target device is captured. The device image is preprocessed to reduce noise interference, and then image processing algorithms (such as edge detection, feature point matching, etc.) are used to identify the features of the memory slot. For example, the outline and internal structure details of the memory slot can be identified by template matching or deep learning algorithms, and then the position and possible tilt angle of the memory slot can be determined. After identifying the features of the memory slot, combined with the known layout information of the device and the theoretical position information of the slot, similar image processing and geometric calculation methods (such as Radon transform and PnP problem solving) are used to determine the position information of the specified memory slot, and based on the position information of the specified memory slot, the slot position of the specified memory slot is calculated, which can specifically include the coordinate values ​​and various rotation angles of the specified memory slot on the x, y, and z axes. Move the robot arm according to the slot position of the specified memory slot to ensure that the memory module can be accurately inserted into the corresponding slot to avoid collision during installation or installation failure due to posture mismatch.

[0135] Optionally, the process of determining the slot position of a specified memory slot may be: preprocessing the device image, including but not limited to edge enhancement, denoising, and contrast adjustment, to enhance the visibility of the slot features and reduce interference. Use image enhancement techniques (such as histogram equalization and contrast stretching) to improve the image contrast of the memory slot to facilitate subsequent feature recognition. Perform edge detection on the preprocessed device image to obtain edge information of the memory slot. Use Hough line detection to identify straight line segments from the edge information, which correspond to the contour of the memory slot, thereby edge features, and use template matching or deep learning methods (such as convolutional neural networks (CNNs)) to identify the specific shape and structure of the memory slot. Determine which areas in the image are memory slots, as well as the specific locations and possible tilt angles of the memory slots through feature point matching, contour analysis, or deep learning classification.

[0136] In order to better understand the memory assembly process, an example is used to illustrate. Figure 4 As shown, Figure 4 A schematic diagram of an optional memory assembly provided according to an embodiment of the present application.

[0137] Among them, the robot is a composite robot, the end of the robot's mechanical arm can be a two-finger gripper, a built-in visual sensor (i.e., a hand-eye camera) is configured near the end of the mechanical arm, the two-finger gripper clamps a memory stick (i.e., the memory stick to be installed), the first visual sensor is a monocular camera, the target device to be installed can be a server motherboard, and the memory stick to be installed is installed into the designated memory slot corresponding to the memory stick to be installed by moving the mechanical arm after adjusting the posture of the end.

[0138] Through this embodiment, by extracting slot features and calculating position information of the device image, this solution can accurately determine the position and tilt angle of the memory slot, and even under complex environmental conditions, it can achieve accurate alignment between the memory stick and the slot, significantly improving assembly accuracy. By accurately calculating the position information of the slot, it is ensured to a certain extent that the memory stick to be installed is installed in the slot with the correct posture and position, avoiding hardware damage that may be caused by inaccurate positioning, such as damage to the contacts of the memory stick or wear of the slot of the motherboard. In addition, combined with the adjustment of the memory stick posture and the accurate calculation of the slot position information, the success rate of memory installation can be greatly improved, the failure rate can be reduced, thereby reducing the product defect rate and improving the overall production quality.

[0139] In an exemplary embodiment, the method further includes: acquiring an image of a detection area through a first visual sensor to obtain a second area image, wherein the detection area is an area where a set carrier is located; extracting an area of ​​interest in the second area image, and performing barcode recognition on the extracted area of ​​interest to obtain a barcode recognition result; when the barcode recognition result indicates that a memory stick to be installed is identified, determining a carrier on which the memory stick to be installed is placed as a target carrier.

[0140] It should be noted that the first visual sensor refers to a visual device installed outside the robot, such as a monocular camera or a stereo camera, which is used to collect images of the detection area to obtain the second area image.

[0141] The detection area can be the area where the vehicle is set to be located, which generally refers to a specific spatial area that needs to be detected during the robot's operation.

[0142] The second area image may be an image of the detection area captured by the first visual sensor, including one or more vehicles and memory bars that may be placed on the vehicles. The region of interest may be an image region in the second area image that may contain memory bar information that is screened out after image processing.

[0143] In the region of interest of the second area image, the area where the barcode on each carrier is located can be extracted through image processing algorithms such as color threshold segmentation and dilation erosion, and optical character recognition (OCR) or a special barcode recognition algorithm can be performed on the barcode in the ROI to obtain the information carried by the barcode, which is used here to identify the existence and type of the memory bar, so as to determine whether there is a memory bar to be installed. When it is determined that there is a memory bar to be installed, the carrier on which the memory bar to be installed is placed is determined as the target carrier.

[0144] In one example, the robot moves to a working area containing carriers, adjusts the shooting angle and distance of the first visual sensor to ensure that all carriers can be fully captured, and the image obtained by shooting is the second area image. The second area image is preprocessed, such as grayscale, filtering and denoising, to enhance image contrast and reduce interference. Image processing techniques such as color threshold segmentation, dilation, and corrosion are used to identify the barcode area in the image. Barcode recognition uses a barcode recognition algorithm in a determined ROI to optically recognize the barcode and obtain the barcode information. When the barcode information contains the identification information of the memory stick to be installed, the carrier is determined to be the "target carrier" containing the memory stick to be installed.

[0145] Through this embodiment, through barcode recognition, it is possible to quickly and accurately determine which carrier carries the memory stick to be installed, thereby avoiding the detection and processing of irrelevant carriers and improving the overall recognition efficiency.

[0146] In an exemplary embodiment, the end of the robotic arm includes a clamp; installing the memory stick to be installed into a designated memory slot corresponding to the memory stick to be installed includes: obtaining force information of the clamp, wherein the force information is used to indicate the contact force between the clamp and the memory stick to be installed and the torque of the clamp grasping the memory stick to be installed; adjusting the posture of the clamp according to the force information, and installing the memory stick to be installed into the designated memory slot through the clamp when a first preset condition is met, wherein the first preset condition is that the value of the contact force is greater than or equal to a first preset contact force threshold, the value of the contact force is less than a second preset contact force threshold, and the value of the torque is less than a first preset torque threshold.

[0147] It should be noted that the end of the robotic arm may include a gripper, which may refer to a tool at the end of the robotic arm for grasping and manipulating objects, and may generally have two or more mechanical fingers, which may be used to accurately grasp the memory stick to be installed. The force information may refer to the information of the force and torque generated when contacting the memory stick to be installed during the installation of the memory stick to be installed. The contact force may be the pressure or thrust between the gripper and the memory stick to be installed, and the torque may refer to any force that causes the memory stick to be installed to rotate.

[0148] Optionally, the first preset contact force threshold may be a preset minimum contact force threshold to ensure that the clamp can firmly grasp the memory stick to be installed, thereby preventing the memory stick from slipping or falling out of the clamp during the installation process. The second preset contact force threshold may be a preset maximum contact force threshold to prevent the clamp from applying excessive force to the memory stick to be installed, thereby preventing damage to the contacts of the memory stick to be installed or the slot of the device to be installed. The first preset torque threshold may be a preset maximum torque threshold, which is used to limit the influence of the clamp on the rotation of the memory stick to be installed during the installation process, thereby preventing the memory stick to be installed from being damaged due to excessive torque when inserted into the slot.

[0149] Optionally, if a second preset condition is met, stop installing the memory stick to be installed into the designated memory slot, wherein the second preset condition is that the value of the contact force is greater than or equal to a second preset contact force threshold, or the value of the torque is greater than a first preset torque threshold.

[0150] Optionally, the robot can grasp the memory stick by using different types of grippers (such as soft or rigid) to accommodate memory sticks of different sizes or shapes. The choice of grippers is based on the size and material of the memory stick to avoid damaging the memory stick during the grasping process.

[0151] Through this embodiment, by real-time monitoring of the force information of the clamping claw, it is possible to prevent and avoid the memory stick to be installed from slipping, being damaged, or causing damage to the device to be installed during the installation process, thereby improving the safety of the installation process. By detecting the position information of the memory stick and the slot, as well as the feedback of the force information, it is possible to ensure that the memory stick is installed in the designated slot with the correct posture and position, thereby improving the accuracy and success rate of assembly. At the same time, during the installation process, the force information feedback mechanism of the clamping claw allows the robot arm to dynamically adjust its posture to adapt to the slight deviation that may exist between the memory stick and the slot, thereby improving the robustness and adaptability of the system.

[0152] It should be noted that the memory assembly method in the above embodiments is not limited to the assembly of memory bars, but is also applicable to the precise positioning and assembly of components of PCIe (Peripheral Component Interconnect Express) slots. Through the above description of the implementation methods, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method.

[0153] The embodiment of the present application also provides a memory assembly system, Figure 5 A structural block diagram of an optional memory assembly system provided according to an embodiment of the present application is shown in FIG. Figure 5As shown in , the memory assembly system includes a control device 502 , a robot 504 and a first visual sensor 506 , and the robot 504 is configured with a mechanical arm 5042 .

[0154] The control device 502 is used to control the mechanical arm to grab the memory bar to be installed in the target carrier when identifying the target carrier where the memory bar to be installed is located, and control the first visual sensor to obtain a captured image of the memory bar, wherein the captured image of the memory bar is an image when the mechanical arm grabs the memory bar to be installed; construct an image pyramid of the captured image of the memory bar, wherein the image pyramid includes a group of down-sampled images obtained by performing a group of down-sampling operations of different granularities on the captured image of the memory bar; perform a feature extraction operation on the image pyramid to obtain feature point information of the memory bar to be installed; determine the actual posture of the memory bar to be installed according to the feature point information of the memory bar to be installed; adjust the end posture of the mechanical arm based on the actual posture and the preset memory bar posture to adjust the posture of the memory bar to be installed; control the mechanical arm after the end posture adjustment to move, and install the memory bar to be installed into the designated memory slot corresponding to the memory bar to be installed.

[0155] The robot arm 5042 is used to grab the memory stick to be installed in the target carrier, adjust the end posture of the robot arm, move and install the memory stick to be installed into the designated memory slot in response to the control of the control device.

[0156] The first visual sensor 506 is used to obtain the memory bar captured image in response to the control of the control device.

[0157] It should be noted that the control device may be built into the robot or may not be located in the robot.

[0158] Through the embodiments provided by the present application, when a target carrier where a memory stick to be installed is located is identified, the memory stick to be installed in the target carrier is grabbed by a robotic arm, and a grabbed image of the memory stick is obtained by a first visual sensor, and an image pyramid of the grabbed image of the memory stick is constructed, allowing feature detection at different image resolutions, thereby improving the accuracy of feature extraction operations, thereby obtaining more accurate feature point information of the memory stick to be installed, and determining the actual posture of the memory stick. According to the actual posture of the memory stick to be installed and the preset posture of the memory stick, the end posture of the robotic arm is adjusted, so that the robotic arm can dynamically correct the posture of the memory stick to be installed, thereby solving the problem of low precision of memory assembly caused by posture deviation that may occur in the memory stick after grabbing in the related art, improving the accuracy and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving production efficiency.

[0159] In an exemplary embodiment, Figure 6 As shown, Figure 6 A structural block diagram of another optional memory assembly system provided for an embodiment of the present application; the robot 504 is configured with a second visual sensor 5044; the control device 502 is also used to control the second visual sensor to obtain a first area image, wherein the first area image is an image of the area where the memory bar to be installed is located in the target carrier; based on the first area image, the relative position of the memory bar to be installed and the robotic arm and the relative rotation angle between the memory bar to be installed and the robotic arm are identified; based on the relative position and the relative rotation angle, the posture of the robotic arm is adjusted, and the robotic arm with the adjusted posture is controlled to grab the memory bar to be installed.

[0160] The second visual sensor 5044 is used to acquire the first area image in response to the control of the control device.

[0161] In an exemplary embodiment, the control device 502 is also used to determine target area information, wherein the target area information is used to indicate the area of ​​the memory bar to be installed in the target carrier; according to the target area information, the second visual sensor is controlled to photograph the memory bar to be installed to obtain a first area image.

[0162] The second visual sensor 5044 is used to acquire the first area image in response to the control of the control device.

[0163] In an exemplary embodiment, the control device 502 is also used to perform edge detection on the first area image to obtain line segment information of the first area image, wherein the line segment information of the first area image is used to indicate the edge of the memory bar to be installed; based on the line segment information of the first area image and the posture information of the robotic arm, determine the relative position of the memory bar to be installed and the robotic arm and the relative rotation angle between the memory bar to be installed and the robotic arm.

[0164] In an exemplary embodiment, the control device 502 is also used to perform edge detection on the downsampled images in the image pyramid to obtain a set of edge maps; construct an image based on the pixel values ​​of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image; perform contour recognition on the first intermediate image, extract and filter the contours in the first intermediate image to obtain a second intermediate image, wherein the second intermediate image is an area image within the contours in the first intermediate image; identify feature points in the second intermediate image to obtain pixel coordinate information of the feature points in the second intermediate image, wherein the pixel coordinate information of the feature points in the second intermediate image is feature point information of the memory stick to be installed.

[0165] In an exemplary embodiment, the control device 502 is also used to traverse an edge map in a group of edge maps, and obtain pixel values ​​of pixel points of an edge map in a group of edge maps; use the pixel maximum value of pixel points at the same position in a group of edge maps as the pixel value of the pixel point at the corresponding position in the image to be constructed, perform image construction, and obtain a first intermediate image.

[0166] In an exemplary embodiment, the control device 502 is further used to perform edge detection on the second intermediate image to obtain an edge image corresponding to the second intermediate image; perform Radon transform on the edge image corresponding to the second intermediate image to obtain a Radon transform result; perform peak detection on the Radon transform result to obtain straight line parameters in the second intermediate image; determine feature points in the second intermediate image based on the straight line parameters in the second intermediate image, and calculate pixel coordinate information of the feature points in the second intermediate image.

[0167] In an exemplary embodiment, the feature point information of the memory bar to be installed is pixel coordinate information of the feature point of the memory bar to be installed;

[0168] The control device 502 is further used to map the pixel coordinate information of the feature points of the memory bar to be installed according to the specification information of the memory bar to be installed and the world coordinate system to obtain the three-dimensional coordinate information of the feature points of the memory bar to be installed; calculate the parameter information of the memory bar to be installed according to the camera coordinate system of the first visual sensor, the three-dimensional coordinate information of the feature points of the memory bar to be installed, and the pixel coordinate information of the feature points of the memory bar to be installed, wherein the parameter information of the memory bar to be installed includes the rotation parameters of the memory bar to be installed relative to the first visual sensor and the translation parameters of the memory bar to be installed relative to the first visual sensor; calculate the actual posture of the memory bar to be installed according to the parameter information of the memory bar to be installed.

[0169] In an exemplary embodiment, before installing the memory module to be installed into the designated memory slot corresponding to the memory module to be installed by moving the robot arm after the posture adjustment of the terminal;

[0170] The control device 502 is also used to obtain a device image, wherein the device image is an image of a target device into which the memory stick to be installed is to be installed; perform slot feature extraction on the device image to obtain slot features; determine the position information of the specified memory slot based on the slot features, and calculate the slot position of the specified memory slot based on the position information of the specified memory slot, wherein controlling the robotic arm after the posture adjustment of the mobile terminal is performed according to the slot position of the specified memory slot.

[0171] In an exemplary embodiment, the control device 502 is further used to control the first visual sensor to collect an image of the detection area to obtain a second area image, wherein the detection area is an area where a set carrier is located; extract an area of ​​interest in the second area image, and perform barcode recognition on the extracted area of ​​interest to obtain a barcode recognition result; when the barcode recognition result indicates that a memory bar to be installed is recognized, determine the carrier on which the memory bar to be installed is placed as a target carrier;

[0172] The first visual sensor 506 is used to collect images of the detection area in response to the control of the control device to obtain a second area image.

[0173] In an exemplary embodiment, the first visual sensor 506 is further configured to collect images of the detection area in response to the control of the control device to obtain a second area image.

[0174] In an exemplary embodiment, the end of the robotic arm 5042 includes a gripper; the control device 502 is also used to obtain force information of the gripper, wherein the force information is used to indicate the contact force between the gripper and the memory stick to be installed and the torque of the gripper grasping the memory stick to be installed; the posture of the gripper is adjusted according to the force information, and the gripper is controlled to install the memory stick to be installed into a designated memory slot when a first preset condition is met, wherein the first preset condition is that the value of the contact force is greater than or equal to a first preset contact force threshold, the value of the contact force is less than a second preset contact force threshold, and the value of the torque is less than a first preset torque threshold; the gripper is used to install the memory stick to be installed into the designated memory slot in response to the control of the control device 502.

[0175] The description of the features in the embodiment corresponding to the memory assembly system can refer to the relevant description of the embodiment corresponding to the memory assembly method, which will not be repeated here.

[0176] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above-mentioned memory assembly method embodiments.

[0177] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned memory assembly method embodiments when running.

[0178] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0179] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned memory assembly method embodiments are implemented.

[0180] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned memory assembly method embodiments are implemented.

[0181] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0182] The above is a detailed introduction to a memory assembly method, system, electronic device, storage medium and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A memory assembly method, characterized in that: Applied to a memory assembly system, the memory assembly system includes a robot and a first visual sensor, the robot is equipped with a mechanical arm, and the method includes: When the target carrier where the memory bar to be installed is located is identified, the memory bar to be installed in the target carrier is grabbed by the mechanical arm, and a memory bar grabbing image is acquired by the first visual sensor, wherein the memory bar grabbing image is an image when the mechanical arm grabs the memory bar to be installed; constructing an image pyramid of the memory bar captured image, wherein the image pyramid includes a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the memory bar captured image; Performing a feature extraction operation on the image pyramid to obtain feature point information of the memory stick to be installed; Determining the actual posture of the memory bar to be installed according to the feature point information of the memory bar to be installed; Based on the actual posture and the preset posture of the memory stick, adjusting the posture of the end of the robot arm to adjust the posture of the memory stick to be installed; The memory bar to be installed is installed into a designated memory slot corresponding to the memory bar to be installed by the mechanical arm after the end posture is adjusted.

2. The method according to claim 1, characterized in that The robot is equipped with a second visual sensor; the method of grabbing the memory bar to be installed in the target carrier by the robot arm includes: Acquire a first area image by using the second visual sensor, wherein the first area image is an image of an area in the target carrier where the memory bar to be installed is located; According to the first area image, identifying the relative position of the memory bar to be installed and the mechanical arm and the relative rotation angle of the memory bar to be installed and the mechanical arm; The posture of the mechanical arm is adjusted according to the relative position and the relative rotation angle, and the mechanical arm with the adjusted posture is used to grab the memory stick to be installed.

3. The method according to claim 2, characterized in that The acquiring the first area image by the second visual sensor comprises: Determine target area information, wherein the target area information is used to indicate an area in the target carrier where the memory stick to be installed is located; According to the target area information, the memory bar to be installed is photographed by the second visual sensor to obtain the first area image.

4. The method according to claim 2, characterized in that: The step of identifying the relative position of the memory bar to be installed and the mechanical arm and the relative rotation angle of the memory bar to be installed and the mechanical arm according to the first area image includes: Performing edge detection on the first region image to obtain line segment information of the first region image, wherein the line segment information of the first region image is used to indicate an edge of the memory bar to be installed; The relative position of the memory bar to be installed and the mechanical arm and the relative rotation angle of the memory bar to be installed and the mechanical arm are determined according to the line segment information of the first area image and the position information of the mechanical arm.

5. The method according to claim 1, characterized in that: The performing a feature extraction operation on the image pyramid to obtain feature point information of the memory stick to be installed includes: Performing edge detection on the downsampled images in the image pyramid to obtain a set of edge maps; Performing image construction based on pixel values ​​of pixel points of the edge images in the set of edge images to obtain a first intermediate image; Performing contour recognition on the first intermediate image, extracting and screening the contour in the first intermediate image, and obtaining a second intermediate image, wherein the second intermediate image is an image of a region within the contour in the first intermediate image; Identify feature points in the second intermediate image and obtain pixel coordinate information of the feature points in the second intermediate image, wherein the pixel coordinate information of the feature points in the second intermediate image is feature point information of the memory stick to be installed.

6. The method according to claim 5, characterized in that The step of constructing an image based on the pixel values ​​of the pixel points of the edge images in the set of edge images to obtain a first intermediate image comprises: Traversing edge maps in the set of edge maps, and obtaining pixel values ​​of pixel points of the edge maps in the set of edge maps; The maximum pixel value of the pixel points at the same position in the set of edge images is used as the pixel value of the pixel points at the corresponding position in the image to be constructed, and the image is constructed to obtain the first intermediate image.

7. The method according to claim 5, characterized in that The identifying feature points in the second intermediate image and obtaining pixel coordinate information of the feature points in the second intermediate image includes: performing edge detection on the second intermediate image to obtain an edge image corresponding to the second intermediate image; performing a Radon transform on an edge image corresponding to the second intermediate image to obtain a Radon transform result; Performing peak detection on the Radon transform result to obtain straight line parameters in the second intermediate image; According to the straight line parameters in the second intermediate image, feature points in the second intermediate image are determined, and pixel coordinate information of the feature points in the second intermediate image is calculated.

8. The method according to claim 1, characterized in that The characteristic point information of the memory bar to be installed is the pixel coordinate information of the characteristic point of the memory bar to be installed; The determining, according to the feature point information of the memory bar to be installed, the actual posture of the memory bar to be installed includes: According to the specification information of the memory bar to be installed and the world coordinate system, mapping the pixel coordinate information of the feature point of the memory bar to be installed to obtain the three-dimensional coordinate information of the feature point of the memory bar to be installed; Calculating parameter information of the memory bar to be installed according to the camera coordinate system of the first visual sensor, the three-dimensional coordinate information of the feature points of the memory bar to be installed, and the pixel coordinate information of the feature points of the memory bar to be installed, wherein the parameter information of the memory bar to be installed includes a rotation parameter of the memory bar to be installed relative to the first visual sensor and a translation parameter of the memory bar to be installed relative to the first visual sensor; The actual posture of the memory bar to be installed is calculated according to the parameter information of the memory bar to be installed.

9. The method according to claim 1, characterized in that: Before the mechanical arm, after adjusting the posture of the mobile terminal, installs the memory bar to be installed into the designated memory slot corresponding to the memory bar to be installed, the method further includes: Acquire a device image, wherein the device image is an image of a target device where the memory module to be installed is to be installed; Extracting slot features from the device image to obtain slot features; Based on the slot feature, the position information of the designated memory slot is determined, and based on the position information of the designated memory slot, the slot position of the designated memory slot is calculated, wherein the movement of the robotic arm after the posture adjustment of the mobile terminal is performed according to the slot position of the designated memory slot.

10. The method according to claim 1, characterized in that The method further comprises: Capturing an image of a detection area through the first visual sensor to obtain a second area image, wherein the detection area is an area where a set vehicle is located; Extracting a region of interest in the second region image, and performing barcode recognition on the extracted region of interest to obtain a barcode recognition result; In a case where the barcode recognition result indicates that the memory bar to be installed is recognized, a carrier on which the memory bar to be installed is placed is determined as the target carrier.

11. The method according to any one of claims 1 to 10, characterized in that The end of the mechanical arm includes a clamping claw; the step of installing the memory bar to be installed into a designated memory slot corresponding to the memory bar to be installed includes: Acquiring force information of the clamping claw, wherein the force information is used to indicate the contact force between the clamping claw and the memory bar to be installed and the torque of the clamping claw grasping the memory bar to be installed; According to the force information, the posture of the clamp is adjusted, and when a first preset condition is met, the memory stick to be installed is installed into the designated memory slot through the clamp, wherein the first preset condition is that the value of the contact force is greater than or equal to a first preset contact force threshold, the value of the contact force is less than a second preset contact force threshold, and the value of the torque is less than the first preset torque threshold.

12. A memory assembly system, characterized in that: It includes a control device, a robot and a first visual sensor, wherein the robot is equipped with a mechanical arm; wherein, The control device is used to control the mechanical arm to grab the memory bar to be installed in the target carrier when identifying the target carrier where the memory bar to be installed is located, and control the first visual sensor to obtain a captured image of the memory bar, wherein the captured image of the memory bar is an image when the mechanical arm grabs the memory bar to be installed; construct an image pyramid of the captured image of the memory bar, wherein the image pyramid includes a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the captured image of the memory bar; perform a feature extraction operation on the image pyramid to obtain feature point information of the memory bar to be installed; determine the actual posture of the memory bar to be installed according to the feature point information of the memory bar to be installed; adjust the end posture of the mechanical arm based on the actual posture and the preset memory bar posture to adjust the posture of the memory bar to be installed; control the mechanical arm after the end posture is adjusted to install the memory bar to be installed into the designated memory slot corresponding to the memory bar to be installed; The robotic arm is used to, in response to the control of the control device, grab the memory bar to be installed in the target carrier, adjust the end posture of the robotic arm, and install the memory bar to be installed into the designated memory slot; The first visual sensor is used to obtain the captured image of the memory bar in response to the control of the control device.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the memory assembly method as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the memory assembly method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the memory assembly method according to any one of claims 1 to 11 are implemented.

Images