Memory assembly method, system, electronic device, storage medium and product

Abstract

The present application discloses a memory assembly method, system, electronic device, storage medium, and product, which relates to the field of server manufacturing. It includes: when the target carrier where the memory module to be installed is recognized, the robotic arm grabs the memory module to be installed in the target carrier, acquires the image of the grabbed memory module, constructs an image pyramid of the grabbed memory module image, performs a feature extraction operation on the image pyramid, obtains the feature point information of the memory module to be installed, so as to determine the actual posture of the memory module, and adjusts the end posture of the robotic arm according to the actual posture of the memory module to be installed and the preset memory module posture, enabling the robotic arm to dynamically correct the posture of the memory module to be installed, solving the problem of relatively low precision of memory assembly caused by possible posture deviation of the memory module after grabbing in the related art, improving the precision and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving the production efficiency.

Description

Technical Field

[0001] This application relates to the field of server manufacturing, and particularly to a memory assembly method, system, electronic device, storage medium, and product. Background Art

[0002] During the memory assembly process in the related art, the positioning and grasping of memory modules often rely on fixed mechanical structures and perspectives. This limits the flexibility and compatibility of the assembly. Especially when facing the diverse layouts of different server motherboards and the slight deflection of memory modules, the adaptability to the assembly environment is limited. In particular, after the memory module is grasped, it may deflect due to factors such as the slight vibration of the robotic arm, resulting in a relatively low precision problem in memory assembly in the related art. Summary of the Invention

[0003] This application provides a memory assembly method, system, electronic device, storage medium, and product to at least solve the problem of relatively low precision in memory assembly in the related art.

[0004] This application provides a memory assembly method applied to a memory assembly system. The memory assembly system includes a robot and a first vision sensor, and the robot is configured with a robotic arm. The method includes: when identifying a target carrier where a memory module to be installed is located, grasping the memory module to be installed in the target carrier through the robotic arm, and acquiring a memory module grasping image through the first vision sensor, where the memory module grasping image is an image when the robotic arm grasps the memory module to be installed; constructing an image pyramid of the memory module grasping image, where the image pyramid includes a set of downsampled images obtained by performing a set of downsampling operations with different granularities on the memory module grasping image; performing a feature extraction operation on the image pyramid to obtain feature point information of the memory module to be installed; determining the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed; adjusting the end pose of the robotic arm based on the actual pose and a preset memory module pose to adjust the pose of the memory module to be installed; moving the robotic arm with the adjusted end pose and installing the memory module to be installed into a specified memory slot corresponding to the memory module to be installed.

[0005] The present application also provides a memory assembly system, including a control device, a robot, and a first vision sensor. The robot is configured with a robotic arm. Among them, the control device is used to control the robotic arm to grab the memory module to be installed in the target carrier when the target carrier where the memory module to be installed is recognized, and control the first vision sensor to obtain an image of the grabbed memory module, where the image of the grabbed memory module is an image when the robotic arm grasps the memory module to be installed; construct an image pyramid of the image of the grabbed memory module, where the image pyramid includes a set of downsampled images obtained by performing a set of downsampling operations with different granularities on the image of the grabbed memory module; perform a feature extraction operation on the image pyramid to obtain feature point information of the memory module to be installed; determine the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed; based on the actual pose and a preset memory module pose, adjust the end pose of the robotic arm to adjust the pose of the memory module to be installed; control the robotic arm with the adjusted end pose to install the memory module to be installed into the specified memory slot corresponding to the memory module to be installed; the robotic arm is used to respond to the control of the control device, grab the memory module to be installed in the target carrier, adjust the end pose of the robotic arm, and install the memory module to be installed into the specified memory slot; the first vision sensor is used to respond to the control of the control device and obtain the image of the grabbed memory module.

[0006] The present application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above memory assembly methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above memory assembly methods are implemented.

[0008] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any of the above memory assembly methods are implemented.

[0009] Through this application, when the target vehicle where the memory module to be installed is recognized, the robotic arm is used to grasp the memory module to be installed in the target vehicle, and the first vision sensor is used to obtain the image of the grasped memory module. By constructing the image pyramid of the image of the grasped memory module, feature detection is allowed at different image resolutions, thereby improving the accuracy of the feature extraction operation, and thus obtaining more accurate feature point information of the memory module to be installed to determine the actual posture of the memory module. According to the actual posture of the memory module to be installed and the preset posture of the memory module, the end posture of the robotic arm is adjusted so that the robotic arm can dynamically correct the posture of the memory module to be installed, solving the problem in the related art that the accuracy of memory assembly is relatively low due to the possible posture deviation after the memory module is grasped, improving the accuracy and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving the production efficiency. Description of the Drawings

[0010] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0011] Figure 1 It is a schematic flowchart of an optional memory assembly method provided according to an embodiment of the present application.

[0012] Figure 2 It is a schematic diagram of an optional memory assembly method provided according to an embodiment of the present application.

[0013] Figure 3 It is a schematic diagram of the feature points of an optional memory module provided according to an embodiment of the present application.

[0014] Figure 4 It is a schematic diagram of an optional memory assembly provided according to an embodiment of the present application.

[0015] Figure 5 It is a block diagram of the structure of an optional memory assembly system provided according to an embodiment of the present application.

[0016] Figure 6 It is a block diagram of the structure of another optional memory assembly system provided according to an embodiment of the present application. Detailed Embodiments

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0018] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0019] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In the related art, during the memory assembly process, the positioning and grasping of memory modules often rely on fixed mechanical structures and perspectives. This limits the flexibility and compatibility of the assembly. Especially when facing the diverse layouts of different server motherboards and the slight deflection of memory modules, the adaptability to the assembly environment is limited. In particular, the deflection caused by factors such as the slight vibration of the robotic arm after the memory module is grasped may lead to the problem of relatively low precision in memory assembly in the related art. To solve the above problems, the embodiments of the present application provide a memory assembly method. When the target carrier where the memory module to be installed is recognized, the robotic arm is used to grasp the memory module to be installed in the target carrier, an image pyramid of the memory module grasping image is constructed, and a feature extraction operation is performed on the image pyramid to obtain the feature point information of the memory module to be installed, so as to determine the actual pose of the memory module, and then adjust the end pose of the robotic arm to enable the robotic arm to dynamically correct the pose of the memory module to be installed, solving the problem of relatively low precision in memory assembly caused by the possible pose deviation of the memory module after grasping in the related art, and improving the precision and reliability of memory assembly.

[0021] The embodiments of the present application provide a memory assembly method. Taking the memory assembly method in this embodiment to be executed by a control device as an example, it is applied to a memory assembly system. The memory assembly system includes a robot and a first vision sensor, and the robot is configured with a robotic arm. Figure 1 As a schematic flowchart of an optional memory assembly method provided by the embodiments of the present application, as Figure 1 shown, the process of this method may include steps S102 to step S112.

[0022] Step S102, when the target vehicle where the memory module to be installed is recognized, use the robotic arm to grasp the memory module to be installed in the target vehicle, and obtain a memory module grasping image through the first vision sensor, where the memory module grasping image is the image when the robotic arm grasps the memory module to be installed.

[0023] It should be noted that the memory assembly system may include a robot and a first vision sensor, and the robot may be configured with a robotic arm. Among them, the robot can be a composite robot, and a composite robot can refer to an advanced robot system that combines multiple technologies and functions, usually including but not limited to: a multi-degree-of-freedom robotic arm, various types or modes of end effectors (such as grippers, suction cups, etc.), a vision sensor (such as a monocular camera, a stereo camera or a depth camera) highly integrated control system. The composite robot can, through the robotic arm and an intelligent sensing system, autonomously perform precise grasping, operation and assembly tasks in a complex environment, and has a high degree of autonomy, adaptability and operation accuracy.

[0024] The memory module to be installed refers to a memory component that has not been installed in the memory slot of the server motherboard or other electronic devices. The memory module to be installed usually consists of a circuit board, memory chips (storage chips), gold fingers (metal pieces used to contact the motherboard slot), etc., and has standard specifications and dimensions. The target vehicle can refer to a container or platform that carries the memory module to be installed, and plays a role in positioning and transporting the memory module in the memory assembly process, such as a Tray (tray).

[0025] The robot can move to the workbench where multiple vehicles are placed. After recognizing the target vehicle where the memory module to be installed is located, the robot can move to the vicinity of the workbench where the target vehicle is located, can accurately move the robotic arm above the area where the target vehicle is located, and use the robotic arm to grasp the memory module to be installed. After grasping the memory module to be installed, the robotic arm can be photographed through the first vision sensor to obtain a memory module grasping image, that is, the image when the robotic arm of the composite robot grasps the memory module to be installed. Optionally, the first vision sensor can be an external vision sensor, and various vision sensor technologies such as stereo vision, infrared or laser scanning can be used to obtain the memory module grasping image. The use of the vision sensor is not limited to a monocular camera, and can also be an infrared camera or a structured light scanning system to adapt to different lighting conditions or obtain more accurate depth information.

[0026] It should be noted that after grasping the memory module to be installed, there will be a certain deflection between the end of the robotic arm and the memory module to be installed, such as Figure 2As shown, this deflection will seriously affect the success rate of assembly to a certain extent. If the deflection is not taken into account, it may lead to installation failure at the least and damage to components or even the mainboard at the worst. Therefore, it is necessary to adjust the posture of the end of the robot arm, that is, the assembly angle of the gripper, so as to improve the success rate and reliability of assembly.

[0027] Step S104 : constructing an image pyramid of the memory bar captured image, wherein the image pyramid includes a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the memory bar captured image.

[0028] It should be noted that when constructing the image pyramid of the memory bar captured image, the memory bar captured image can be first filtered, and optionally, the memory bar captured image is filtered using a preset filtering algorithm to obtain the filtered memory bar captured image, wherein the preset filtering algorithm may include at least one of the following: bilateral filtering, median filtering, and Gaussian filtering. The purpose of filtering is to remove noise, smooth the image, and provide a better basis for subsequent feature point extraction and edge detection.

[0029] The image pyramid may include a group of downsampled images obtained by performing a group of downsampling operations of different granularities on the filtered memory bar captured image. Specifically, the filtered memory bar captured image is subjected to multiple downsampling operations to generate downsampled images of different granularities, thereby obtaining an image pyramid of the memory bar captured image.

[0030] Specifically, the downsampling operation can be used to reduce the resolution of the image to achieve feature point detection at different scales. Optionally, the downsampling operation on the filtered memory bar captured image may include: merging continuous pixel blocks in the filtered memory bar captured image into a new pixel, and the new pixel can be determined by the average value, median value or maximum value of the original pixel block. Specifically, the filtered memory bar captured image is divided into pixel blocks of equal size, and the statistical value (such as the average value, median value or maximum value) of each pixel block is calculated as the value of the new pixel, and each new pixel is copied to the original pixel position to form a downsampled image.

[0031] Of course, in the process of constructing the image pyramid, a Gaussian pyramid can be used, by performing Gaussian blur on the memory bar captured image after filtering, and then performing pixel merging processing. Specifically, the memory bar captured image is processed by a Gaussian filter, and then the Gaussian blurred image is downsampled using pixel merging.

[0032] Of course, the downsampling operation may also use bilinear interpolation, bicubic interpolation or wavelet transform downsampling, which is not limited in this application.

[0033] The filtered memory module capture image constructed through the downsampling operation can reduce the computational complexity of subsequent image processing (such as edge detection, etc.). Moreover, by constructing image pyramids with different resolutions, feature detection can be performed at different scales, which can improve the robustness and accuracy of feature detection.

[0034] Step S106: Perform a feature extraction operation on the image pyramid to obtain the feature point information of the memory module to be installed.

[0035] It should be noted that the feature points of the memory module to be installed refer to the points with specific meanings in the memory module capture image. For example, the edges and corner points of the memory module to be installed in the memory module capture image. The feature point information of the memory module to be installed is obtained by extracting feature points from the memory module capture image and is used to determine the actual pose of the memory module to be installed.

[0036] Optionally, during the feature extraction operation (i.e., feature point extraction), image processing techniques can be used. For example, SIFT (Scale-Invariant Feature Transform) or SURF (Speeded-Up Robust Features) can be used to identify the key points in the image, such as the edges and corner points of the memory module to be installed.

[0037] In specific practice, to improve the accuracy of actual pose determination, feature point extraction can also consider the standard size and shape features of the memory module to be installed. For example, memory modules usually have a standard aspect ratio and a specific edge profile.

[0038] Optionally, the number of feature points of the memory module to be installed in the selected memory module capture image can be determined according to the size and shape features of the memory module to be installed. For example, when the size of the memory module is larger or its shape is more complex, more feature points may need to be selected to obtain a more accurate pose.

[0039] Step S108: Determine the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed.

[0040] It should be noted that the actual pose of the memory module to be installed can refer to the actual position and orientation of the memory module to be installed in space. Optionally, the actual pose of the memory module to be installed can refer to the position and rotation angle of the memory module to be installed relative to the first vision sensor.

[0041] Optionally, the memory module to be installed has a corresponding standard size. By using the feature point information and the standard size of the memory module to be installed, the actual posture of the memory module to be installed can be determined. Specifically, when determining the actual posture of the memory module, RANSAC (Random Sample Consensus) can be used to remove outliers in the feature points and improve the robustness of posture estimation. In addition, to adapt to different types of memory modules, it may be necessary to store the standard sizes and shape information of multiple memory modules to ensure the accuracy of posture estimation.

[0042] Step S110: Based on the actual posture and the preset posture of the memory module, adjust the end posture of the robotic arm to adjust the posture of the memory module to be installed.

[0043] It should be noted that since there are multiple different types and specifications of memory modules during actual memory module assembly. Therefore, a series of preset postures of the memory module can be stored in advance. The preset posture corresponding to the memory module to be installed can be found, and based on the deviation between the actual posture and the preset posture of the memory module, the adjustment posture of the end of the robotic arm can be calculated to adjust the end posture of the robotic arm. The preset posture corresponding to the memory module to be installed can be the preset standard posture corresponding to the memory module to be installed.

[0044] Among them, the adjustment posture of the end of the robotic arm can include the rotation and translation adjustment amounts that the end of the robotic arm needs to perform to ensure that the memory module to be installed can be aligned with the corresponding memory slot of the memory module to be installed in the correct posture.

[0045] Optionally, during the adjustment process of the end posture of the robotic arm, force feedback control can be set to ensure that the memory module is not damaged while adjusting the posture.

[0046] Step S112: Install the memory module to be installed into the specified memory slot corresponding to the memory module to be installed through the robotic arm with the adjusted end posture.

[0047] It should be noted that the specified memory slot can refer to the specific slot position on the device to be installed where the memory module is to be installed. The device to be installed can be a server motherboard.

[0048] Optionally, in the adjusted posture, the robotic arm moves to move the memory module to be installed above the specified memory slot and install it. During the installation process, force feedback control is used to ensure the safety threshold of the force during installation. The safety threshold of the force can be determined based on multiple factors, such as the physical characteristics of the memory module and the server motherboard (including the hardness and thickness of the memory module and the strength and toughness of the server motherboard material), the design of the memory slot, etc.

[0049] In specific practice, during the installation stage of the memory module to be installed, the robot may adopt a progressive force control strategy, that is, first gently contact the memory slot, and then adjust according to the force feedback information obtained in real time and the preset safety threshold to ensure that the memory module to be installed is safely inserted without damaging the motherboard.

[0050] Through the embodiments provided by the present application, when the target vehicle where the memory module to be installed is recognized, the memory module to be installed in the target vehicle is grasped by the robotic arm, and the grasping image of the memory module is obtained through the first vision sensor, and an image pyramid of the grasping image of the memory module is constructed, allowing feature detection at different image resolutions, thereby improving the accuracy of the feature extraction operation, and thus obtaining more accurate feature point information of the memory module to be installed to determine the actual pose of the memory module. According to the actual pose of the memory module to be installed and the preset pose of the memory module, the end pose of the robotic arm is adjusted, so that the robotic arm can dynamically correct the pose of the memory module to be installed, solving the problem in the related art that the accuracy of memory assembly is relatively low due to the possible pose deviation of the memory module after grasping, improving the accuracy and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving the production efficiency.

[0051] In an exemplary embodiment, the robot is configured with a second vision sensor; grasping the memory module to be installed in the target vehicle by the robotic arm includes: obtaining a first area image through the second vision sensor, where the first area image is an image of the area where the memory module to be installed is located in the target vehicle; identifying the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the first area image; adjusting the pose of the robotic arm according to the relative position and the relative rotation angle, and grasping the memory module to be installed with the robotic arm after the pose adjustment.

[0052] It should be noted that the robot can be configured with a second vision sensor. The second vision sensor in the robot can refer to an internal vision sensor installed at the end effector of the robot, which can be used to capture images when the robotic arm needs to grasp the memory module to be installed. Compared with the first vision sensor, the second vision sensor can observe the target closer and provide more detailed visual data. The first area image can be an image taken by the second vision sensor of the robot of the area of the target vehicle (Tray or other carrier platform) where the memory module to be installed is located before grasping the memory module to be installed. The first area image can include the environmental background of the memory module to be installed and the specific position of the memory module to be installed in the Tray.

[0053] The relative position between the memory module to be installed and the robotic arm may refer to the spatial position relationship between the memory module to be installed and the end effector of the robotic arm, that is, the distances between the two on the x, y, and z axes. The relative rotation angle between the memory module to be installed and the robotic arm may refer to the spatial rotation angle relationship between the memory module to be installed and the end effector of the robotic arm, that is, the rotation angles of the two around the x, y, and z axes.

[0054] Optionally, during the process of grasping the memory module to be installed by the robotic arm, the second vision sensor on the robot is used to capture the memory module to be installed in the target vehicle to obtain a first region image. By performing image processing on the first region image, for example, color threshold segmentation, template matching processing, etc., based on the processed first region image, the relative position and relative rotation angle between the memory module to be installed and the end effector of the robotic arm are calculated, and the posture of the robotic arm is adjusted according to the relative position and relative rotation angle, so as to use the adjusted robotic arm to grasp the memory module to be installed.

[0055] Optionally, when the image processing adopts template matching, the first region image is preprocessed. The preprocessing steps may include image processing techniques such as grayscale conversion, edge detection, smoothing (such as Gaussian filtering, median filtering), and binarization. The preprocessed first region image is subjected to template matching with the template image, such as correlation matching, normalized cross-correlation (NCC) or fast template matching algorithms (such as feature-based matching algorithms), to search for the region most similar to the template image in the preprocessed first region image to obtain the position information of the memory module to be installed in the first region image, that is, the coordinates of the center of the memory module or a certain feature point in the image coordinate system. By comparing the similarity or correlation between the template image and the first region image (mainly the long side and short side directions of the memory module), the relative rotation angle between the memory module to be installed and the robotic arm is calculated.

[0056] Through this embodiment, the estimation of the relative position and relative rotation angle by the second vision sensor improves the accuracy of the robot in grasping the memory module to be installed, and to a certain extent reduces the grasping failure rate caused by the posture error of the robotic arm.

[0057] In an exemplary embodiment, obtaining the first region image through the second vision sensor includes: determining target region information, where the target region information is used to indicate the region of the memory module to be installed in the target vehicle; according to the target region information, the second vision sensor is used to capture the memory module to be installed to obtain a first region image.

[0058] It should be noted that the target region information may be used to indicate the region of the memory module to be installed in the target vehicle.

[0059] Optionally, when using the second vision sensor in the robot to search for and locate the target vehicle, that is, when the target vehicle where the memory module to be installed is recognized, through a preset target detection algorithm, such as barcode recognition based on machine vision, the precise position of the memory module to be installed on the target vehicle can be determined. Once the position of the memory module to be installed on the target vehicle is determined, by adjusting the viewing angle and focal length of the second vision sensor, ensure that the sensor can clearly capture the image of the memory module to be installed, that is, the first area image.

[0060] Optionally, based on the barcode position of the preset target vehicle and the preset position offset, the target area information of the memory module to be installed in the target vehicle can be determined.

[0061] In one example, through the second vision sensor installed at the end of the robotic arm, a barcode on a target vehicle is recognized. According to the barcode information and the preset position offset algorithm, the specific position of the memory module in the target vehicle can be determined. For example, through barcode recognition, it is known that the coordinates of the barcode on a Tray are (100, 150). According to the preset spatial relationship between the memory module and the barcode, it can be deduced that the first memory module is located at (150, 200). In this way, the target area information of the memory module is determined. Then, by adjusting the second vision sensor at the end of the robotic arm to align it with the area indicated by the target area information, a high-definition image of the memory module is captured.

[0062] Through this embodiment, by using the second vision sensor to closely and accurately capture the image of the memory module to be installed, the specific position and range of the memory module on the target vehicle (such as a Tray) can be more accurately identified. This close-range visual feedback can significantly improve the accuracy of feature point detection; before the robotic arm grasps the memory module, the first area image obtained by the second vision sensor can provide real-time visual feedback, enabling the immediate adjustment of the robotic arm posture to align with the memory module for grasping. To a certain extent, the assembly process is optimized, the preparation time before assembly and operation errors are reduced, and the assembly quality and success rate are improved. The built-in vision sensor enables the robot to reduce its dependence on fixed external cameras, and the robotic arm can move freely in space to adapt to the positions and postures of memory modules in different working scenarios, enhancing the flexibility of memory assembly. This flexibility enables the assembly system to more efficiently process memory modules of multiple models and sizes.

[0063] In an exemplary embodiment, identifying the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the first region image includes: performing edge detection on the first region image to obtain line segment information of the first region image, where the line segment information of the first region image is used to indicate the edge of the memory module to be installed; determining the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the line segment information of the first region image and the pose information of the robotic arm.

[0064] It should be noted that the line segment information of the first region image may refer to a set of line segments extracted from the first region image through edge detection processing, which can represent the shape and position of the memory module to be installed. Each line segment in the set of line segments can be a digital representation of the edge of the memory module to be installed, and is used for subsequent feature point recognition and position and pose calculation. The pose information of the robotic arm may include its current position coordinates and planar rotation angle, which are pose parameters describing the pose of the robotic arm in three-dimensional space.

[0065] Optionally, the relative position between the memory module to be installed and the robotic arm refers to the distance and translation vector between the center point of the memory module to be installed and the center point of the end effector of the robotic arm in the world coordinate system; the relative rotation angle refers to the angle difference between the long side and short side directions of the memory module to be installed and the long side and short side directions of the end effector of the robotic arm.

[0066] Optionally, performing edge detection on the first region image to obtain line segment information of the first region image may specifically include: preprocessing the first region image, using an edge detection algorithm, such as the Canny edge detection or Sobel operator, to extract the line segment information of the edge of the memory module to be installed from the first region image, such as the long side and short side of the memory module to be installed, and the straight line segments of each edge.

[0067] Optionally, determining the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the line segment information of the first region image and the pose information of the robotic arm includes: based on the extracted line segment information of the first region image and the pose information of the robotic arm, through coordinate transformation and pose estimation algorithms, such as solving the PnP (Perspective-n-Point) problem, calculating the relative position and translation vector between the memory module and the robotic arm, and the relative rotation angle.

[0068] Through this embodiment, by identifying the relative position and rotation angle between the memory module to be installed and the robotic arm, the robot can accurately adjust the pose of the robotic arm to ensure that it can accurately align with the edge of the memory module to be installed when grasping the memory module to be installed, which can avoid grasping failures or damages caused by mismatched poses to a certain extent, and improve the stability of the memory module to be installed during the grasping process.

[0069] In an exemplary embodiment, feature points are extracted from the captured image of the memory module to obtain the feature point information of the memory module to be installed, including: performing edge detection on the downsampled images in the image pyramid respectively to obtain a set of edge maps; constructing an image based on the pixel values of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image; performing contour recognition on the first intermediate image, extracting and filtering the contours in the first intermediate image to obtain a second intermediate image, where the second intermediate image is the regional image within the contours in the first intermediate image; and recognizing the feature points in the second intermediate image to obtain the pixel coordinate information of the feature points in the second intermediate image, where the pixel coordinate information of the feature points in the second intermediate image is the feature point information of the memory module to be installed.

[0070] Optionally, performing edge detection on the downsampled images in the image pyramid respectively to obtain a set of edge maps can discover the contour features of the memory module at different scales, so as to construct a first intermediate image based on the pixel values of the pixel points of the edge maps in the set of edge maps.

[0071] The first intermediate image can be constructed based on the edge maps of the downsampled images in the image pyramid, and the contour information of the memory module to be installed can be obtained through pixel value analysis. The second intermediate image can be the regional image within the contours in the first intermediate image, and the second intermediate image can be used for accurate recognition of feature points. Optionally, by generating edge maps from the downsampled images in the image pyramid, and then analyzing the pixel values corresponding to each pixel point in the set of edge maps, a first intermediate image can be constructed, and contour recognition is performed on the first intermediate image, extracting and filtering the contours in the first intermediate image to obtain a second intermediate image. Optionally, in the process of extracting and filtering the contours in the first intermediate image, polygon approximation can be performed on all the contours of the first intermediate image, and the contours approximated as quadrilaterals and having an area meeting the specified threshold are filtered out, where the specified threshold can be determined based on the dimension information of the memory module to be installed.

[0072] The feature point information of the memory module to be installed is obtained by performing a feature extraction operation on the captured image of the memory module, and is the information used to determine the actual posture of the memory module to be installed. Specifically, the feature point information of the memory module to be installed can include the pixel coordinate information of the feature points in the second intermediate image.

[0073] Optionally, by performing feature point detection on the second intermediate image, the feature points of the memory module to be installed are obtained. Feature point detection can include but is not limited to corner detection, edge detection, Radon transform, etc. The feature points of the memory module to be installed can include but are not limited to corner points or specific edge points.

[0074] In specific practice, other methods such as deep learning and probability projection can also be adopted for the extraction process of the feature points of the memory module to be installed.

[0075] Through this embodiment, through image processing processes such as downsampling and edge detection, a preprocessing foundation is provided for feature point extraction, ensuring high-precision feature point detection. At the same time, by constructing and analyzing the image pyramid and edge map, the processing speed and calculation efficiency of feature point extraction are improved to a certain extent.

[0076] In an exemplary embodiment, an image is constructed based on the pixel values of the pixel points of the edge map in a set of edge maps to obtain a first intermediate image, including: traversing the edge maps in a set of edge maps and obtaining the pixel values of the pixel points of the edge maps in a set of edge maps; using the maximum pixel value of the pixel points at the same position in a set of edge maps as the pixel value of the pixel point at the corresponding position in the image to be constructed, and performing image construction to obtain a first intermediate image.

[0077] It should be noted that a set of edge maps may refer to the image set obtained by performing edge detection on each layer of downsampled images during the construction of the image pyramid. A set of edge maps can highlight the contour and edge information of the memory module to be installed.

[0078] Optionally, the image pyramid may include downsampled images at different levels. On each level of the image pyramid, a preset edge detection algorithm, such as the Canny algorithm or Sobel operator, etc., is used to process the downsampled images in the image pyramid to obtain the edge maps at the corresponding levels.

[0079] The pixel value of a pixel point refers to the gray value or color intensity value of each pixel point in the image, which reflects the image feature intensity of the pixel point. The first intermediate image is a comprehensive image obtained by merging the maximum pixel values of the pixel points at the same position in a set of edge maps. The first intermediate image can centrally display the most prominent edge features of the memory module to be installed.

[0080] In an example, by traversing the pixel points of each edge map, the pixel value of each pixel point is recorded. The higher the gray value, the more obvious the edge; the lower the gray value, the weaker the edge or it is not an edge. For example, at a certain pixel position (200, 150) in the edge map, the gray values recorded for different layers are 255, 230, 240, and 250 respectively, then the maximum gray value (such as 255) is selected as the pixel value at the corresponding position in the first intermediate image. After this process, the contour line of the memory module is clearly revealed in the first intermediate image, facilitating subsequent feature point extraction.

[0081] Through this embodiment, the first intermediate image constructed by merging the maximum gray values in a group of edge images can strengthen the edge features of the memory module to be installed, making the extraction of feature points of the memory module to be installed more accurate. By constructing the first intermediate image instead of processing each edge image individually, the recognition of the feature points of the memory module to be installed can be completed with fewer computing resources and time, improving the efficiency of the memory assembly process to a certain extent.

[0082] In an exemplary embodiment, identifying the feature points in the second intermediate image and obtaining the pixel coordinate information of the feature points in the second intermediate image includes: performing edge detection on the second intermediate image to obtain the edge image corresponding to the second intermediate image; performing Radon transform on the edge image corresponding to the second intermediate image to obtain the Radon transform result; performing peak detection on the Radon transform result to obtain the straight line parameters in the second intermediate image; determining the feature points in the second intermediate image according to the straight line parameters in the second intermediate image, and calculating the pixel coordinate information of the feature points in the second intermediate image.

[0083] It should be noted that the second intermediate image may refer to an image that only contains the memory module to be installed after contour recognition and screening. The second intermediate image has undergone contour screening, removing the non-memory module areas in the image and retaining the clear contour and internal structure of the memory module.

[0084] Use edge detection algorithms (such as Canny edge detection, Sobel operator, etc.) to identify the boundaries and feature lines of the memory module to be installed in the second intermediate image, and obtain the edge image corresponding to the second intermediate image. The Radon transform result may include, but is not limited to, the radial distance and angle parameters of the edge lines of the memory module to be installed, the peak points in the Radon space, and the distribution information of the straight line parameters. Among them, the radial distance represents the perpendicular distance from the straight line to the center of the image, and the angle represents the included angle between the straight line direction and the horizontal axis. Each combination of radial distance and angle corresponds to a straight line on the memory module. The peak points in the Radon space can represent the most obvious straight line features on the memory module to be installed. The distribution information of the straight line parameters can include the distribution of different straight line parameters in the Radon space.

[0085] Optionally, performing Radon transform on the edge image corresponding to the second intermediate image to obtain the Radon transform result may include: mapping the edge lines of the memory module to be installed into the Radon space, using the radial distance and angle of the straight line as parameters to form the Radon transform result. Specifically, perform Radon transform on the edge image corresponding to the second intermediate image, and project the edge lines of the long side and short side of the memory module to be installed into the Radon space. Each edge line is represented as a curve in the Radon space, and the peak points of these curves correspond to the straight lines on the edge of the memory module to be installed. The radial distance and angle of the straight lines on the edge of the memory module to be installed are the straight line parameters.

[0086] Specifically, the Radon transform of the edge image corresponding to the second intermediate image can be performed using Equation (1):

[0087] ; (1)

[0088] where is the result of the Radon transform, is the perpendicular distance from the origin to the line, is the perpendicular line (i.e., the vector perpendicular to the line) and the positive direction of the axis, and are the positions along the axis and axis in the image coordinate system.

[0089] Peak detection is performed on the result of the Radon transform. The local maximum detection algorithm is used to find the peak points. The peak points correspond to the lines on the edge of the memory module, and the radial distance and angle can be extracted from them to describe the position and direction of the lines, thereby obtaining the line parameters in the second intermediate image. The line equation can be as shown in Equation (2):

[0090] ; (2)

[0091] where represents the coordinate points in the image coordinate system.

[0092] Specifically, since the edge of the memory module to be installed is a straight line and the feature points can be corner points, and the angle corresponding to the feature points is a right angle, the angle range of the Radon transform can be mainly set near 0°, 90°, 180°, and 270°, and the sampling density can be appropriately increased.

[0093] Specifically, through the system of equations formed by two lines in the local area (i.e., Equation (3)), the corner point coordinates can be solved:

[0094] ; (3)

[0095] where , ; , can be the angles between the normal lines of the two lines and the positive direction of the axis; and can be the corner point coordinates.

[0096] Specifically, for each feature point, the corner position of the memory module to be installed can be determined by formula (3), that is, the pixel coordinate information of the feature point in the second intermediate image. In specific practice, in different forms of gripper grasping states, some corners may be blocked. Therefore, the unblocked corners can be defined as feature points to participate in the calculation.

[0097] Through this embodiment, by performing Radon transform and peak detection on the Radon transform result, the straight-line features of the second intermediate image can be obtained, and the key feature points on the memory module can be accurately identified. Moreover, through precise feature point identification and coordinate information calculation, the assembly error caused by inaccurate grasping angle or position can be effectively reduced, improving the assembly quality and success rate.

[0098] In an exemplary embodiment, the feature point information of the memory module to be installed is the pixel coordinate information of the feature points of the memory module to be installed.

[0099] According to the feature point information of the memory module to be installed, determine the actual pose of the memory module to be installed, including: mapping the pixel coordinate information of the feature points of the memory module to be installed according to the specification information of the memory module to be installed and the world coordinate system to obtain the three-dimensional coordinate information of the feature points of the memory module to be installed; calculating the parameter information of the memory module to be installed according to the camera coordinate system of the first vision sensor, the three-dimensional coordinate information of the feature points of the memory module to be installed, and the pixel coordinate information of the feature points of the memory module to be installed, where the parameter information of the memory module to be installed includes the rotation parameter of the memory module to be installed relative to the first vision sensor and the translation parameter of the memory module to be installed relative to the first vision sensor; calculating the actual pose of the memory module to be installed according to the parameter information of the memory module to be installed.

[0100] It should be noted that the specification information of the memory module to be installed includes detailed parameters such as the size, shape, and three-dimensional coordinates of the key feature points of the memory module. The specification information of the memory module to be installed can usually be obtained from the manufacturing data or specification manual of the memory module to be installed. In specific practice, since the processing accuracy of the memory module size is ±0.15 mm, the three-dimensional coordinate information of the feature points of the memory module to be installed can be directly determined according to the size of the memory module.

[0101] The feature point information of the memory module to be installed is the pixel coordinate information of the feature points of the memory module to be installed, that is, the pixel coordinate information of the key feature points in the second intermediate image. For example, the coordinate information of four corner points (150, 100), (350, 100), (150, 200), (350, 200) is detected from the second intermediate image.

[0102] The three-dimensional coordinate information of the feature points of the memory module to be installed can be a description of the positions of the feature points of the memory module to be installed in space, usually including the x, y, and z coordinates, and is used to characterize the positions of the feature points in the world coordinate system. Specifically, the pixel coordinate information of the feature points is converted into three-dimensional coordinates in the world coordinate system through mapping processing.

[0103] The parameter information of the memory module to be installed includes the rotation parameter of the memory module to be installed relative to the first vision sensor and the translation parameter of the memory module to be installed relative to the first vision sensor. Among them, the relative rotation parameter and the relative translation parameter describe the pose change of the memory module to be installed relative to the first vision sensor and are used to represent the three-dimensional pose of the memory module. Specifically, according to the pixel coordinates of the corner points in the image coordinate system, the camera parameters of the first vision sensor, and the camera coordinate system, the coordinate information of the corner points in the camera coordinate system is determined; by solving the Perspective-n-Point (PnP) problem and using the three-dimensional coordinate information of the corner points and the coordinate information of the corner points in the camera coordinate system, the rotation parameter and the translation parameter of the memory module to be installed relative to the first vision sensor are calculated.

[0104] Specifically, taking Figure 3 as an example for illustration, Figure 3 FIG. is a schematic diagram of the feature points of an optional memory module provided according to an embodiment of the present application. Among them, Figure 3 The part pointed by the arrow in Figure 3 can be the feature points of the memory module. Taking as an example, taking corner point A as the origin of the coordinate system and the two orthogonal axes in the plane as the axis and the axis (generally, the long side is the axis and the short side is the axis), and the direction of the plane normal vector is the axis to define the coordinate system, and this coordinate system is used as the world coordinate system ; define the first vision sensor as a monocular camera, and the corresponding camera coordinate system is , the origin is located at the optical center of the external monocular camera, the axis is along the optical axis direction of the external monocular camera, and the direction is the same as the xy definition of the image plane; the image plane coordinate system is , the axis and the axis are respectively parallel to the horizontal and vertical directions of the image. Then, the two-dimensional pixel coordinates of the corner points obtained through corner point detection are shown in formula (4):

[0105] ; (4)

[0106] where N is the number of corner points, ( ) ( ) represents the two-dimensional pixel coordinates of corner point j in the image coordinate system.

[0107] According to the coordinate representation of points in the camera coordinate system of a monocular camera as shown in formula (5):

[0108] ; (5)

[0109] Among them, can represent the position of corner point j in the camera coordinate system;

[0110] Then the coordinates projected onto the image plane can be calculated by formula (6):

[0111] ; (6)

[0112] Among them, is the equivalent focal length in the x-axis direction of the camera coordinate system, is the equivalent focal length in the y-axis direction of the camera coordinate system, is the pixel size in the x-axis direction of the camera coordinate system, is the pixel size in the y-axis direction of the camera coordinate system, is the distance of the origin of the camera coordinate system in the x-axis direction of the image coordinate system, is the distance of the origin of the camera coordinate system in the y-axis direction of the image coordinate system, which can be obtained through calibration.

[0113] The transformation of the corner point from the world coordinate system to the camera coordinate system can be obtained through the rotation matrix and the translation vector as shown in formula (7) specifically:

[0114] ; (7)

[0115] Among them, is the rotation matrix; is the translation vector; is the position of corner point j in the world coordinate system.

[0116] The representation forms of the rotation matrix and the translation vector can be referred to as shown in formula (8):

[0117] ; (8)

[0118] Among them, represents the rotation transformation parameter, , , represents the translation transformation parameter.

[0119] For each corner point j, two equations can be obtained (such as Equation (9) and Equation (10)) (ignoring the non-zero factors multiplied on both sides of the equation), which can be:

[0120] ; (9)

[0121] ; (10)

[0122] where, ( , ) is the focal length of the monocular camera; ( , ) is the pixel size of the monocular camera, usually the physical size of a unit pixel; ( , ) are the pixel coordinates of the projection of the optical center of the monocular camera on the image.

[0123] Among them, the rotation matrix has 9 elements, and the translation vector has 3 elements. However, since the rotation matrix is an orthogonal matrix, it satisfies Equation (11):

[0124] ; (11)

[0125] where, I is the identity matrix; is the transpose matrix of, that is, the rows and columns of are interchanged. Therefore, there are only 6 parameters (3 rotation parameters and 3 translation parameters) to be solved, that is, the parameter information of the memory module to be installed.

[0126] In specific practice, the nonlinear least squares method can be used to solve for the pose parameters (that is, the parameter information of the memory module to be installed), as shown in Equation (12):

[0127] ; (12)

[0128] where, and are the projected coordinates of each corner point calculated under the parameter information of the memory module to be solved, is the error value. By using the method of SVD (Singular Value Decomposition), the rotation matrix and the translation vector t that minimize can be solved, and then the pose parameters obtained by solving are determined.

[0129] Based on the obtained attitude parameters (parameter information of the memory module to be installed), determine the actual attitude of the memory module relative to the robot's operating space (i.e., the end of the robotic arm). Among them, the actual attitude can include the position coordinates and deflection angles of the memory module, so as to adjust the end attitude of the robotic arm according to the difference between the actual attitude and the preset attitude of the memory module, thereby adjusting the attitude of the memory module to be installed.

[0130] In this embodiment, the pixel coordinate information of the feature points is converted into three-dimensional coordinate information, and then the relative rotation parameters and translation parameters are calculated, and finally the actual attitude of the memory module is determined, so as to adjust the attitude of the end of the robotic arm, which improves the assembly success rate and reliability to a certain extent.

[0131] In an exemplary embodiment, before installing the memory module to be installed into the corresponding specified memory slot by moving the robotic arm with the adjusted end attitude, the above method further includes:

[0132] Obtain a device image, where the device image is an image of the target device to which the memory module to be installed is to be installed; extract slot features from the device image to obtain slot features; based on the slot features, determine the position and attitude information of the specified memory slot, and based on the position and attitude information of the specified memory slot, calculate the slot position of the specified memory slot, where the robotic arm with the adjusted end attitude is executed according to the slot position of the specified memory slot.

[0133] It should be noted that the device image may refer to an image of the target device taken by the robotic arm before preparing to install the memory module to be installed. For example, when the target device is a server, the device image contains various features of the server motherboard, especially the image information of the memory slots. Slot feature extraction refers to identifying specific features of the memory slots from the device image, such as shape, size, position, etc., which will be used to calculate the position and attitude information of the slots. The position and attitude information of the specified memory slot can include the position coordinates and attitude angles of the specified slot, that is, the three-dimensional coordinates and rotation angles relative to the world coordinate system. The slot position may refer to the exact position coordinates of the specified memory slot relative to the robot or a preset reference point in the world coordinate system.

[0134] To further ensure that the memory module to be installed can be accurately installed into the target device to which it is to be installed, it is also necessary to consider identifying the slot characteristics of the target device and calculating the actual position and attitude information of the slot. Specifically, after detecting and adjusting the attitude of the memory module, the end of the robot is adjusted to a specified position to ensure that a device image of the target device is captured. The device image is preprocessed to reduce noise interference, and then image processing algorithms (such as edge detection, feature point matching, etc.) are used to identify the characteristics of the memory slot. For example, the contour and internal structure details of the memory slot can be identified through template matching or deep learning algorithms, and then the position and possible tilt angle of the memory slot can be determined. After identifying the characteristics of the memory slot, combining the known layout information of the device and the theoretical position and attitude information of the slot, using similar image processing and geometric calculation methods (such as Radon transform and PnP problem solving), the position and attitude information of the specified memory slot are determined, and based on the position and attitude information of the specified memory slot, the slot position of the specified memory slot is calculated, which can specifically include the coordinate values of the specified memory slot on the x, y, and z axes and the respective rotation angles. According to the slot position of the specified memory slot, the robotic arm is moved to ensure that the memory module can be accurately inserted into the corresponding slot, avoiding collisions during installation or installation failures caused by mismatched attitudes.

[0135] Optionally, the process of determining the slot position of the specified memory slot can be as follows: The device image is preprocessed, including but not limited to edge enhancement, denoising, and contrast adjustment, to enhance the visibility of the slot characteristics and reduce interference. Image enhancement techniques (such as histogram equalization, contrast stretching) are used to improve the image contrast of the memory slot for subsequent feature recognition. Edge detection is performed on the preprocessed device image to obtain the edge information of the memory slot. Hough line detection is used to identify line segments from the edge information, and these line segments correspond to the contour of the memory slot, thereby obtaining edge features. Template matching or deep learning methods (such as convolutional neural network CNN) are used to identify the specific shape and structure of the memory slot. Through feature point matching, contour analysis, or deep learning classification, it is determined which regions in the image are the memory slot and the specific position and possible tilt angle of the memory slot.

[0136] To better understand the process of memory assembly, an example is used for illustration, such as Figure 4 shown Figure 4 is a schematic diagram of an optional memory assembly provided according to an embodiment of the present application.

[0137] Among them, the robot is a composite robot. The end of the robot's robotic arm can be a two-finger gripper. A built-in vision sensor (i.e., an eye-in-hand camera) is configured near the end of the robotic arm. A memory module (i.e., the memory module to be installed) is gripped by the two-finger gripper. The first vision sensor is a monocular camera. The target device to be installed can be a server motherboard. By moving the robotic arm with the adjusted end pose, the memory module to be installed is installed into the designated memory slot corresponding to the memory module to be installed.

[0138] Through this embodiment, by extracting the slot features and calculating the pose information of the device image, this solution can accurately determine the position and tilt angle of the memory slot. Even under complex environmental conditions, accurate alignment between the memory module and the slot can be achieved, significantly improving the assembly accuracy. By accurately calculating the pose information of the slot, to a certain extent, it ensures that the memory module to be installed is installed into the slot in the correct pose and position, avoiding possible hardware damage caused by inaccurate positioning, such as damage to the contacts of the memory module or wear of the slot on the motherboard. Moreover, by combining the pose adjustment of the memory module and the accurate calculation of the slot pose information, the success rate of memory installation can be greatly improved, the failure rate can be reduced, thereby reducing the product defect rate and improving the overall production quality.

[0139] In an exemplary embodiment, the above method further includes: performing image acquisition on the detection area through the first vision sensor to obtain a second area image, where the detection area is the area where a set vehicle is located; extracting the region of interest in the second area image, and performing barcode recognition on the extracted region of interest to obtain a barcode recognition result; in the case where the barcode recognition result indicates that the memory module to be installed is recognized, determining the vehicle on which the memory module to be installed is placed as the target vehicle.

[0140] It should be noted that the first vision sensor refers to a vision device installed outside the robot, such as a monocular camera or a stereo camera, which is used to perform image acquisition on the detection area to obtain a second area image.

[0141] The detection area can be the area where a set vehicle is located, generally referring to a specific spatial area that needs to be detected during the operation of the robot.

[0142] The second area image can be an image of the detection area taken by the first vision sensor, including one or more vehicles and the memory modules that may be placed on the vehicles. The region of interest can be an image area in the second area image that is screened out after image processing and may contain memory module information.

[0143] In the region of interest of the second area image, image processing algorithms such as color threshold segmentation, dilation and erosion can be used to extract the area where the barcode on each vehicle is located. Optical character recognition (OCR) or a dedicated barcode recognition algorithm is performed on the barcode in the ROI to obtain the information carried by the barcode. Here, it is used to identify the presence and type of the memory module, so as to determine whether there is a memory module to be installed. In the case of determining that there is a memory module to be installed, the vehicle on which the memory module to be installed is placed is determined as the target vehicle.

[0144] In one example, the robot moves to the working area containing the vehicle, adjusts the shooting angle and distance of the first vision sensor to ensure that all vehicles can be captured completely, and the obtained image after shooting is the second area image. The second area image is preprocessed, such as grayscale conversion, filtering and denoising, etc., to enhance the image contrast and reduce interference. Image processing techniques such as color threshold segmentation, dilation and erosion are used to identify the barcode area in the image. Barcode recognition uses a barcode recognition algorithm in the determined ROI to optically recognize the barcode and obtain barcode information. In the case where the barcode information contains the identification information of the memory module to be installed, then the vehicle is determined as the "target vehicle" containing the memory module to be installed.

[0145] Through this embodiment, through barcode recognition, it is possible to quickly and accurately determine which vehicle is carrying the memory module to be installed, avoiding the detection and processing of irrelevant vehicles and improving the overall recognition efficiency.

[0146] In an exemplary embodiment, the end of the robotic arm includes a gripper; installing the memory module to be installed into the specified memory slot corresponding to the memory module to be installed includes: obtaining the force information of the gripper, where the force information is used to indicate the contact force between the gripper and the memory module to be installed and the torque of the gripper grasping the memory module to be installed; according to the force information, adjusting the posture of the gripper, and when the first preset condition is met, installing the memory module to be installed into the specified memory slot through the gripper, where the first preset condition is that the value of the contact force is greater than or equal to the first preset contact force threshold, the value of the contact force is less than the second preset contact force threshold, and the value of the torque is less than the first preset torque threshold.

[0147] It should be noted that the end of the robotic arm may include a gripper. The gripper may refer to a tool at the end of the robotic arm for grasping and manipulating objects. Usually, it can have two or more mechanical fingers and can be used to precisely grasp the memory module to be installed. The force information may refer to the information of the force and torque generated when contacting the memory module to be installed during the installation process of the memory module to be installed. Among them, the contact force may be the pressure or thrust between the gripper and the memory module to be installed, and the torque may refer to any force that causes a rotational tendency of the memory module to be installed.

[0148] Optionally, the first preset contact force threshold may be the preset minimum contact force threshold to ensure that the gripper can firmly grasp the memory module to be installed, avoiding the memory module slipping or detaching from the gripper during the installation process. The second preset contact force threshold may be the preset maximum contact force threshold to prevent the gripper from applying too much force to the memory module to be installed, so as to avoid damaging the contacts of the memory module to be installed or the slot of the device to be installed. The first preset torque threshold may be the preset maximum torque threshold to limit the influence of the gripper on the rotation of the memory module to be installed during the installation process, avoiding damage to the memory module to be installed due to excessive torque when inserting it into the slot.

[0149] Optionally, when the second preset condition is met, stop installing the memory module to be installed into the specified memory slot, where the second preset condition is that the value of the contact force is greater than or equal to the second preset contact force threshold, or the value of the torque is greater than the first preset torque threshold.

[0150] Optionally, the robot can grasp the memory module through different types of grippers (such as soft or rigid) to adapt to memory modules of different sizes or shapes. The selection of the gripper is based on the size and material of the memory module to avoid damaging the memory module during the grasping process.

[0151] Through this embodiment, by real-time monitoring of the force information of the gripper, it is possible to prevent and avoid the memory module to be installed from slipping, being damaged or causing damage to the device to be installed during the installation process, improving the safety of the installation process. Through the obtained bit state information of the memory module and the slot, as well as the feedback of the force information, it can be ensured that the memory module is installed into the specified slot in the correct posture and position, improving the assembly accuracy and success rate. At the same time, during the installation process, the force information feedback mechanism of the gripper allows the robotic arm to dynamically adjust its posture to adapt to the possible small deviations between the memory module and the slot, improving the robustness and adaptability of the system.

[0152] It should be noted that the memory assembly methods in the above various embodiments are not limited to the assembly of memory modules, and are equally applicable to the precise positioning and assembly of components in PCIe (Peripheral Component Interconnect Express) slots. Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0153] The embodiment of the present application also provides a memory assembly system. Figure 5 For the structural block diagram of an optional memory assembly system provided according to the embodiment of the present application, as Figure 5As shown in the figure, the memory assembly system includes a control device 502, a robot 504, and a first vision sensor 506. The robot 504 is configured with a robotic arm 5042.

[0154] The control device 502 is used to, when identifying the target vehicle where the memory module to be installed is located, control the robotic arm to grasp the memory module to be installed in the target vehicle, and control the first vision sensor to obtain an image of grasping the memory module, where the image of grasping the memory module is an image when the robotic arm grabs the memory module to be installed; construct an image pyramid of the image of grasping the memory module, where the image pyramid includes a set of downsampled images obtained by performing a set of downsampling operations with different granularities on the image of grasping the memory module; perform a feature extraction operation on the image pyramid to obtain the feature point information of the memory module to be installed; determine the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed; based on the actual pose and the preset memory module pose, adjust the end pose of the robotic arm to adjust the pose of the memory module to be installed; control the robotic arm with the adjusted end pose to move and install the memory module to be installed into the corresponding designated memory slot.

[0155] The robotic arm 5042 is used to, in response to the control of the control device, grasp the memory module to be installed in the target vehicle, adjust the end pose of the robotic arm, move and install the memory module to be installed into the designated memory slot.

[0156] The first vision sensor 506 is used to, in response to the control of the control device, obtain an image of grasping the memory module.

[0157] It should be noted that the control device can be built into the robot or not located in the robot.

[0158] Through the embodiments provided in this application, when identifying the target vehicle where the memory module to be installed is located, the robotic arm is used to grasp the memory module to be installed in the target vehicle, and the first vision sensor is used to obtain an image of grasping the memory module, and an image pyramid of the image of grasping the memory module is constructed, allowing feature detection at different image resolutions, thereby improving the accuracy of the feature extraction operation, and thus obtaining more accurate feature point information of the memory module to be installed to determine the actual pose of the memory module. According to the actual pose of the memory module to be installed and the preset memory module pose, the end pose of the robotic arm is adjusted, enabling the robotic arm to dynamically correct the pose of the memory module to be installed, solving the problem of low precision in memory assembly caused by possible pose deviation of the memory module after grasping in the related art, improving the precision and reliability of memory assembly, reducing the assembly failure rate to a certain extent, and improving production efficiency.

[0159] In an exemplary embodiment, as Figure 6 shownFigure 6 It is a structural block diagram of another optional memory assembly system provided by an embodiment of the present application; the robot 504 is configured with a second vision sensor 5044; the control device 502 is further configured to control the second vision sensor to obtain a first area image, where the first area image is an image of the area where the memory module to be installed is located in the target vehicle; based on the first area image, identify the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm; based on the relative position and the relative rotation angle, adjust the posture of the robotic arm, and control the robotic arm with the adjusted posture to grasp the memory module to be installed.

[0160] The second vision sensor 5044 is configured to obtain a first area image in response to the control of the control device.

[0161] In an exemplary embodiment, the control device 502 is further configured to determine target area information, where the target area information is used to indicate the area of the memory module to be installed in the target vehicle; based on the target area information, control the second vision sensor to photograph the memory module to be installed, and obtain a first area image.

[0162] The second vision sensor 5044 is configured to obtain a first area image in response to the control of the control device.

[0163] In an exemplary embodiment, the control device 502 is further configured to perform edge detection on the first area image to obtain line segment information of the first area image, where the line segment information of the first area image is used to indicate the edge of the memory module to be installed; based on the line segment information of the first area image and the pose information of the robotic arm, determine the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm.

[0164] In an exemplary embodiment, the control device 502 is further configured to perform edge detection on the downsampled images in the image pyramid respectively to obtain a set of edge maps; perform image construction based on the pixel values of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image; perform contour recognition on the first intermediate image, extract and filter the contours in the first intermediate image to obtain a second intermediate image, where the second intermediate image is the area image within the contour in the first intermediate image; identify the feature points in the second intermediate image to obtain the pixel coordinate information of the feature points in the second intermediate image, where the pixel coordinate information of the feature points in the second intermediate image is the feature point information of the memory module to be installed.

[0165] In an exemplary embodiment, the control device 502 is further configured to traverse the edge maps in a set of edge maps, and obtain the pixel values of the pixel points of the edge maps in the set of edge maps; use the maximum pixel value of the pixel points at the same position in the set of edge maps as the pixel value of the pixel point at the corresponding position in the image to be constructed, and perform image construction to obtain a first intermediate image.

[0166] In an exemplary embodiment, the control device 502 is further configured to perform edge detection on the second intermediate image to obtain an edge image corresponding to the second intermediate image; perform Radon transform on the edge image corresponding to the second intermediate image to obtain a Radon transform result; perform peak detection on the Radon transform result to obtain the straight line parameters in the second intermediate image; determine the feature points in the second intermediate image according to the straight line parameters in the second intermediate image, and calculate the pixel coordinate information of the feature points in the second intermediate image.

[0167] In an exemplary embodiment, the feature point information of the memory module to be installed is the pixel coordinate information of the feature points of the memory module to be installed.

[0168] The control device 502 is further configured to perform mapping processing on the pixel coordinate information of the feature points of the memory module to be installed according to the specification information of the memory module to be installed and the world coordinate system to obtain the three-dimensional coordinate information of the feature points of the memory module to be installed; calculate the parameter information of the memory module to be installed according to the camera coordinate system of the first vision sensor, the three-dimensional coordinate information of the feature points of the memory module to be installed, and the pixel coordinate information of the feature points of the memory module to be installed, where the parameter information of the memory module to be installed includes the rotation parameter of the memory module to be installed relative to the first vision sensor and the translation parameter of the memory module to be installed relative to the first vision sensor; calculate the actual pose of the memory module to be installed according to the parameter information of the memory module to be installed.

[0169] In an exemplary embodiment, before the robot arm with the adjusted end pose moves to install the memory module to be installed into the specified memory slot corresponding to the memory module to be installed;

[0170] The control device 502 is further configured to obtain a device image, where the device image is an image of the target device where the memory module to be installed is to be installed; perform slot feature extraction on the device image to obtain slot features; determine the pose information of the specified memory slot based on the slot features, and calculate the slot position of the specified memory slot based on the pose information of the specified memory slot, where the control of the robot arm with the adjusted end pose is performed according to the slot position of the specified memory slot.

[0171] In an exemplary embodiment, the control device 502 is further configured to control the first vision sensor to collect an image of a detection area to obtain a second area image, where the detection area is a set area where the vehicle is located; extract a region of interest in the second area image, and perform barcode recognition on the extracted region of interest to obtain a barcode recognition result; in the case where the barcode recognition result indicates that a memory module to be installed is recognized, determine the vehicle on which the memory module to be installed is placed as the target vehicle;

[0172] The first vision sensor 506 is configured to, in response to the control of the control device, collect an image of a detection area to obtain a second area image.

[0173] In an exemplary embodiment, the first vision sensor 506 is further configured to, in response to the control of the control device, collect an image of a detection area to obtain a second area image.

[0174] In an exemplary embodiment, the end of the robotic arm 5042 includes a gripper; the control device 502 is further configured to obtain the force information of the gripper, where the force information is used to indicate the contact force between the gripper and the memory module to be installed and the torque of the gripper grasping the memory module to be installed; according to the force information, adjust the posture of the gripper, and in the case where a first preset condition is satisfied, control the gripper to install the memory module to be installed into a specified memory slot, where the first preset condition is that the value of the contact force is greater than or equal to a first preset contact force threshold, the value of the contact force is less than a second preset contact force threshold, and the value of the torque is less than a first preset torque threshold; the gripper is configured to, in response to the control of the control device 502, install the memory module to be installed into a specified memory slot.

[0175] For the description of the features in the embodiments corresponding to the memory assembly system, reference may be made to the relevant descriptions in the embodiments corresponding to the memory assembly method, which will not be elaborated here one by one.

[0176] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the embodiments of the above-mentioned memory assembly method.

[0177] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any one of the embodiments of the above-mentioned memory assembly method when running.

[0178] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs.

[0179] The embodiments of the present application also provide a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments of the memory assembly method are implemented.

[0180] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments of the memory assembly method are implemented.

[0181] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0182] The above has introduced in detail a memory assembly method, system, electronic device, storage medium, and product provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A memory assembly method, characterized in that, Applied to a memory assembly system, the memory assembly system includes a robot and a first vision sensor, and the robot is configured with a robotic arm. The method includes: When a target vehicle where a memory module to be installed is recognized, the memory module to be installed in the target vehicle is grasped by the robotic arm, and a memory module grasping image is obtained by the first vision sensor, where the memory module grasping image is an image when the robotic arm grasps the memory module to be installed; Construct an image pyramid of the memory module grasping image, where the image pyramid includes a set of downsampled images obtained by performing a set of downsampling operations with different granularities on the memory module grasping image; the downsampling operation is to merge consecutive pixel blocks in the filtered memory module grasping image into a new pixel; Perform edge detection on the downsampled images in the image pyramid respectively to obtain a set of edge maps; Perform image construction based on the pixel values of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image; Perform contour recognition on the first intermediate image, extract and filter the contours in the first intermediate image to obtain a second intermediate image, where the second intermediate image is an area image within the contours in the first intermediate image; Identify the feature points in the second intermediate image to obtain the pixel coordinate information of the feature points in the second intermediate image, where the pixel coordinate information of the feature points in the second intermediate image is the feature point information of the memory module to be installed; Determine the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed; Based on the actual pose and a preset memory module pose, adjust the end pose of the robotic arm to adjust the pose of the memory module to be installed; Install the memory module to be installed into the specified memory slot corresponding to the memory module to be installed by the robotic arm with the adjusted end pose.

2. The method according to claim 1, characterized in that, The robot is configured with a second vision sensor; the grasping of the memory module to be installed in the target vehicle by the robotic arm includes: Obtain a first area image by the second vision sensor, where the first area image is an image of the area where the memory module to be installed is located in the target vehicle; Identify the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the first area image; Adjust the pose of the robotic arm according to the relative position and the relative rotation angle, and grasp the memory module to be installed with the robotic arm with the adjusted pose.

3. The method according to claim 2, wherein The obtaining of the first area image by the second vision sensor includes: Determine target area information, where the target area information is used to indicate the area of the memory module to be installed in the target vehicle; Take a picture of the memory module to be installed by the second vision sensor according to the target area information to obtain the first area image.

4. The method according to claim 2, wherein Identifying the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the first region image includes: Performing edge detection on the first region image to obtain line segment information of the first region image, where the line segment information of the first region image is used to indicate the edge of the memory module to be installed; Determining the relative position between the memory module to be installed and the robotic arm and the relative rotation angle between the memory module to be installed and the robotic arm according to the line segment information of the first region image and the pose information of the robotic arm.

5. The method according to claim 1, wherein Constructing an image based on the pixel values of the pixel points of the edge images in the set of edge images to obtain a first intermediate image, including: Traversing the edge images in the set of edge images and obtaining the pixel values of the pixel points of the edge images in the set of edge images; Using the maximum pixel value of the pixel points at the same position in the set of edge images as the pixel value of the pixel point at the corresponding position in the image to be constructed, and performing image construction to obtain the first intermediate image.

6. The method according to claim 1, characterized in that Identifying the feature points in the second intermediate image to obtain the pixel coordinate information of the feature points in the second intermediate image, including: Performing edge detection on the second intermediate image to obtain the corresponding edge image of the second intermediate image; Performing a Radon transform on the corresponding edge image of the second intermediate image to obtain a Radon transform result; Performing peak detection on the Radon transform result to obtain the line parameters in the second intermediate image; Determining the feature points in the second intermediate image according to the line parameters in the second intermediate image, and calculating the pixel coordinate information of the feature points in the second intermediate image.

7. The method according to claim 1, wherein The feature point information of the memory module to be installed is the pixel coordinate information of the feature points of the memory module to be installed; Determining the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed, including: Performing mapping processing on the pixel coordinate information of the feature points of the memory module to be installed according to the specification information of the memory module to be installed and the world coordinate system to obtain the three-dimensional coordinate information of the feature points of the memory module to be installed; Calculating the parameter information of the memory module to be installed according to the camera coordinate system of the first vision sensor, the three-dimensional coordinate information of the feature points of the memory module to be installed, and the pixel coordinate information of the feature points of the memory module to be installed, where the parameter information of the memory module to be installed includes the rotation parameter of the memory module to be installed relative to the first vision sensor and the translation parameter of the memory module to be installed relative to the first vision sensor; Calculating the actual pose of the memory module to be installed according to the parameter information of the memory module to be installed.

8. The method according to claim 1, wherein Before installing the memory module to be installed into the corresponding designated memory slot of the memory module to be installed by moving the robotic arm with the adjusted end pose, the method further includes: Obtaining a device image, where the device image is an image of the target device to which the memory module to be installed is to be installed; Performing slot feature extraction on the device image to obtain slot features; Based on the slot features, determine the position state information of the specified memory slot, and based on the position state information of the specified memory slot, calculate the slot position of the specified memory slot, wherein the robotic arm after the end posture adjustment is executed according to the slot position of the specified memory slot.

9. The method according to claim 1, wherein The method further includes: Collect an image of a detection area through the first vision sensor to obtain a second area image, wherein the detection area is the area where the set vehicle is located; Extract the region of interest in the second area image, and perform barcode recognition on the extracted region of interest to obtain a barcode recognition result; When the barcode recognition result indicates that the memory module to be installed is recognized, determine the vehicle on which the memory module to be installed is placed as the target vehicle.

10. The method according to any one of claims 1 to 9, characterized in that, The end of the robotic arm includes a gripper; the step of installing the memory module to be installed into the corresponding specified memory slot of the memory module to be installed includes: Obtain the force information of the gripper, wherein the force information is used to indicate the contact force between the gripper and the memory module to be installed and the torque of the gripper grasping the memory module to be installed; According to the force information, adjust the posture of the gripper, and when a first preset condition is met, install the memory module to be installed into the specified memory slot through the gripper, wherein the first preset condition is that the value of the contact force is greater than or equal to a first preset contact force threshold, the value of the contact force is less than a second preset contact force threshold, and the value of the torque is less than a first preset torque threshold.

11. A memory assembly system, characterized in that, It includes a control device, a robot, and a first vision sensor, and the robot is configured with a robotic arm; wherein, The control device is configured to, when a target vehicle where a memory module to be installed is recognized, control the robotic arm to grasp the memory module to be installed in the target vehicle, and control the first vision sensor to acquire a memory module grasping image, where the memory module grasping image is an image when the robotic arm grasps the memory module to be installed; construct an image pyramid of the memory module grasping image, where the image pyramid includes a set of downsampled images obtained by performing a set of downsampling operations with different granularities on the memory module grasping image; the downsampling operation is to merge consecutive pixel blocks in the filtered memory module grasping image into a new pixel; perform edge detection on the downsampled images in the image pyramid respectively to obtain a set of edge maps; perform image construction based on the pixel values of the pixel points of the edge maps in the set of edge maps to obtain a first intermediate image; perform contour recognition on the first intermediate image, extract and filter the contours in the first intermediate image to obtain a second intermediate image, where the second intermediate image is a regional image within the contours in the first intermediate image; identify the feature points in the second intermediate image to obtain the pixel coordinate information of the feature points in the second intermediate image, where the pixel coordinate information of the feature points in the second intermediate image is the feature point information of the memory module to be installed; determine the actual pose of the memory module to be installed according to the feature point information of the memory module to be installed; adjust the end pose of the robotic arm based on the actual pose and a preset memory module pose to adjust the pose of the memory module to be installed; control the robotic arm with the adjusted end pose to install the memory module to be installed into the designated memory slot corresponding to the memory module to be installed; The robotic arm is configured to, in response to the control of the control device, grasp the memory module to be installed in the target vehicle, adjust the end pose of the robotic arm, and install the memory module to be installed into the designated memory slot; The first vision sensor is configured to, in response to the control of the control device, acquire the memory module grasping image.

12. An electronic device, characterized in that, Comprising: A memory for storing a computer program; A processor for implementing the steps of the memory assembly method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program, when executed by the processor, implements the steps of the memory assembly method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the memory assembly method according to any one of claims 1 to 10.

Images