Socket pose determination method and apparatus, device, and storage medium

Abstract

The present disclosure relates to a socket pose determination method, device, equipment and storage medium. The method comprises: in response to obtaining an original image of a current environment, determining a socket local image of a target socket and key point information of each socket hole in the target socket based on the original image; determining local image feature point information of each socket hole based on the socket local image; and determining a pose of the target socket based on a preset socket standard image, the key point information and the local image feature point information. The present disclosure can accurately estimate the pose of the socket according to the hole geometry characteristics of the socket, without the need to make various pose templates of the target object in advance, thereby saving manpower and resources in the early stage of pose estimation, and without the need to use an R-GBD camera to collect a depth image, thereby reducing the cost of the equipment.

Description

Socket position determination method, device, equipment and storage medium Technical Field

[0001] This disclosure relates to the field of robot perception technology, and in particular to a method, apparatus, device and storage medium for determining the pose of a socket. Background Technology

[0002] In related technologies, object pose estimation can typically be achieved using geometry-based methods, which include feature matching-based algorithms and template matching-based algorithms. Feature matching-based algorithms are only suitable for objects with rich texture features; for objects with less texture, such as sockets, feature matching is ineffective, leading to low pose output accuracy. While template matching-based algorithms can be applied to textureless scenes, they require the creation of multiple pose templates for the target object beforehand, necessitating the use of an R-GBD camera to acquire depth images, resulting in higher equipment costs. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, the present disclosure provides a socket pose determination method, apparatus, device and storage medium to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, a method for determining the pose of a socket is provided, the method comprising:

[0005] In response to acquiring the original image of the current environment, a partial image of the target socket and key point information of each socket in the target socket are determined based on the original image;

[0006] Based on the local image of the socket, determine the local image feature point information of each socket;

[0007] The pose of the target socket is determined based on a preset standard socket image, the key point information, and the local image feature point information.

[0008] In some embodiments, determining a partial image of the target socket and key point information of each socket in the target socket based on the original image includes:

[0009] The original image is input into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, as well as the key point information of each socket in the target socket;

[0010] Based on the position and size of the two-dimensional border, the image within the two-dimensional border is segmented from the original image to obtain a partial image of the socket.

[0011] In some embodiments, determining the local image feature point information of each socket based on the local image of the socket includes:

[0012] The partial image of the socket is processed to obtain a processed image, which is used to distinguish and display each socket and panel of the target socket;

[0013] Based on the processed image, the local image feature point information of each socket is determined.

[0014] In some embodiments, the image processing of the partial image of the socket to obtain a processed image includes:

[0015] Convert the partial image of the socket into a grayscale image;

[0016] Obtain the binarized image of the grayscale image;

[0017] The processed image is determined based on the binarized image.

[0018] In some embodiments, converting the partial image of the socket into a grayscale image includes:

[0019] The local image of the socket is subjected to denoising and / or edge enhancement processing to obtain the processed image;

[0020] The processed image is converted into a grayscale image.

[0021] In some embodiments, obtaining the binarized image of the grayscale image includes:

[0022] Perform histogram equalization on the grayscale image to obtain an equalized image;

[0023] The equalized image is binarized using a threshold segmentation method to obtain the binarized image.

[0024] In some embodiments, the method further includes:

[0025] The equalized image is subjected to edge detection based on a preset edge detection algorithm to obtain an edge-detected image;

[0026] Determining the processed image based on the binarized image includes:

[0027] Perform an OR operation between the edge detection image and the binarized image to obtain an ORed image;

[0028] The OR operation image is subjected to morphological closing operation to obtain the processed image. In some embodiments, determining the local image feature point information of each socket based on the processed image includes:

[0029] Contour extraction is performed on the processed image to obtain at least one first image contour;

[0030] Determine the target image contour in the at least one first image contour whose pixel area satisfies the set conditions;

[0031] The local image feature point information is determined based on the minimum convex bounding contour of the target image contour.

[0032] In some embodiments, determining a target image contour whose pixel area satisfies a set condition in the at least one first image contour includes:

[0033] Obtain a second image contour in the at least one first image contour whose pixel area is greater than or equal to a first set area threshold, wherein the first set area threshold is determined based on a first preset ratio of the area of ​​the processed image;

[0034] Determine the convex bounding contour of the second image contour;

[0035] The contour in the second image contour whose pixel area is greater than or equal to a second set area threshold is determined as the target image contour. The second set area threshold is determined based on a second preset ratio of the pixel area occupied by the convex bounding contour.

[0036] In some embodiments, determining the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information includes:

[0037] The local image feature point information is matched with the socket standard image to obtain the standard image feature point information in the socket standard image;

[0038] Based on the correspondence between the key point information and the standard image feature point information, the corresponding homography matrix is ​​determined;

[0039] The pose of the target socket is obtained by performing singular value decomposition on the homography matrix.

[0040] In some embodiments, determining the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information includes:

[0041] Using the key point information as a reference value, a one-to-one correspondence is made with the standard image feature point information to obtain multiple feature point information pairs;

[0042] The homography matrix is ​​constructed based on the information of the multiple feature points.

[0043] The homography matrix is ​​determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

[0044] According to a second aspect of the present disclosure, a socket pose determination device is provided, the device comprising:

[0045] The local image determination module is used to determine, in response to acquiring the original image of the current environment, a local image of the target socket and key point information of each socket in the target socket based on the original image;

[0046] The local feature point determination module is used to determine the local image feature point information of each socket based on the local image of the socket;

[0047] The socket pose determination module is used to determine the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information.

[0048] In some embodiments, the local image determination module includes:

[0049] The key point information determination unit is used to input the original image into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, as well as the key point information of each socket in the target socket.

[0050] The local image determination unit is used to segment the image within the two-dimensional border from the original image based on the position and size of the two-dimensional border to obtain the local image of the socket.

[0051] In some embodiments, the local feature point determination module includes:

[0052] An image processing and acquisition unit is used to perform image processing on a partial image of the socket to obtain a processed image, which is used to distinguish and display each socket and panel of the target socket.

[0053] The local feature point determination unit is used to determine the local image feature point information of each jack based on the processed image.

[0054] In some embodiments, the image processing and acquisition unit is further configured to:

[0055] Convert the partial image of the socket into a grayscale image;

[0056] Obtain the binarized image of the grayscale image;

[0057] The processed image is determined based on the binarized image.

[0058] In some embodiments, the image processing and acquisition unit is further configured to:

[0059] The local image of the socket is subjected to denoising and / or edge enhancement processing to obtain the processed image;

[0060] The processed image is converted into a grayscale image.

[0061] In some embodiments, the image processing and acquisition unit is further configured to:

[0062] Perform histogram equalization on the grayscale image to obtain an equalized image;

[0063] The equalized image is binarized using a threshold segmentation method to obtain the binarized image.

[0064] In some embodiments, the local feature point determination module further includes:

[0065] The edge detection image acquisition unit is used to perform edge detection on the equalized image based on a preset edge detection algorithm to obtain an edge detection image;

[0066] The image processing and acquisition unit is also used for:

[0067] Perform an OR operation between the edge detection image and the binarized image to obtain an ORed image;

[0068] The processed image is obtained by performing a morphological closing operation on the OR operation image.

[0069] In some embodiments, the local feature point determination unit is further configured to:

[0070] Contour extraction is performed on the processed image to obtain at least one first image contour;

[0071] Determine the target image contour in the at least one first image contour whose pixel area satisfies the set conditions;

[0072] The local image feature point information is determined based on the minimum convex bounding contour of the target image contour.

[0073] In some embodiments, the local feature point determination unit is further configured to:

[0074] Obtain a second image contour in the at least one first image contour whose pixel area is greater than or equal to a first set area threshold, wherein the first set area threshold is determined based on a first preset ratio of the area of ​​the processed image;

[0075] Determine the convex bounding contour of the second image contour;

[0076] The contour in the second image contour whose pixel area is greater than or equal to a second set area threshold is determined as the target image contour. The second set area threshold is determined based on a second preset ratio of the pixel area occupied by the convex bounding contour.

[0077] In some embodiments, the socket pose determination module includes:

[0078] A standard feature point determination unit is used to match the local image feature point information with the socket standard image to obtain the standard image feature point information in the socket standard image;

[0079] The homography matrix determination unit is used to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information;

[0080] The socket pose determination unit is used to perform singular value decomposition on the homography matrix to obtain the pose of the target socket.

[0081] In some embodiments, the homography matrix determining unit is further configured to:

[0082] Using the key point information as a reference value, a one-to-one correspondence is made with the standard image feature point information to obtain multiple feature point information pairs;

[0083] The homography matrix is ​​constructed based on the information of the multiple feature points.

[0084] The homography matrix is ​​determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

[0085] According to a third aspect of the present disclosure, an electronic device is provided, the device comprising:

[0086] Processor and memory used to store computer programs;

[0087] The processor is configured to, when executing the computer program, implement:

[0088] In response to acquiring the original image of the current environment, a partial image of the target socket and key point information of each socket in the target socket are determined based on the original image;

[0089] Based on the local image of the socket, determine the local image feature point information of each socket;

[0090] The pose of the target socket is determined based on a preset standard socket image, the key point information, and the local image feature point information.

[0091] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the program being implemented when executed by a processor:

[0092] In response to acquiring the original image of the current environment, a partial image of the target socket and key point information of each socket in the target socket are determined based on the original image;

[0093] Based on the local image of the socket, determine the local image feature point information of each socket;

[0094] The pose of the target socket is determined based on a preset standard socket image, the key point information, and the local image feature point information.

[0095] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0096] This disclosure, in response to acquiring an original image of the current environment, determines a local image of the target socket and key point information of each socket in the target socket based on the original image, and determines local image feature point information of each socket based on the local image of the socket. Then, based on a preset standard socket image, the key point information, and the local image feature point information, the pose of the target socket is determined. This can achieve accurate socket pose estimation based on the geometric features of the socket's holes without the need to create multiple pose templates for the target object in the early stage, thus saving manpower and resources invested in pose estimation in the early stage, and without the need to use an R-GBD camera to acquire depth images, thus reducing equipment costs.

[0097] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0098] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0099] Figure 1A is a flowchart illustrating a socket pose determination method according to an exemplary embodiment of the present disclosure;

[0100] Figure 1B is a schematic diagram illustrating a standard image of a socket according to an exemplary embodiment of the present disclosure;

[0101] Figure 2 is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to determine a partial image of a target socket and key point information of each socket in the target socket based on the original image;

[0102] Figure 3 is a flowchart illustrating how to determine local image feature point information of each socket based on a local image of the socket according to an exemplary embodiment of the present disclosure;

[0103] Figure 4 is a flowchart illustrating how to perform image processing on a partial image of the socket according to an exemplary embodiment of the present disclosure;

[0104] Figure 5A is a flowchart illustrating how to determine local image feature point information of each jack based on the processed image according to another exemplary embodiment of the present disclosure;

[0105] Figure 5B is a schematic diagram illustrating a socket image according to an exemplary embodiment of the present disclosure;

[0106] Figure 6 is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to determine the pose of the target socket based on a preset socket standard image, the key point information, and the local image feature point information;

[0107] Figure 7 is a flowchart illustrating how to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information according to an exemplary embodiment of the present disclosure;

[0108] Figure 8 is a block diagram illustrating a socket pose determination device according to an exemplary embodiment of the present disclosure;

[0109] Figure 9 is a block diagram illustrating another socket pose determination device according to an exemplary embodiment of the present disclosure;

[0110] Figure 10 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0111] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0112] Figure 1A is a flowchart illustrating a socket pose determination method according to an exemplary embodiment. The method of this embodiment can be executed by a socket pose determination device, which can be configured in an electronic device, such as a server, workstation, personal computer, mobile terminal (e.g., mobile phone, tablet computer), wearable device (e.g., glasses, watch), etc. Specifically, as shown in Figure 1A, the method includes the following steps S101-S103:

[0113] In step S101, in response to acquiring the original image of the current environment, a partial image of the target socket and key point information of each socket in the target socket are determined based on the original image.

[0114] In this embodiment, when the electronic device determines the socket pose, it can acquire the original image of the current environment (e.g., an unprocessed image) based on an image acquisition device such as a monocular camera, and then determine the socket local image of the target socket and the key point information of each socket in the target socket based on the original image.

[0115] The target socket can be a five-hole socket, a three-hole socket, or a two-hole socket; this embodiment does not limit this. The partial image of the socket can be a local area where the target socket is located in the original image. The key point information of each socket in the target socket refers to the information of the key points of each socket in the target socket, such as the coordinates of the key points and the label of the socket to which the key points belong. The key points of the socket are actually a rough estimate of the feature points of the socket (such as the center point of the socket).

[0116] In some embodiments, a partial image of the target socket can be extracted from the original image based on image recognition and image segmentation techniques, thereby identifying key point information of each socket in the target socket. This embodiment does not limit the specific identification method.

[0117] In other embodiments, a partial image of the target socket and key point information of each socket in the target socket can also be determined based on the method shown in Figure 2 below. This embodiment will not be described in detail here.

[0118] In step S102, the local image feature point information of each socket is determined based on the local image of the socket.

[0119] In this embodiment, after determining the local image of the target socket based on the original image, the local image feature point information of each socket can be determined based on the local image of the socket.

[0120] The aforementioned local image feature point information refers to the information of local image feature points of each socket in the target socket located in the local image of the socket, such as the coordinates of the local image feature points and the label of the socket to which the local image feature points belong. The local image feature points of the sockets are actually the refined identification results of the feature points of the sockets (such as the center point of the sockets).

[0121] In some embodiments, the local image feature point information of each socket can also be determined based on the method shown in FIG3 below, which will not be described in detail in this embodiment.

[0122] In step S103, the pose of the target socket is determined based on the preset standard socket image, the key point information, and the local image feature point information.

[0123] In this embodiment, after determining the key point information of each socket in the target socket and the local image feature point information of each socket, the pose of the target socket can be determined based on the preset standard socket image, the key point information and the local image feature point information.

[0124] For example, the aforementioned socket standard image can be a hole distribution diagram of a national standard five-hole socket (as shown in Figure 1B). As shown in Figure 1B, the national standard five-hole socket includes holes 1 to 5. Furthermore, the diagram may also include the dimensional information of the target socket standard (not shown in the diagram).

[0125] In some embodiments, after obtaining the key point information of each socket in the target socket and the local image feature point information of each socket, the pose of the target socket, such as the position and rotation matrix of the target interpolation, can be calculated by calculating the homography matrix and using singular value decomposition (SVD).

[0126] In other embodiments, the pose of the target socket can also be determined based on the method shown in FIG6 below, which will not be described in detail in this embodiment.

[0127] As described above, the method of this embodiment, in response to acquiring the original image of the current environment, determines the local image of the target socket and the key point information of each socket in the target socket based on the original image, and determines the local image feature point information of each socket based on the local image of the socket. Then, based on the preset standard image of the socket, the key point information and the local image feature point information, the pose of the target socket is determined. This can achieve accurate socket pose estimation based on the geometric features of the socket's holes, without the need to create multiple pose templates of the target object in the early stage. Therefore, it can save manpower and resources invested in pose estimation in the early stage, and it does not require the use of an R-GBD camera to acquire depth images, thus reducing equipment costs.

[0128] Figure 2 is a flowchart illustrating, according to an exemplary embodiment of the present disclosure, how to determine a partial image of a target socket and key point information of each socket in the target socket based on the original image;

[0129] This embodiment, based on the above embodiment, takes as an example how to determine the local image of the target socket and the key point information of each socket in the target socket based on the original image.

[0130] As shown in Figure 3, the step S101 above, which involves determining the local image of the target socket and the key point information of each socket in the target socket based on the original image, may include the following steps S201-S202:

[0131] In step S201, the original image is input into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, as well as the key point information of each socket in the target socket.

[0132] In this embodiment, a deep learning model for recognizing the two-dimensional border of the socket and the key point information of the socket can be pre-trained based on sample images. Then, when the original image img_ori of the current environment is obtained, the original image img_ori can be input into the pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, as well as the key point information of each socket in the target socket.

[0133] It is worth noting that the specific type of the deep learning model mentioned above can be selected based on actual business needs, such as selecting YOLOv8 deep neural network, etc. This embodiment does not limit this.

[0134] This embodiment uses a deep learning model to detect key points of the target socket and jack, which can generate reference points for the jack position (i.e., the key points of the jack) while removing background interference from the original image, thereby reducing the complexity of subsequent key point matching calculations.

[0135] In step S202, based on the position and size of the two-dimensional border, the image within the two-dimensional border is segmented from the original image to obtain a partial image of the socket.

[0136] In this embodiment, after obtaining the position and size of the two-dimensional border of the target socket, and the key point information of each socket in the target socket, the image within the two-dimensional border can be segmented from the original image based on the position and size of the two-dimensional border to obtain the socket local image img_tgt, wherein the socket local image includes the key point information P = {p1p2…p} of each socket. n}. Where n is the number of key information points, i.e., the number of holes in the target socket. For example, if the target socket is a standard five-hole socket, then n = 5. Label i x is the label of the socket to which the i-th key point belongs. i ,y i Let be the coordinates of the i-th key point in the local image img_tgt of the socket.

[0137] As described above, this embodiment obtains the position and size of the two-dimensional bounding box of the target socket and the key point information of each socket in the target socket by inputting the original image into a pre-trained deep learning model. Based on the position and size of the two-dimensional bounding box, the image within the two-dimensional bounding box is segmented from the original image to obtain a local image of the socket. This allows for accurate determination of the local image of the target socket and the key point information of each socket in the target socket. This lays the foundation for subsequent determination of the local image feature point information of each socket based on the local image of the socket, and for determining the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information. This enables accurate socket pose estimation, saving manpower and resources invested in pose estimation in the early stages and reducing equipment costs.

[0138] Figure 3 is a flowchart illustrating how to determine the local image feature point information of each socket based on a local image of the socket according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the local image feature point information of each socket based on a local image of the socket as an example for illustrative explanation.

[0139] As shown in Figure 3, the step S102 above, which involves determining the local image feature point information of each socket based on the local image of the socket, may include the following steps S301-S302:

[0140] In step S301, image processing is performed on the partial image of the socket to obtain a processed image.

[0141] In this embodiment, after determining the local image of the target socket based on the original image, image processing can be performed on the local image of the socket to obtain a processed image.

[0142] The processed image can be used to distinguish and display the various sockets and the panel of the target socket.

[0143] For example, the aforementioned partial image of the socket can be grayscaled and binarized to obtain the processed image.

[0144] In some embodiments, the image processing method for the partial image of the socket described above can be referred to the embodiment shown in FIG4 below, which will not be described in detail here.

[0145] In step S302, local image feature point information of each socket is determined based on the processed image.

[0146] In this embodiment, after processing the local image of the socket to obtain a processed image, the local image feature point information of each socket of the target socket can be determined in the processed image based on the geometric features of the socket positions.

[0147] In some embodiments, the method for determining the local image feature point information of each socket can be referred to the embodiment shown in FIG5A below, which will not be described in detail here.

[0148] As described above, this embodiment obtains a processed image by processing a partial image of the socket. The processed image is used to distinguish and display each socket and panel of the target socket, and the local image feature point information of each socket is determined based on the processed image. This allows for accurate determination of the local image feature point information of each socket based on image processing, which lays the foundation for subsequent determination of the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information. This enables accurate socket pose estimation, saves manpower and resources invested in pose estimation in the early stages, and reduces equipment costs.

[0149] Figure 4 is a flowchart illustrating how to perform image processing on a partial image of the socket according to an exemplary embodiment of the present disclosure;

[0150] This embodiment, based on the above embodiments, provides an exemplary description of how to perform image processing on a partial image of the socket.

[0151] As shown in Figure 4, the image processing of the partial image of the socket described in step S301 above to obtain a processed image may include the following steps S401-S403:

[0152] In step S401, the partial image of the socket is converted into a grayscale image.

[0153] In this embodiment, after obtaining a partial image of the socket, the partial image of the socket can be converted into a grayscale image.

[0154] In some embodiments, the local image of the socket can be denoised and / or edge-enhanced to obtain a processed image, which can then be converted into a grayscale image.

[0155] For example, Gaussian filtering can be applied to the local image img_tgt of the socket (which is an RGB image) to remove noise and obtain a filtered image. Then, the Laplacian operator can be used to enhance the edges of the filtered image to obtain an edge-enhanced RGB image. Finally, the edge-enhanced RGB image can be converted into a grayscale image based on a weighted average method.

[0156] In step S402, the binarized image of the grayscale image is obtained.

[0157] In this embodiment, after converting the partial image of the socket into a grayscale image, a binarized image of the grayscale image can be obtained.

[0158] For example, histogram equalization can be performed on the grayscale image to obtain an equalized image, which improves contrast by evenly distributing the grayscale values. Furthermore, considering that the obvious features of the socket are mainly on the socket holes, the most obvious feature being that the brightness inside the holes is lower than the average brightness of the surrounding environment, this embodiment can use threshold segmentation to binarize the equalized image to obtain the binarized image. Exemplarily, the threshold T in this threshold segmentation method can be calculated as shown in equation (4-1):

[0159] T=mean(top_k(sort(img_tgt)) / ; (4-1)

[0160] In the above formula, `sort()` means arranging all pixels of the image in ascending order, `top_k()` means taking the first k data values, and `mean()` means calculating the average value of the data. That is to say, in this embodiment, all pixel values ​​in the equalized image are sorted in ascending order, and then the average value of the k pixels with the lowest grayscale is calculated as the segmentation threshold T, thereby effectively distinguishing low-brightness jacks from high-brightness panels.

[0161] In step S403, the processed image is determined based on the binarized image.

[0162] In this embodiment, after obtaining the binarized image of the grayscale image, the processed image can be determined based on the binarized image.

[0163] For example, after obtaining the binarized image described above, the binarized image can be used as the processed image.

[0164] In other embodiments, considering that the binarization method using only threshold segmentation is prone to edge distortion in strong lighting conditions, affecting the accuracy of feature point extraction, this embodiment can perform edge detection on the equalized image based on a preset edge detection algorithm to obtain an edge-detected image. Based on this, the edge-detected image and the binarized image can be ORed to obtain an OR image, and morphological closing processing (i.e., supplementing edge details) can be performed on the OR image to obtain the final binarized image img_thr, which is the processed image described above.

[0165] It is worth noting that the aforementioned preset edge detection algorithm can be selected based on actual business needs, such as the Canny edge detection algorithm, etc. This embodiment does not limit this selection. Specifically, the Canny edge detection algorithm was proposed by John F. Canny in 1986. Its core idea is to smooth the image using a filter to remove noise, then use a gradient operator to detect edges in the image, and finally determine the final edge information through non-maximum suppression and double thresholding. Because this algorithm can effectively extract edge information from the image, it can improve the quality of the processed image, enabling it to more accurately distinguish and display the various sockets and panels of the target socket.

[0166] As described above, this embodiment converts the local image of the socket into a grayscale image and obtains a binarized image of the grayscale image. Then, based on the binarized image, it determines the processed image. This allows for the acquisition of processed images used to distinguish and display each socket and panel of the target socket. Subsequently, it enables the determination of local image feature point information of each socket based on the processed image, ensuring the accuracy of determining the local image feature point information of each socket and improving the accuracy of subsequent socket pose determination.

[0167] Figure 5A is a flowchart illustrating how to determine local image feature point information of each socket based on the processed image according to another exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine local image feature point information of each socket based on the processed image as an example for illustrative explanation.

[0168] As shown in Figure 5A, the step S302 above, which involves determining the local image feature point information of each socket based on the processed image, may include the following steps S501-S503:

[0169] In step S501, contour extraction is performed on the processed image to obtain at least one first image contour.

[0170] In this embodiment, after obtaining the processed image img_thr, contour extraction can be performed on the processed image to obtain at least one first image contour.

[0171] Specifically, contour extraction algorithms from related technologies can be used to extract contours from the processed image to obtain at least one foreground contour in the image (named the first image contour for easy distinction).

[0172] In step S502, a target image contour whose pixel area satisfies a set condition is determined in the at least one first image contour.

[0173] In this embodiment, after obtaining at least one first image contour, in order to remove interfering contours, a target image contour whose pixel area meets the set conditions can be determined among the at least one first image contour.

[0174] For example, a second image contour can be obtained from at least one of the aforementioned first image contours, where the pixel area occupied is greater than or equal to a first preset area threshold. This first preset area threshold can be determined based on a first preset proportion of the area of ​​the processed image. For instance, the first preset area threshold can be set to 2% of the area of ​​the processed image (i.e., the first preset proportion is 2%). That is, contours from at least one of the first image contours whose pixel area is less than 2% of the area of ​​the processed image are removed, resulting in the remaining first image contours, i.e., the aforementioned second image contour. As an example, the algorithm for determining the pixel area occupied by the aforementioned first image contour can be based on a breadth-first search (BFS) algorithm.

[0175] Furthermore, considering that the standard socket hole is fixed as a rectangle, its projection in the image must be a convex polygon. Therefore, this embodiment can filter severely distorted contours by judging the ratio of the original contour to the area enclosed by the convex shape. Specifically, the convex shape enclosed contour of the second image contour can be determined, wherein the convex shape enclosed contour can specifically be the minimum convex shape enclosed contour of the second image contour. Specifically, when the second image contour is a convex shape, the minimum convex shape enclosed contour of the second image contour is the second image contour itself; while when the second image contour is a non-convex shape, the Sklansky algorithm can be applied to the second image contour to obtain the minimum convex shape enclosed contour of the contour.

[0176] Based on this, the contours in the second image contour whose pixel area is greater than or equal to a second preset area threshold can be determined as the target image contour. The second preset area threshold can be determined based on a second preset proportion of the pixel area occupied by the convex bounding contour. For example, the second preset area threshold can be set to 80% of the area of ​​the processed image (i.e., the second preset proportion is 80%). In other words, contours in the second image contour whose pixel area is less than 80% of the area of ​​the processed image are removed to obtain the remaining second image contour, i.e., the aforementioned target image contour.

[0177] In step S503, the local image feature point information is determined based on the minimum convex bounding contour of the target image contour.

[0178] In this embodiment, after determining that the target image contour whose pixel area meets the set conditions in the at least one first image contour, the local image feature point information can be determined based on the centroid of the largest inscribed quadrilateral of the smallest convex bounding contour of the target image contour.

[0179] For example, FIG5B is a schematic diagram of a socket image according to an exemplary embodiment of the present disclosure; as shown in FIG5B, after determining the target image contour 510, the minimum convex bounding contour 520 of the target image contour 510 can be determined, and then the maximum inscribed quadrilateral 530 of the minimum convex bounding contour 520 can be determined, thereby determining the centroid 540 of the maximum inscribed quadrilateral 530 as the local image feature point information.

[0180] As described above, this embodiment extracts contours from the processed image to obtain at least one first image contour, and determines a target image contour whose pixel area meets a set condition. Then, based on the minimum convex bounding contour of the target image contour, the local image feature point information is determined. This can accurately determine the local image feature point information based on the geometric features of the socket holes, improve the accuracy of determining the local image feature point information of each socket hole, and thus achieve accurate determination of the socket pose in the subsequent process.

[0181] Figure 6 is a flowchart illustrating how to determine the pose of the target socket based on a preset socket standard image, the key point information, and the local image feature point information, according to an exemplary embodiment of the present disclosure. This embodiment, based on the above embodiment, takes how to determine the pose of the target socket based on a preset socket standard image, the key point information, and the local image feature point information as an example for illustrative explanation.

[0182] As shown in Figure 6, the determination of the pose of the target socket based on the preset socket standard image, the key point information, and the local image feature point information in step S103 above may include the following steps S601-S603:

[0183] In step S601, the local image feature point information is matched with the socket standard image to obtain the standard image feature point information in the socket standard image.

[0184] In this embodiment, after obtaining the local image feature point information, the local image feature point information can be matched with a preset socket standard image to obtain the standard image feature point information in the socket standard image.

[0185] Taking the target socket as a national standard five-hole socket as an example, after obtaining the five local image feature point information of the target socket, the five local image feature point information can be matched to the standard socket image to obtain the five standard image feature point information in the standard socket image.

[0186] In step S602, the corresponding homography matrix is ​​determined based on the correspondence between the key point information and the standard image feature point information.

[0187] In this embodiment, after obtaining the standard image feature point information in the standard image of the socket, the corresponding homography matrix (i.e., the matrix used to characterize the correspondence between the key point information and the standard image feature point information) can be determined based on the correspondence between the key point information and the standard image feature point information.

[0188] In some embodiments, after obtaining the standard image feature point information in the above-mentioned socket standard image, the homography matrix can be determined based on the homography matrix determination method in related technologies. This embodiment does not limit the specific determination method.

[0189] In other embodiments, the method for determining the homography matrix described above can also be found in the embodiment shown in Figure 7 below, which will not be described in detail here.

[0190] In step S603, singular value decomposition is performed on the homography matrix to obtain the pose of the target socket.

[0191] In this embodiment, after determining the corresponding homography matrix, singular value decomposition (SVD) can be performed on the homography matrix to obtain the pose of the target socket.

[0192] As described above, this embodiment obtains standard image feature point information in the standard image of the socket by matching the local image feature point information with the standard image of the socket. Based on the correspondence between the key point information and the standard image feature point information, the corresponding homography matrix is ​​determined. Then, singular value decomposition is performed on the homography matrix to obtain the pose of the target socket. This can achieve accurate socket pose estimation based on homography matrix and singular value decomposition (SVD) algorithm without the need to create multiple pose templates of the target object in the early stage. Therefore, it can save manpower and resources invested in pose estimation in the early stage. Moreover, it does not require the use of R-GBD camera to collect depth images, thus reducing equipment costs.

[0193] Figure 7 is a flowchart illustrating how to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information according to an exemplary embodiment of the present disclosure; this embodiment is based on the above embodiment and takes how to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information as an example for illustrative explanation.

[0194] As shown in Figure 7, determining the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information in step S602 above may include the following steps S701-S703:

[0195] In step S701, the key point information is used as a reference value and matched one-to-one with the standard image feature point information to obtain multiple feature point information pairs.

[0196] In this embodiment, in order to determine the above homography matrix, the key point information can be used as a reference value and matched one-to-one with the standard image feature point information to obtain multiple feature point information pairs.

[0197] For example, for each key point, the standard image feature point with the smallest Euclidean distance can be selected to pair the two, resulting in multiple feature point information pairs.

[0198] In step S702, the homography matrix is ​​constructed based on the multiple feature point information.

[0199] In this embodiment, after obtaining multiple feature point information pairs, the homography matrix can be constructed based on the multiple feature point information pairs.

[0200] Taking the target socket as a standard five-hole socket as an example, based on the above step S701, five feature point information pairs can be obtained. In this embodiment, all combinations can be traversed, that is, four feature point information pairs are selected from the five (the homography matrix requires four reference points; since there are five standard image feature points in this embodiment, random sampling can be changed to traversing all cases, i.e., taking out four standard image feature points to obtain the corresponding four feature point information pairs), the homography matrix is ​​calculated, and then the remaining fifth feature point information pair is used for verification. If the verification passes, the homography matrix can be constructed using the least squares method for these five feature point information pairs, and then the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix is ​​solved to obtain the homography matrix. If the verification fails, four new feature point information pairs are selected from the above five feature point information pairs, and the above process is repeated until the verification passes using the remaining fifth feature point information pair. If, after traversing all combinations, no combination that can pass verification is found, then the combination with the better verification result (i.e., 4 feature point information pairs) can be used to calculate the homography matrix.

[0201] For example, after extracting four feature point pairs, the following equation (7-1) can be constructed:

[0202]

[0203] In the above formula, x1 is a standard image feature point, and x2 is a key point that matches x1. This is an estimate of the homography matrix. Expanding equation (7-1) above, we obtain equation (7-2):

[0204]

[0205] In the above formula, the coordinates of pixel x1 are (u1, v1) and the coordinates of pixel x2 are (u2, v2).

[0206] Substituting the third constraint in equation (7-2) into the first two constraints, we get:

[0207]

[0208]

[0209] Simplifying the above two equations, we get:

[0210] h1u1+h2v1+h3-h7u1u2+h8v1u2+h9u2=0; (7-5)

[0211] h4u1+h5v1+h6-h7u1v2+h8v1v2+h9v2=0; (7-6)

[0212] Therefore, a single feature point information pair can provide two equations. The homography matrix has 8 degrees of freedom, so substituting four feature point information pairs will allow us to solve for the matrix.

[0213] Based on this, the remaining information of the 5th feature point can be used for verification. That is, the projection point of the 5th feature point can be calculated using the obtained homography matrix, and the Euclidean distance between it and the 5th feature point in the standard image can be calculated as the one-sided projection error.

[0214]

[0215] Furthermore, if ε is greater than the threshold, four feature point information pairs are reselected until the unilateral projection error is less than or equal to the threshold. At this point, a fifth feature point information pair can be added, and the following equation (7-8) can be reconstructed using the least squares method:

[0216]

[0217] It is worth noting that in related technologies, solving the homography matrix only requires 4 feature point information pairs, while this embodiment uses 5, which can further reduce the error.

[0218] However, if no combination that satisfies the condition (i.e., the one-sided projection error is less than or equal to the threshold) is found after all combinations have been traversed, then the combination of feature points with the smallest Euclidean distance can be selected to calculate the homography matrix.

[0219] In step S703, the homography matrix is ​​determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

[0220] In this embodiment, after constructing the homography matrix based on the multiple feature point information pairs, the homography matrix can be determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

[0221] In other words, after constructing the above homography matrix, the coefficient matrix A of the homography matrix can be solved. T The smallest eigenvalue of A is then used to determine the eigenvector corresponding to that smallest eigenvalue as the homography matrix.

[0222] As described above, this embodiment uses the key point information as a reference value and maps it one-to-one with the standard image feature point information to obtain multiple feature point information pairs. Based on these multiple feature point information pairs, the homography matrix is ​​constructed. Then, based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix, the homography matrix is ​​determined. This allows for a reasonable and accurate determination of the homography matrix, which in turn enables the subsequent determination of the target socket's pose based on the homography matrix. This improves the accuracy of socket pose estimation, saves manpower and resources invested in pose estimation in the early stages, and reduces equipment costs.

[0223] Figure 8 is a block diagram illustrating a socket pose determination device according to an exemplary embodiment of the present disclosure. The device of this embodiment can be configured in electronic devices, such as robots, or as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as glasses, watches, etc.) serving as robot control devices. Specifically, as shown in Figure 8, the device may include: a local image determination module 110, a local feature point determination module 120, and a socket pose determination module 130, wherein:

[0224] The local image determination module 110 is used to determine, in response to acquiring the original image of the current environment, a local image of the target socket and key point information of each socket in the target socket based on the original image;

[0225] The local feature point determination module 120 is used to determine the local image feature point information of each socket based on the local image of the socket;

[0226] The socket pose determination module 130 is used to determine the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information.

[0227] As described above, the device in this embodiment, in response to acquiring an original image of the current environment, determines a local image of the target socket and key point information of each socket in the target socket based on the original image, and determines local image feature point information of each socket based on the local image of the socket. Then, based on a preset standard socket image, the key point information and the local image feature point information, the pose of the target socket is determined. This can achieve accurate socket pose estimation based on the geometric features of the socket's holes, without the need to create multiple pose templates for the target object in the early stage. Therefore, it can save manpower and resources invested in pose estimation in the early stage, and it does not require the use of an R-GBD camera to acquire depth images, thus reducing equipment costs.

[0228] Figure 9 is a block diagram illustrating another socket pose determination device according to an exemplary embodiment of the present disclosure. The device of this embodiment can be configured in electronic devices, such as robots, or as servers, workstations, personal computers, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as glasses, watches, etc.) serving as robot control devices. The local image determination module 210, local feature point determination module 220, and socket pose determination module 230 have the same functions as the local image determination module 110, local feature point determination module 120, and socket pose determination module 130 in the embodiment shown in Figure 8 above, and will not be described again here.

[0229] As shown in Figure 9, the local image determination module 210 may include:

[0230] The key point information determination unit 211 is used to input the original image into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, as well as the key point information of each socket in the target socket.

[0231] The local image determination unit 212 is used to segment the image within the two-dimensional border from the original image based on the position and size of the two-dimensional border to obtain the local image of the socket.

[0232] In some embodiments, the local feature point determination module 220 may include:

[0233] The image processing and acquisition unit 221 is used to perform image processing on the partial image of the socket to obtain a processed image, which is used to distinguish and display each socket and panel of the target socket.

[0234] The local feature point determination unit 222 is used to determine the local image feature point information of each jack based on the processed image.

[0235] In some embodiments, the image acquisition unit 221 can also be used for:

[0236] Convert the partial image of the socket into a grayscale image;

[0237] Obtain the binarized image of the grayscale image;

[0238] The processed image is determined based on the binarized image.

[0239] In some embodiments, the image acquisition unit 221 is further configured to:

[0240] The local image of the socket is subjected to denoising and / or edge enhancement processing to obtain the processed image;

[0241] The processed image is converted into a grayscale image.

[0242] In some embodiments, the image acquisition unit 221 is further configured to:

[0243] Perform histogram equalization on the grayscale image to obtain an equalized image;

[0244] The equalized image is binarized using a threshold segmentation method to obtain the binarized image.

[0245] In some embodiments, the local feature point determination module 220 may further include:

[0246] The edge detection image acquisition unit 223 is used to perform edge detection on the equalized image based on a preset edge detection algorithm to obtain an edge detection image;

[0247] Furthermore, the image processing and acquisition unit 221 can also be used for:

[0248] Perform an OR operation between the edge detection image and the binarized image to obtain an ORed image;

[0249] The processed image is obtained by performing a morphological closing operation on the OR operation image.

[0250] In some embodiments, the local feature point determination unit 222 can also be used for:

[0251] Contour extraction is performed on the processed image to obtain at least one first image contour;

[0252] Determine the target image contour in the at least one first image contour whose pixel area satisfies the set conditions;

[0253] The local image feature point information is determined based on the minimum convex bounding contour of the target image contour.

[0254] In some embodiments, the local feature point determination unit 222 can also be used for:

[0255] Obtain a second image contour in the at least one first image contour whose pixel area is greater than or equal to a first set area threshold, wherein the first set area threshold is determined based on a first preset ratio of the area of ​​the processed image;

[0256] Determine the convex bounding contour of the second image contour;

[0257] The contour in the second image contour whose pixel area is greater than or equal to a second set area threshold is determined as the target image contour. The second set area threshold is determined based on a second preset ratio of the pixel area occupied by the convex bounding contour.

[0258] In some embodiments, the socket orientation determination module 230 may include:

[0259] The standard feature point determination unit 231 is used to match the local image feature point information with the socket standard image to obtain the standard image feature point information in the socket standard image;

[0260] The homography matrix determination unit 232 is used to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information;

[0261] The socket pose determination unit 233 is used to perform singular value decomposition on the homography matrix to obtain the pose of the target socket.

[0262] In some embodiments, the homography matrix determining unit 232 is further configured to:

[0263] Using the key point information as a reference value, a one-to-one correspondence is made with the standard image feature point information to obtain multiple feature point information pairs;

[0264] The homography matrix is ​​constructed based on the information of the multiple feature points.

[0265] The homography matrix is ​​determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

[0266] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0267] Figure 10 is a block diagram illustrating an electronic device according to an exemplary embodiment. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0268] Referring to FIG10, device 900 may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, input / output (I / O) interface 912, sensor component 914, and communication component 916.

[0269] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the socket pose determination method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0270] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0271] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0272] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0273] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0274] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0275] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 can detect the on / off state of device 900, the relative positioning of components such as the display panel and keypad of device 900, changes in the position of device 900 or a component of device 900, the presence or absence of user contact with device 900, the orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0276] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0277] In an exemplary embodiment, device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the socket pose determination method described above.

[0278] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of device 900 to complete the socket pose determination method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0279] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0280] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for determining the position and orientation of a socket, characterized in that, The method includes: in response to acquiring an original image of the current environment, determining a partial image of the target socket and key point information of each socket in the target socket based on the original image; determining local image feature point information of each socket based on the partial image of the socket; determining the pose of the target socket based on a preset standard image of the socket, the key point information, and the local image feature point information; the step of determining the local image feature point information of each socket based on the partial image of the socket includes: performing image processing on the partial image of the socket to obtain a processed image, the processed image being used to distinguish and display each socket and panel of the target socket; performing contour extraction on the processed image to obtain at least one first image contour; determining a target image contour whose pixel area satisfies a set condition among the at least one first image contour; and determining the local image feature point information based on the largest inscribed quadrilateral of the smallest convex bounding contour of the target image contour.

2. The method according to claim 1, characterized in that, The step of determining the local image of the target socket and the key point information of each socket in the target socket based on the original image includes: inputting the original image into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket and the key point information of each socket in the target socket; and segmenting the image within the two-dimensional border from the original image based on the position and size of the two-dimensional border to obtain the local image of the socket.

3. The method according to claim 1, characterized in that, The step of processing the partial image of the socket to obtain a processed image includes: converting the partial image of the socket into a grayscale image; obtaining a binarized image of the grayscale image; and determining the processed image based on the binarized image.

4. The method according to claim 3, characterized in that, The step of converting the partial image of the socket into a grayscale image includes: performing noise reduction and / or edge enhancement processing on the partial image of the socket to obtain a processed image; and converting the processed image into a grayscale image.

5. The method according to claim 3, characterized in that, The step of obtaining the binarized image of the grayscale image includes: performing histogram equalization on the grayscale image to obtain an equalized image; and performing binarization processing on the equalized image using a threshold segmentation method to obtain the binarized image.

6. The method according to claim 5, characterized in that, The method further includes: performing edge detection on the equalized image based on a preset edge detection algorithm to obtain an edge detection image; determining the processed image based on the binarized image includes: performing an OR operation on the edge detection image and the binarized image to obtain an OR operation image; performing morphological closing operation on the OR operation image to obtain the processed image.

7. The method according to claim 1, characterized in that, The step of determining the target image contour in the at least one first image contour whose pixel area meets the set conditions includes: obtaining a second image contour in the at least one first image contour whose pixel area is greater than or equal to a first set area threshold, wherein the first set area threshold is determined based on a first preset ratio of the area of ​​the processed image; determining the convex bounding contour of the second image contour; and determining the contour in the second image contour whose pixel area is greater than or equal to a second set area threshold as the target image contour, wherein the second set area threshold is determined based on a second preset ratio of the pixel area occupied by the convex bounding contour.

8. The method according to claim 1, characterized in that, The step of determining the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information includes: matching the local image feature point information with the standard socket image to obtain standard image feature point information in the standard socket image; determining the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information; and performing singular value decomposition on the homography matrix to obtain the pose of the target socket.

9. The method according to claim 8, characterized in that, The step of determining the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information includes: using the key point information as a reference value and making a one-to-one correspondence with the standard image feature point information to obtain multiple feature point information pairs; constructing the homography matrix based on the multiple feature point information pairs; and determining the homography matrix based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

10. A socket position determination device, characterized in that, The device includes: a local image determination module, configured to, in response to acquiring an original image of the current environment, determine a local image of a target socket and key point information of each socket in the target socket based on the original image; a local feature point determination module, configured to determine local image feature point information of each socket based on the local image of the socket; and a socket pose determination module, configured to determine the pose of the target socket based on a preset standard socket image, the key point information, and the local image feature point information; the local feature point determination module includes: an image processing acquisition unit, configured to perform image processing on the local image of the socket to obtain a processed image, the processed image being used to distinguish and display each socket and panel of the target socket; and a local feature point determination unit, configured to: extract contours from the processed image to obtain at least one first image contour; determine a target image contour whose pixel area satisfies a set condition among the at least one first image contour; and determine the local image feature point information based on the largest inscribed quadrilateral of the smallest convex bounding contour of the target image contour.

11. The apparatus according to claim 10, characterized in that, The local image determination module includes: a key point information determination unit, used to input the original image into a pre-trained deep learning model to obtain the position and size of the two-dimensional border of the target socket, and the key point information of each socket in the target socket; and a local image determination unit, used to segment the image within the two-dimensional border from the original image based on the position and size of the two-dimensional border to obtain a local image of the socket.

12. The apparatus according to claim 10, characterized in that, The image processing acquisition unit is further configured to: convert the local image of the socket into a grayscale image; acquire a binarized image of the grayscale image; and determine the processed image based on the binarized image.

13. The apparatus according to claim 12, characterized in that, The image processing acquisition unit is further configured to: perform noise reduction and / or edge enhancement processing on the local image of the socket to obtain a processed image; and convert the processed image into a grayscale image.

14. The apparatus according to claim 12, characterized in that, The image processing acquisition unit is further configured to: perform histogram equalization on the grayscale image to obtain an equalized image; and perform binarization on the equalized image using a threshold segmentation method to obtain the binarized image.

15. The apparatus according to claim 14, characterized in that, The local feature point determination module further includes: an edge detection image acquisition unit, used to perform edge detection on the equalized image based on a preset edge detection algorithm to obtain an edge detection image; the processed image acquisition unit is further used to: perform an OR operation on the edge detection image and the binarized image to obtain an OR operation image; and perform morphological closing operation processing on the OR operation image to obtain the processed image.

16. The apparatus according to claim 10, characterized in that, The local feature point determination unit is further configured to: acquire a second image contour in the at least one first image contour whose pixel area is greater than or equal to a first preset area threshold, wherein the first preset area threshold is determined based on a first preset ratio of the area of ​​the processed image; determine the convex bounding contour of the second image contour; and determine the contour in the second image contour whose pixel area is greater than or equal to a second preset area threshold as the target image contour, wherein the second preset area threshold is determined based on a second preset ratio of the pixel area occupied by the convex bounding contour.

17. The apparatus according to claim 10, characterized in that, The socket pose determination module includes: a standard feature point determination unit, used to match the local image feature point information with the socket standard image to obtain standard image feature point information in the socket standard image; a homography matrix determination unit, used to determine the corresponding homography matrix based on the correspondence between the key point information and the standard image feature point information; and a socket pose determination unit, used to perform singular value decomposition on the homography matrix to obtain the pose of the target socket.

18. The apparatus according to claim 17, characterized in that, The homography matrix determination unit is further configured to: use the key point information as a reference value and make a one-to-one correspondence with the standard image feature point information to obtain multiple feature point information pairs; and construct the homography matrix based on the multiple feature point information pairs. The homography matrix is ​​determined based on the eigenvector corresponding to the smallest eigenvalue of the coefficient matrix of the homography matrix.

19. An electronic device, characterized in that, The device includes a processor and a memory for storing a computer program; wherein the processor is configured to, when executing the computer program, perform the following actions: in response to acquiring an original image of the current environment, determine a partial image of a target socket and key point information of each socket in the target socket based on the original image; determine local image feature point information of each socket based on the partial image of the socket; determine the pose of the target socket based on a preset standard image of the socket, the key point information, and the local image feature point information; the step of determining the local image feature point information of each socket based on the partial image of the socket includes: performing image processing on the partial image of the socket to obtain a processed image, the processed image being used to distinguish and display each socket and panel of the target socket; performing contour extraction on the processed image to obtain at least one first image contour; determining a target image contour whose pixel area satisfies a set condition among the at least one first image contour; and determining the local image feature point information based on the largest inscribed quadrilateral of the smallest convex bounding contour of the target image contour.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the following: in response to acquiring the original image of the current environment, it determines the local image of the target socket and the key point information of each socket in the target socket based on the original image; it determines the local image feature point information of each socket based on the local image of the socket; and it determines the pose of the target socket based on a preset standard socket image, the key point information and the local image feature point information. The step of determining the local image feature point information of each socket based on the local image of the socket includes: performing image processing on the local image of the socket to obtain a processed image, the processed image being used to distinguish and display each socket and panel of the target socket; extracting contours from the processed image to obtain at least one first image contour; determining a target image contour whose pixel area satisfies a set condition among the at least one first image contour; and determining the local image feature point information based on the maximum inscribed quadrilateral of the minimum convex bounding contour of the target image contour.

Images