Connector hole position detection method, device and storage medium

Abstract

The present application discloses a hole position detection method, device and storage medium for a connector. The hole position detection method for the connector includes: updating the hole position tracking result corresponding to the previous frame image based on the position change of the connector to obtain the hole position tracking result corresponding to the current frame image; and performing image recognition processing on the current frame image to obtain the hole position recognition result; calculating the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation; and obtaining the connection hole position corresponding to the connector based on the optimized hole position tracking result corresponding to the current frame image. The hole position tracking result can be optimized according to the hole position recognition result to improve the accuracy of the hole position tracking result, and the optimized hole position tracking result can avoid omissions in hole position detection and obtain a hole position detection result with high detection rate and high accuracy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and storage medium for detecting the hole position of a connector. Background Art

[0002] Electrical connectors are important components in electrical interconnection systems. They transmit signals or energy from one end of a cable conductor to the other end by bridging. Therefore, the correctness, reliability and safety of the connection of electrical connectors are crucial to the smooth operation of the entire electrical interconnection system.

[0003] At present, the mainstream electrical connector termination method in industrial operations is mainly manual, and hole location search and detection rely on naked eye visual inspection, which is not only inefficient and poorly accurate, but also relatively labor-intensive. Although with the development of computer and artificial intelligence technology, many intelligent means have emerged in the field of electrical connector termination operations, such as industrial robot automated termination, augmented reality intelligent assisted assembly and other technologies, but there are still problems of poor accuracy, such as partial hole location occlusion caused by the harsh conditions of the on-site industrial operation environment, defects in the hole location detection algorithm, etc., resulting in inaccurate or missed detection of some holes, affecting the accuracy of termination. Summary of the invention

[0004] In order to solve the above technical problems, the present application at least provides a connector hole position detection method, device and storage medium.

[0005] The first aspect of the present application provides a method for detecting the hole position of a connector, the method comprising: obtaining a current frame image of the connector, a previous frame image corresponding to the current frame image, and a hole position tracking result corresponding to the previous frame image; wherein the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image, and the hole position tracking result includes the positions of all connecting holes of the connector obtained by tracking the position change of the connector; obtaining the position change amount of the connector based on the image change between the current frame image and the previous frame image, updating the hole position tracking result corresponding to the previous frame image based on the position change amount of the connector, and obtaining the hole position tracking result corresponding to the current frame image; and performing image recognition processing on the current frame image to obtain a hole position recognition result, the hole position recognition result including the positions of the connecting holes detectable by the current frame image; calculating the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation; obtaining the connection hole position corresponding to the connector based on the optimized hole position tracking result corresponding to the current frame image

[0006] In one embodiment, calculating the connection hole position deviation between the hole position tracking result and the hole position recognition result corresponding to the current frame image includes: calculating the position distance between each connection hole in the hole position tracking result and each connection hole in the hole position recognition result; associating the connection holes in the hole position recognition result with the connection holes in the hole position tracking result having the smallest position distance to obtain the minimum adjacent holes corresponding to each connection hole in the hole position tracking result; summing the position distances between each connection hole in the hole position tracking result and its corresponding associated minimum adjacent hole, and using the summed calculation result as the connection hole position deviation between the hole position tracking result and the hole position recognition result corresponding to the current frame image.

[0007] In one embodiment, optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation includes: iteratively updating the position of each connection hole in the hole position tracking result corresponding to the current frame image with the goal of minimizing the connection hole position deviation.

[0008] In one embodiment, the connector is provided with a position marker, the hole tracking result includes the position of the position marker obtained by tracking the position change of the connector, and the hole recognition result includes the position of the position marker detected by the current frame image; with the goal of minimizing the position deviation of the connecting holes, the position of each connecting hole in the hole tracking result corresponding to the current frame image is iteratively updated, including: calculating the distance between the position of the position marker contained in the hole tracking result and the position of the position marker contained in the hole recognition result to obtain a translation matrix; and with the goal of minimizing the position deviation of the connecting holes, solving the rotation matrix required for a successful match between the hole tracking result and the hole recognition result; using the rotation matrix and the translation matrix, updating the position of each connecting hole in the hole tracking result corresponding to the current frame image.

[0009] In one embodiment, the method further includes: obtaining a position deviation threshold; in response to the connection hole position deviation being less than the position deviation threshold, using the hole position tracking result corresponding to the current frame image obtained in the current iteration as the hole position tracking result corresponding to the optimized current frame image.

[0010] In one embodiment, the pose change of the connector is obtained based on the image change between the current frame image and the previous frame image, including: obtaining a tracking model corresponding to the connector, the tracking model being obtained by performing multi-view image acquisition on the connector; determining all pose change schemes corresponding to the previous frame image based on the tracking model; comparing the image change between the current frame image and the previous frame image with all pose change schemes respectively, and selecting a pose change scheme that matches the image change from all pose change schemes; and obtaining the pose change of the connector based on the matching pose change scheme.

[0011] In one embodiment, before comparing the image change between the current frame image and the previous frame image with all posture change schemes respectively, and selecting the posture change scheme that matches the image change from all posture change schemes, it also includes: calculating the image gradient between the current frame image and the previous frame image; and, respectively calculating the Hessian matrix of the current frame image and the Hessian matrix of the previous frame image; using the image gradient, the Hessian matrix of the current frame image and the Hessian matrix of the previous frame image as the image change between the current frame image and the previous frame image.

[0012] In one embodiment, image recognition processing is performed on the current frame image to obtain a hole position recognition result, including: inputting the current frame image into a pre-trained hole position recognition model to obtain a first connection hole position output by the hole position recognition model; and performing color separation on the current frame image, performing connection hole screening on connected domains corresponding to different colors obtained by separation to obtain a second connection hole position; and fusing the first connection hole position and the second connection hole position to obtain a hole position recognition result.

[0013] According to a second aspect of the present application, there is provided a hole position detection device for a connector, the device comprising: a data acquisition module, for acquiring a current frame image of the connector, a previous frame image corresponding to the current frame image, and a hole position tracking result corresponding to the previous frame image; wherein the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image, and the hole position tracking result comprises the positions of all connecting holes of the connector obtained by tracking the position change of the connector; a hole position tracking and identification module, for obtaining the position change of the connector based on the image change between the current frame image and the previous frame image, and updating the hole position tracking result corresponding to the previous frame image based on the position change of the connector to obtain the hole position tracking result corresponding to the current frame image; and performing image recognition processing on the current frame image to obtain the hole position recognition result, the hole position recognition result comprises the position of the connecting hole detectable by the current frame image; a tracking optimization module, for calculating the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation; and a hole position determination module, for obtaining the connection hole position corresponding to the connector based on the optimized hole position tracking result corresponding to the current frame image.

[0014] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned connector hole position detection method.

[0015] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned connector hole position detection method when the program instructions are executed by a processor.

[0016] The above scheme obtains the position change of the connector based on the image change between the current frame image and the previous frame image, updates the hole position tracking result corresponding to the previous frame image based on the position change of the connector, and obtains the hole position tracking result corresponding to the current frame image; and performs image recognition processing on the current frame image to obtain a hole position recognition result, and the hole position recognition result includes the position of the connecting hole that can be detected by the current frame image; calculates the connecting hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizes the hole position tracking result corresponding to the current frame image based on the connecting hole position deviation; based on the optimized hole position tracking result corresponding to the current frame image, obtains the connecting hole position corresponding to the connector, and can optimize the hole position tracking result according to the hole position recognition result to improve the accuracy of the hole position tracking result, and the optimized hole position tracking result contains the position of the connecting hole that is not detected in the hole position recognition result, thereby avoiding omissions in hole position detection and obtaining a hole position detection result with a high detection rate and high accuracy.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0019] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0020] Figure 2 is a flow chart of a connector hole position detection method shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a scene for training a hole position recognition model shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of the arrangement of connection holes shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of building a tracking model according to an exemplary embodiment of the present application;

[0024] Figure 6 is a block diagram of a connector hole position detection device shown in an exemplary embodiment of the present application;

[0025] Figure 7 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0028] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0029] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] The following is a description of the connector hole position detection method provided in an embodiment of the present application.

[0031] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are connected to each other in communication.

[0032] The number of the terminal 110 may be one or more. The terminal 110 may be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.

[0033] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0034] In one example, the server 120 can perform hole position detection based on the connector image obtained from the terminal 110 to obtain the connection hole position of the connector. Of course, the server 120 can store the connection hole position locally, transmit it back to the terminal 110, or transmit it to other terminals 110.

[0035] In one example, a client running a target application is installed in the terminal 110. The target application may be an application that provides a hole position detection function. The target application performs hole position detection on the acquired connector image to obtain the connection hole position of the connector. The server 120 may be a background server of the target application, which is used to provide background services for the client of the target application.

[0036] In the connector hole position detection method provided in the embodiment of the present application, the execution subject of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other part of the steps are executed by the server 120.

[0037] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of a connector hole position detection method shown in an exemplary embodiment of the present application. The connector hole position detection method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0038] like Figure 2 As shown, the connector hole position detection method includes at least steps S210 to S240, which are described in detail as follows:

[0039] Step S210: Obtain a current frame image of the connector, a previous frame image corresponding to the current frame image, and a hole position tracking result corresponding to the previous frame image; wherein the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image, and the hole position tracking result includes the positions of all connection holes of the connector obtained by tracking the posture changes of the connector.

[0040] Capture images of the connector to obtain an image frame sequence of the connector.

[0041] The current frame image is any image frame in the image frame sequence, and the previous frame image corresponding to the current frame image is extracted from the image frame sequence, and the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image.

[0042] For example, the current frame image is the latest image frame in the image frame sequence, and the image frame adjacent to the current frame image in the image frame sequence is used as its corresponding previous frame image. Of course, a non-latest image frame can also be selected as the current frame image, or a non-adjacent image frame can be selected as its corresponding previous frame image, and this application does not limit this.

[0043] The preceding frame image corresponds to a hole position tracking result, and the hole position tracking result includes positions of all connection holes obtained by tracking the position change of the connector.

[0044] When tracking the pose change of the connector, it is necessary to obtain the initial pose of the connector, and the initial pose corresponds to the initial positions of all the connection holes. The preset pose can be used as the initial pose. For example, when the connector performs the first hole position recognition, the pose of the connector defaults to the preset pose, and the position of the connection hole obtained by the first hole position recognition is used as the initial position; when the image frame of the connector is subjected to image recognition processing, it can also be detected whether the positions of all the connection holes of the connector can be detected. If so, the connector pose corresponding to the image frame is used as the initial pose, and the position of the connection hole obtained by the image frame through image recognition processing is used as the initial position. After obtaining the initial pose of the connector, the pose change is obtained by comparing the subsequently collected image frames with the image frames corresponding to the initial pose, and then the connector pose corresponding to the subsequently collected image frames is obtained, thereby realizing the pose tracking of the connector, and the connection hole position under the corresponding pose is obtained based on the pose tracking result, and the hole position tracking result is obtained.

[0045] Step S220: obtaining a posture change of the connector based on the image change between the current frame image and the previous frame image, updating the hole position tracking result corresponding to the previous frame image based on the posture change of the connector to obtain a hole position tracking result corresponding to the current frame image; and performing image recognition processing on the current frame image to obtain a hole position recognition result, which includes the position of the connection hole detectable by the current frame image.

[0046] The method for obtaining the hole tracking result of the current frame image is similar to the method for obtaining the hole tracking result of the previous frame image. The position change of the connector is obtained by calculating the image change between the current frame image and the previous frame image, and the hole tracking result corresponding to the previous frame image is updated based on the position change of the connector to obtain the hole tracking result corresponding to the current frame image.

[0047] In addition, image recognition processing is performed on the current frame image to obtain a hole position recognition result, which includes the position of the connection hole that can be detected by the current frame image.

[0048] For example, the current frame image is subjected to image recognition processing through a pre-trained neural network model to identify the positions of connection holes detectable in the current frame image and obtain a hole position recognition result.

[0049] For another example, the current frame image is color separated, connected domains corresponding to each color are obtained based on the color separation, and connected domains that conform to the shape of the connection holes are screened to identify the positions of the connection holes that can be detected in the current frame image, thereby obtaining a hole position recognition result.

[0050] Step S230: calculating the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation.

[0051] It should be noted that since the hole position recognition result is the connection hole position obtained by analyzing the image content of the current frame image, it may not be able to successfully detect all the connection holes of the connector. For example, in actual industrial operation scenarios, the connector will move in real time, and as more wires are plugged into the surface of the connector, plug hole occlusion will inevitably occur, resulting in only some of the connection holes of the connector being detected in the hole position recognition result.

[0052] Therefore, the position accuracy of the connection holes directly detected in the hole position recognition result is higher than that of the hole position tracking result, but the anti-occlusion ability of the hole position recognition result is poor. Since the hole position tracking result is the connection hole position obtained by tracking the position of the connector, it can track all the connection holes of the connector and has a certain anti-occlusion ability. However, the hole position tracking result is obtained based on the position projection, and its accuracy is lower than the hole position recognition result.

[0053] The present application calculates the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizes the hole position tracking result corresponding to the current frame image based on the connection hole position deviation.

[0054] Among them, the connection hole position deviation is the position drift between the connection hole of the hole tracking result and the connection hole of the hole recognition result. The larger the connection hole position deviation, the greater the difference between the hole tracking result and the hole recognition result. The smaller the connection hole position deviation, the smaller the difference between the hole tracking result and the hole recognition result.

[0055] The hole tracking result corresponding to the current frame image is optimized according to the connection hole position deviation to reduce the difference between the hole tracking result and the hole recognition result, thereby improving the accuracy of the hole tracking result.

[0056] Step S240: based on the hole position tracking result corresponding to the optimized current frame image, the connection hole position corresponding to the connector is obtained.

[0057] Exemplarily, the connection holes that are not detected in the hole position recognition results are determined, the positions of the connection holes that are not detected in the hole position tracking results corresponding to the optimized current frame image are extracted, and the positions of the connection holes that are not detected in the hole position tracking results and the hole position recognition results are combined to obtain the connection hole positions corresponding to all the connection holes of the connector.

[0058] Exemplarily, the hole position tracking result corresponding to the optimized current frame image can be directly used as the connection hole positions corresponding to all the connection holes of the connector.

[0059] Exemplarily, it is also possible to obtain only the connection hole positions corresponding to some connection holes of the connector, such as determining the connection holes required for the current industrial task to obtain the target connection holes. If the hole position recognition result does not contain the target connection holes, the positions of the target connection holes are obtained from the optimized hole position tracking results. If the hole position recognition result contains the target connection holes, the positions of the target connection holes are obtained from the hole position recognition results.

[0060] The hole position tracking result is optimized according to the hole position recognition result to improve the accuracy of the hole position tracking result, and the optimized hole position tracking result corresponding to the current frame image is obtained. The optimized hole position tracking result contains the positions of the connecting holes that are not detected in the hole position recognition result, thereby improving the anti-occlusion ability of the hole position detection.

[0061] Next, some embodiments of the present application are described in detail.

[0062] In some embodiments, in step S220, image recognition processing is performed on the current frame image to obtain a hole position recognition result, including: inputting the current frame image into a pre-trained hole position recognition model to obtain a first connection hole position output by the hole position recognition model; and performing color separation on the current frame image, performing connection hole screening on the connected domains corresponding to the separated different colors to obtain a second connection hole position; and fusing the first connection hole position and the second connection hole position to obtain a hole position recognition result.

[0063] The hole position recognition is performed on the current frame image based on the pre-trained hole position recognition model to obtain the first connection hole position.

[0064] Among them, the hole position recognition model can be implemented based on convolutional neural networks (CNN), recurrent neural networks (RNN), etc., and this application does not limit this.

[0065] The following example illustrates the training process of the hole position recognition model:

[0066] See also Figure 3 , Figure 3A schematic diagram of a scene for training a hole position recognition model is shown as an exemplary embodiment of the present application. Figure 3 As shown, the process information and 3D model of the connector are imported from the database, and the server inputs the process information and 3D model as prior information to the terminal. The terminal sets a virtual camera and simulates actual lighting conditions to perform virtual imaging on the 3D model to obtain a sample image. The camera intrinsic parameters of the virtual camera can be the same as the camera intrinsic parameters used for image acquisition in the actual application scenario of the terminal.

[0067] After obtaining the sample image, edge detection is performed on the sample image to extract the contour information of the sample image. The contour information is filtered based on the shape and size of the connection holes to obtain the connection hole contour. Then, the actual hole positions contained in the sample image are labeled by combining the process information and the filtered connection hole contour to obtain the sample label corresponding to the sample image.

[0068] Based on the sample images and sample labels obtained above, the initial neural network model is iteratively trained until the training end condition is met, thereby obtaining a trained hole position recognition model.

[0069] The above-mentioned embodiment can use process information and three-dimensional models as prior information to implement unsupervised training of the hole position recognition model, thereby reducing the labor cost in the model training process.

[0070] The trained hole position recognition model has a connection hole detection function. The current frame image is input into the pre-trained hole position recognition model to obtain the first connection hole position output by the hole position recognition model.

[0071] Exemplarily, the hole position recognition model identifies the image area belonging to the connection hole in the current frame image, and then performs grayscale processing, binarization processing, filtering processing, edge detection, contour extraction, etc. on the image area belonging to the connection hole to obtain the contour of the connection hole, and calculates the center of mass corresponding to the contour of the connection hole, and uses the center of mass as the position of the connection hole. Combined with the position of each detected connection hole, the first connection hole position is obtained.

[0072] In addition, because the color of the connection hole of the connector and the image of its wiring surface under normal lighting are completely different, the current frame image is color separated, and then the connected domains corresponding to the separated different colors are filtered out according to the size and / or shape of the connected domains belonging to the connection hole to obtain the second connection hole position.

[0073] Furthermore, the first connection hole position and the second connection hole position are fused to obtain a hole position recognition result, so as to improve the recognition and detection accuracy of the connection holes.

[0074] For example, the first connection hole position and the second connection hole position both contain the positions of multiple connection holes, the positions of the first connection hole position and the second connection hole position belonging to the same connection hole are determined, and the mean of the positions of the first connection hole position and the second connection hole position belonging to the same connection hole is calculated as the position identification result of the connection hole.

[0075] For another example, connection holes that are detected in both the first connection hole position and the second connection hole position are determined, and connection holes that are detected only once in the first connection hole position and the second connection hole position are determined; for connection holes that are all detected, the connection hole position in the first connection hole position or the second connection hole position can be selected as the position recognition result of the connection hole, or the calculated average of the positions belonging to the connection hole in the first connection hole position and the second connection hole position can be used as the position recognition result of the connection hole; for connection holes that are detected only once, the connection hole position in the first connection hole position or the second connection hole position can be directly used as the position recognition result of the connection hole.

[0076] In some implementations, before performing connection hole recognition on the current frame image, the current frame image may also be subjected to perspective transformation to obtain a connector image under a formal viewing angle, thereby improving the accuracy of the hole position recognition result.

[0077] Take a circular connector as an example: because the edge of the connector is obvious, and no matter what angle the connector is under the camera, it will be circular or elliptical, so after binarization, edge extraction, circle fitting and other image processing, the points on the circumference of the connector edge can be obtained, and the perspective transformation is performed based on these circumference points of the edge. The calculation formula is shown in Formula 1 below:

[0078] Formula 1

[0079] In formula 1, is the pixel coordinate of the i-th connector image in the formal viewing angle after perspective transformation; is the pixel coordinate of the i-th pixel in the current frame image; is the perspective transformation matrix.

[0080] Image recognition processing is performed on the current frame image after perspective transformation to obtain the hole position recognition result.

[0081] It should be noted that when performing connection hole recognition on the current frame image, it is necessary to determine the order of the recognized connection holes. For a connection hole layout that is not prone to rotational ambiguity, the order of the connection holes can be directly obtained based on the distribution of the recognized connection holes. However, for a connection hole layout that is prone to rotational ambiguity, such as when the connection holes of a connector have the same features and are centrally symmetrically distributed, rotational ambiguity exists in the detected connection holes, thereby creating a risk of incorrect recognition of the connection hole order.

[0082] To avoid rotational ambiguity that could lead to incorrect identification of the connection hole order, the connector is marked with position markers, for example, see Figure 4 , Figure 4 This is a schematic diagram of the arrangement of the connection holes shown in an exemplary embodiment of the present application. Figure 4 As shown, the connector surface is marked with position markers such as characters, numbers or guide lines to identify the position order of each connection hole in the connector. The order of each connection hole is obtained by performing homography transformation on the connection holes and position markers actually detected by the current frame image and the connector distribution template obtained from the three-dimensional model. The connector distribution template contains the position markers and the template positions of each connection hole.

[0083] Exemplarily, for characters or numbers marked on the surface of the connector, the same unsupervised training method as the above-mentioned hole position recognition model can be used to perform deep learning training on the characters or numbers to form a digital character detector. After detecting two or more characters or numbers, the connection hole closest to the character or number is associated with the detected character or number, and then the connector distribution template is obtained. Based on the detected character or number, the connector distribution template and the detected connection hole are homographically transformed, and the judgment is made based on the minimum distance to achieve virtual and real hole position mapping between the connector distribution template and the detected connection hole, and complete the sorting of the detected disordered connection holes.

[0084] For example, for the guide line marked on the surface of the connector, the guide line image of the connector in the front view state is obtained, and the guide line image in the front view state is stored as a guide line template image as prior information for subsequent matching. The guide line binary image contained in the current frame image can be obtained through region segmentation and color screening, and then the Hu moment feature calculation and template matching are performed on the guide line binary image and the guide line template image at the same time, and the rotation angle difference between the guide line binary image and the guide line template image can be calculated, and the calculation formula is as follows Formula 2:

[0085] Formula 2

[0086] In formula 2, is the i-th eigenvalue in the guide line template image; is the i-th eigenvalue of the guide line binary image. The higher the similarity between the guide line binary image and the guide line template image, the greater the rotation angle difference. The smaller it is, the more complete the match is. After the match is completed, the order of the connection holes in the connector distribution template in the front view state and the connection holes detected after rotation are mapped one by one through homography transformation to complete the sorting of the connection holes.

[0087] The above-mentioned embodiment can obtain the hole position recognition result by performing image recognition processing on the current frame image. However, due to problems such as image occlusion and defects in the recognition algorithm, some connection holes may be missed. During the process of hole position recognition of the connector, the present application tracks the hole position of the connector to determine the position of the obscured connection hole in combination with the hole position tracking result.

[0088] Specifically, the position change of the connector is obtained based on the image change between the current frame image and the previous frame image, and the hole position tracking result corresponding to the previous frame image is updated based on the position change of the connector to obtain the hole position tracking result corresponding to the current frame image.

[0089] In some embodiments, an image gradient between a current frame image and a previous frame image is calculated; and a Hessian matrix of the current frame image and a Hessian matrix of the previous frame image are calculated respectively; and the image gradient, the Hessian matrix of the current frame image, and the Hessian matrix of the previous frame image are used as the image change between the current frame image and the previous frame image.

[0090] Of course, a target may be set in the image acquisition scene, and the image change is obtained based on the difference between the target pose in the current frame image and the target pose in the previous frame image.

[0091] The specific calculation method of the image change can be flexibly determined according to the actual application scenario, and this application does not limit the specific calculation method of the image change.

[0092] In some embodiments, step S220 obtains the pose change of the connector based on the image change between the current frame image and the previous frame image, including: obtaining a tracking model corresponding to the connector, the tracking model being obtained by performing multi-view image acquisition on the connector; determining all pose change schemes corresponding to the previous frame image based on the tracking model; comparing the image change between the current frame image and the previous frame image with all pose change schemes, and selecting a pose change scheme that matches the image change from all pose change schemes; and obtaining the pose change of the connector based on the matching pose change scheme.

[0093] For example, see Figure 5 , Figure 5 A schematic diagram of constructing a tracking model is shown as an exemplary embodiment of the present application. Figure 5 As shown, a virtual camera with N viewpoints is set to shoot the three-dimensional model of the connector, and the optical axis of the virtual camera points to the center of the connector. The virtual camera is used to render and shoot the three-dimensional model of the connector. For all viewpoints, the direction vectors and M model surface points and the normal vectors of these points are calculated, and the tracking model is obtained by recording the direction vectors and normal vectors of each viewpoint.

[0094] When starting to perform connector pose tracking, first give an initial pose for initialization. Each subsequent time the connector moves, determine the image change between the current frame image and the previous frame image, and compare the image change with all pose change schemes corresponding to the previous frame image in the pre-built tracking model. Select the pose change scheme that matches the image change from all pose change schemes to estimate the pose change corresponding to the current frame image to update the real-time pose of the connector and realize the pose tracking of the connector.

[0095] Among them, the initial posture corresponds to a set of initial hole positions, and the initial hole position acquisition step may include: identifying the two-dimensional pixel coordinates of all connecting holes when the connector is not obstructed, and then importing the two-dimensional pixel coordinates into the three-dimensional model of the connector for ray collision detection, to obtain the three-dimensional coordinates of each connecting hole in the three-dimensional model coordinate system, and use the obtained three-dimensional coordinates as the initial hole position corresponding to the initial posture.

[0096] For example, the method of calculating three-dimensional coordinates based on the ‌PNP (Perspective-n-Point) algorithm can be referred to the following formula 3:

[0097] Formula 3

[0098] In formula 3, is the two-dimensional pixel coordinate of the i-th two-dimensional pixel point; K is the intrinsic parameter matrix of the camera that captures the image of the connector; is the scaling factor; is the three-dimensional coordinate mapped to the three-dimensional model coordinate system; j=1,2,3...,q, which represents the coordinates of the virtual control point in the three-dimensional model coordinate system, and the value of q is generally 4; It reflects the weight coefficient of the influence of the i-th three-dimensional coordinate point on the j-th virtual control point.

[0099] Based on the position change of the connector obtained by the above tracking, the position of the connector is tracked with the initial hole position as the starting point to obtain the position change of the connector, and then the hole position tracking result corresponding to the previous frame image is updated according to the position change of the connector to obtain the hole position tracking result corresponding to the current frame image.

[0100] Then, the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result is calculated, and the hole position tracking result corresponding to the current frame image is optimized based on the connection hole position deviation.

[0101] In some embodiments, the step S230 of calculating the connection hole position deviation between the hole position tracking result and the hole position recognition result corresponding to the current frame image includes: calculating the position distance between each connection hole in the hole position tracking result and each connection hole in the hole position recognition result; associating the connection holes in the hole position recognition result with the connection holes in the hole position tracking result having the smallest position distance to obtain the minimum adjacent holes corresponding to each connection hole in the hole position tracking result; summing the position distances between the connection holes in each hole position tracking result and the corresponding associated minimum adjacent hole, and using the summed calculation result as the connection hole position deviation between the hole position tracking result and the hole position recognition result corresponding to the current frame image.

[0102] For example, assuming that the connector has a total of n connection holes, the hole position tracking result can be expressed as , is the position of the nth connection hole, and the hole position recognition result can be expressed as , is the position of the mth connection hole, where n is greater than or equal to m.

[0103] The connecting holes with the smallest position distance between the hole position recognition results and the connecting holes in the hole position tracking results are associated to obtain the minimum adjacent holes corresponding to each connecting hole in the hole position tracking results. With Collection The position distance between each connection hole in is obtained and The position distance between them is the smallest, then it is considered and The probability of belonging to the same connection hole is high. and By analogy, the minimum adjacent hole corresponding to each connecting hole in the hole tracking result is obtained.

[0104] Then, the position distances between the connected holes and their corresponding minimum adjacent holes in each hole tracking result are summed up and the summation result is used as the connected hole position deviation between the hole tracking result and the hole recognition result corresponding to the current frame image.

[0105] In some implementations, the position of each connection hole in the hole position tracking result corresponding to the current frame image is iteratively updated with the goal of minimizing the connection hole position deviation.

[0106] For example, the objective function to be minimized can be expressed as the following formula 4:

[0107] Formula 4

[0108] In formula 4, n is the number of connected holes in the hole tracking result; is the position of the i-th connection hole in the hole position recognition result; Neutralize hole tracking results The positions of the associated connection holes; R is the rotation matrix for optimizing the positions of the various connection holes in the hole tracking results; T is the translation matrix for optimizing the positions of the various connection holes in the hole tracking results.

[0109] Through the above formula 4, with the goal of minimizing the value of e, the positions of each connection hole in the hole position tracking result corresponding to the current frame image are iteratively updated, and the position deviation threshold is obtained. In response to the connection hole position deviation being less than the position deviation threshold, the hole position tracking result corresponding to the current frame image obtained by the current iteration is used as the optimized hole position tracking result corresponding to the current frame image.

[0110] Exemplarily, the connector is also provided with a position marker, the hole tracking result includes the position of the position marker obtained by tracking the position change of the connector, and the hole recognition result includes the position of the position marker detected by the current frame image; with the goal of minimizing the position deviation of the connecting hole, iteratively updating the position of each connecting hole in the hole tracking result corresponding to the current frame image, including: calculating the distance between the position of the position marker contained in the hole tracking result and the position of the position marker contained in the hole recognition result to obtain a translation matrix; and with the goal of minimizing the position deviation of the connecting hole, solving the rotation matrix required for a successful match between the hole tracking result and the hole recognition result; using the rotation matrix and the translation matrix, updating the position of each connecting hole in the hole tracking result corresponding to the current frame image.

[0111] The distance between the position of the position marker contained in the hole tracking result and the position of the position marker contained in the hole recognition result is directly calculated to obtain a translation matrix.

[0112] Among them, position markers include but are not limited to numbers, characters, guide lines, etc.

[0113] For example, the optimal rotation matrix is ​​calculated based on the singular value decomposition (SVD) algorithm, as shown in Formula 5:

[0114] Formula 5

[0115] In formula 5, The position of the position marker contained in the hole tracking result; is the position of the position marker contained in the hole position recognition result; n is the number of connected holes in the hole position tracking result; is the position of the i-th connection hole in the hole position recognition result; Neutralize hole tracking results The positions of the associated connecting holes; R is the rotation matrix for optimizing the positions of each connecting hole in the hole tracking result.

[0116] The present application provides a method for detecting the hole position of a connector, which obtains a posture change of the connector based on an image change between a current frame image and a previous frame image, updates a hole position tracking result corresponding to the previous frame image based on the posture change of the connector, and obtains a hole position tracking result corresponding to the current frame image; and performs image recognition processing on the current frame image to obtain a hole position recognition result, the hole position recognition result includes the position of the connecting hole detectable by the current frame image; calculates the connecting hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizes the hole position tracking result corresponding to the current frame image based on the connecting hole position deviation; obtains the connecting hole position corresponding to the connector based on the hole position tracking result corresponding to the optimized current frame image, and can optimize the hole position tracking result according to the hole position recognition result to improve the accuracy of the hole position tracking result, and the optimized hole position tracking result contains the position of the connecting hole that is not detected in the hole position recognition result, thereby avoiding omissions in hole position detection and obtaining a hole position detection result with a high detection rate and high accuracy.

[0117] Figure 6 FIG. 1 is a block diagram of a connector hole position detection device shown in an exemplary embodiment of the present application. Figure 6 As shown, the exemplary connector hole position detection device 600 includes: a data acquisition module 610, a hole position tracking and identification module 620, a tracking optimization module 630 and a hole position determination module 640. Specifically:

[0118] The data acquisition module 610 is used to acquire a current frame image of the connector, a previous frame image corresponding to the current frame image, and a hole position tracking result corresponding to the previous frame image; wherein the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image, and the hole position tracking result includes the positions of all connection holes of the connector obtained by tracking the position change of the connector;

[0119] The hole position tracking and recognition module 620 is used to obtain the position change of the connector based on the image change between the current frame image and the previous frame image, update the hole position tracking result corresponding to the previous frame image based on the position change of the connector, and obtain the hole position tracking result corresponding to the current frame image; and perform image recognition processing on the current frame image to obtain the hole position recognition result, which includes the position of the connection hole that can be detected by the current frame image;

[0120] A tracking optimization module 630 is used to calculate the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimize the hole position tracking result corresponding to the current frame image based on the connection hole position deviation;

[0121] The hole position determination module 640 is used to obtain the connection hole position corresponding to the connector based on the hole position tracking result corresponding to the optimized current frame image.

[0122] It should be noted that the hole position detection device for the connector provided in the above embodiment and the hole position detection method for the connector provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the hole position detection device for the connector provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0123] See also Figure 7 , Figure 7 7 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 700 includes a memory 701 and a processor 702, and the processor 702 is used to execute program instructions stored in the memory 701 to implement the steps in the embodiment of the hole position detection method of any connector described above. In a specific implementation scenario, the electronic device 700 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 700 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0124] Specifically, the processor 702 is used to control itself and the memory 701 to implement the steps in the embodiment of the hole position detection method of any connector described above. The processor 702 can also be called a central processing unit (CPU). The processor 702 may be an integrated circuit chip with signal processing capabilities. The processor 702 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 702 can be implemented by an integrated circuit chip.

[0125] See also Figure 8 , Figure 8 The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor, and the program instructions 810 are used to implement the steps in any of the above-mentioned embodiments of the hole position detection method of the connector.

[0126] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0127] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0129] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A method for detecting the hole position of a connector, characterized in that: include: Acquire a current frame image of the connector, a previous frame image corresponding to the current frame image, and a hole position tracking result corresponding to the previous frame image; wherein the acquisition time of the previous frame image is earlier than the acquisition time of the current frame image, and the hole position tracking result includes the positions of all connection holes of the connector obtained by tracking the posture change of the connector; Obtaining a position change amount of the connector based on an image change between the current frame image and the previous frame image, and updating a hole position tracking result corresponding to the previous frame image based on the position change amount of the connector to obtain a hole position tracking result corresponding to the current frame image; and performing image recognition processing on the current frame image to obtain a hole position recognition result, wherein the hole position recognition result includes a position of a connection hole detectable by the current frame image; Calculating a connection hole position deviation between a hole position tracking result corresponding to the current frame image and the hole position recognition result, and optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation; Based on the hole position tracking result corresponding to the optimized current frame image, the connection hole position corresponding to the connector is obtained.

2. The method according to claim 1, characterized in that The calculating the connection hole position deviation between the hole position tracking result corresponding to the current frame image and the hole position recognition result includes: Calculating the position distance between each connection hole in the hole position tracking result and each connection hole in the hole position identification result; Associating the connecting holes in the hole position recognition result with the connecting holes in the hole position tracking result whose position distance is the smallest, so as to obtain the minimum adjacent holes corresponding to the respective connecting holes in the hole position tracking result; The position distances between the connecting holes and the corresponding minimum adjacent holes in each hole tracking result are summed and calculated, and the summed calculation result is used as the connecting hole position deviation between the hole tracking result corresponding to the current frame image and the hole recognition result.

3. The method according to claim 1, characterized in that The optimizing the hole position tracking result corresponding to the current frame image based on the connection hole position deviation includes: With the goal of minimizing the connection hole position deviation, the position of each connection hole in the hole position tracking result corresponding to the current frame image is iteratively updated.

4. The method according to claim 3, characterized in that The connector is provided with a position marker, the hole position tracking result includes the position of the position marker obtained by tracking the position change of the connector, and the hole position recognition result includes the position of the position marker detected by the current frame image; The step of iteratively updating the position of each connection hole in the hole position tracking result corresponding to the current frame image with the goal of minimizing the connection hole position deviation includes: Calculating the distance between the position of the position marker in the hole tracking result and the position of the position marker in the hole recognition result to obtain a translation matrix; and With the goal of minimizing the position deviation of the connection hole, solving the rotation matrix required for successful matching between the hole tracking result and the hole recognition result; The positions of the respective connection holes in the hole position tracking result corresponding to the current frame image are updated by using the rotation matrix and the translation matrix.

5. The method according to claim 3, characterized in that: The method further comprises: Get the position deviation threshold; In response to the connection hole position deviation being less than the position deviation threshold, the hole position tracking result corresponding to the current frame image obtained in the current iteration is used as the optimized hole position tracking result corresponding to the current frame image.

6. The method according to claim 1, characterized in that The obtaining of the position change of the connector based on the image change between the current frame image and the previous frame image includes: Acquire a tracking model corresponding to the connector; wherein the tracking model is obtained by performing multi-viewpoint image acquisition on the connector, recording the direction vector of each viewpoint and the normal vector of the surface point of the connector acquired at each viewpoint; Determine all posture change schemes corresponding to the preceding frame image based on the tracking model; Comparing the image change between the current frame image and the previous frame image with all the posture change schemes respectively, and selecting a posture change scheme that matches the image change from all the posture change schemes; Based on the matched posture change scheme, the posture change amount of the connector is obtained.

7. The method according to claim 6, characterized in that Before comparing the image change between the current frame image and the previous frame image with all the posture change schemes respectively and selecting a posture change scheme matching the image change from all the posture change schemes, the method further includes: Calculating an image gradient between the current frame image and the previous frame image; and respectively calculating a Hessian matrix of the current frame image and a Hessian matrix of the previous frame image; The image gradient, the Hessian matrix of the current frame image and the Hessian matrix of the previous frame image are used as the image change between the current frame image and the previous frame image.

8. The method according to claim 1, characterized in that The performing image recognition processing on the current frame image to obtain a hole position recognition result includes: Inputting the current frame image into a pre-trained hole position recognition model to obtain a first connection hole position output by the hole position recognition model; and Performing color separation on the current frame image, and performing connection hole screening on connected domains corresponding to different colors obtained by separation to obtain second connection hole positions; The first connection hole position and the second connection hole position are merged to obtain a hole position recognition result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images