Connector state detection method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a connector state detection method and device, equipment and a storage medium, and the method comprises the steps: carrying out the key point extraction of the edge of a plug and the edge of a socket in an image, and obtaining the first coordinate information of at least two key points corresponding to the plug and at least four key points corresponding to the socket; and performing perspective distortion correction on the at least two key points and the first coordinate information of the at least four key points to obtain correction coordinate information of the at least two key points and correction coordinate information of the at least four key points, and determining the state of the connector. According to the embodiment of the invention, perspective distortion correction is carried out on the first coordinate information of the key point, so that the influence of perspective distortion on the coordinate information of the key point is avoided, and the accuracy of a comparison result is guaranteed. According to the process of determining the state of the connector based on the accurate comparison result, the detection accuracy can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a connector status detection method, device, equipment and storage medium. Background Art

[0002] A connector is a device used to connect various electronic components, devices and equipment. It plays an important role in realizing signal transmission and exchange between various electronic components, devices and equipment.

[0003] The connector consists of a plug and a socket, and the connection status between the plug and the socket is an important factor in determining whether the signal can be transmitted normally. At present, in order to ensure that the connector can effectively transmit signals, the connector can be photographed to obtain an image including the connector, and the connection status between the connector plug and the socket can be further determined based on the image. Specifically, the connection status between the connector plug and the socket can be determined by extracting the edge line segment on the outside of the connector plug and the edge line segment on the outside of the socket in the image, and comparing the distance between the two edge line segments with the standard distance, wherein: the connection status between the plug and the socket includes a plugged state or a loose plugged state.

[0004] However, the result of determining the connection status between the connector plug and the socket in this way is often inaccurate. Therefore, a solution is urgently needed to solve the above problem. Summary of the invention

[0005] Based on the above problems, the present application provides a connector status detection method, device, equipment and storage medium.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In the first aspect, the present application provides a method for detecting a connector state, including:

[0008] acquiring an image including a connector, the connector including a plug and a socket;

[0009] Extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug; extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear;

[0010] According to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points;

[0011] According to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, the state of the connector is determined, the state including that the plug and the socket are in a plugged in state, or that the plug and the socket are in a loosely plugged in state.

[0012] In a possible implementation, the performing perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points to obtain the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, includes:

[0013] Calculating a perspective transformation matrix according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points;

[0014] Based on the perspective transformation matrix, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points.

[0015] In a possible implementation, determining the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points includes:

[0016] The state of the connector is determined according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points.

[0017] In a possible implementation, determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes:

[0018] Determine the edge line segment corresponding to the plug according to the corrected coordinate information of the at least two key points; determine the edge line segment corresponding to the socket according to the corrected coordinate information of at least two key points among the at least four key points, wherein the edge line segment corresponding to the socket does not intersect with the edge line segment corresponding to the plug;

[0019] Calculating the distance between the edge line segment corresponding to the plug and the corrected edge line segment corresponding to the socket to obtain a first distance;

[0020] The state of the connector is determined based on a comparison result obtained by comparing the first distance with a standard value of the first distance; the standard value of the first distance is calculated based on standard coordinate information of the at least two key points and standard coordinate information of at least two key points among the at least four key points.

[0021] In a possible implementation manner, determining the state of the connector according to a comparison result obtained by comparing the first distance with a standard value of the first distance includes:

[0022] When the comparison result is that the difference between the first distance and the standard value of the first distance is less than or equal to a first tolerance threshold, determining that the state of the connector is that the plug and the socket are in a tightly plugged state;

[0023] When the comparison result is that the difference between the first distance and the standard value of the first distance is greater than the first tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged together.

[0024] In a possible implementation, determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes:

[0025] According to the corrected coordinate information of the first key point and the corrected coordinate information of the third key point, a second distance between the first key point and the third key point is calculated; according to the corrected coordinate information of the second key point and the corrected coordinate information of the fourth key point, a third distance between the second key point and the fourth key point is calculated; the at least two key points corresponding to the plug include the first key point and the second key point, and the at least four key points corresponding to the socket include the third key point and the fourth key point;

[0026] An edge line segment with the first key point and the second key point as endpoints does not intersect with an edge line segment with the third key point and the fourth key point as endpoints; an edge line segment with the first key point and the third key point as endpoints does not intersect with an edge line segment with the second key point and the fourth key point as endpoints;

[0027] Calculating a mean of the second distance and the third distance;

[0028] The state of the connector is determined according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point.

[0029] In a possible implementation, determining the state of the connector according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point includes:

[0030] Determining a standard value of the mean according to the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point;

[0031] Comparing the mean with the standard value of the mean to obtain a comparison result;

[0032] When the comparison result is that the difference between the mean and the standard value of the mean is less than or equal to the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the comparison result is that the difference between the mean and the standard value of the mean is greater than the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0033] In a possible implementation, determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes:

[0034] Generate a first feature vector based on the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; generate a second feature vector based on the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points; the first feature vector includes the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; the second feature vector includes the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points;

[0035] Calculating the cosine similarity between the first feature vector and the second feature vector;

[0036] When the cosine similarity is greater than a similarity threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the cosine similarity is less than or equal to the similarity threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0037] In a possible implementation manner, extracting key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and extracting key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket include:

[0038] Based on the key point detection model, key points are extracted from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and key points are extracted from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket; the key point detection model is trained based on the labeled data, and the labeled data is an image including the connector after the key points are labeled.

[0039] A second aspect: The embodiment of the present application provides a connector status detection device, including:

[0040] an acquisition unit, an extraction unit, a correction unit, and a determination unit;

[0041] The acquisition unit is used to acquire an image including a connector, wherein the connector includes a plug and a socket;

[0042] The extraction unit is used to extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and to extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear;

[0043] The correction unit is used to perform perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points, so as to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points;

[0044] The determination unit is used to determine the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, wherein the state includes whether the plug and the socket are in a plugged in state or the plug and the socket are in a loosely plugged in state.

[0045] A third aspect: An embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory;

[0046] The memory is used to store program code and transmit the program code to the processor;

[0047] The processor is used to execute the steps of the connector status detection method as described above according to the instructions in the program code.

[0048] A fourth aspect: An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a connector status detection method as described above are implemented.

[0049] Compared with the prior art, this application has the following beneficial effects:

[0050] In an embodiment of the present application, by acquiring an image including a connector, extracting key points from the edge of the plug in the image, obtaining first coordinate information of at least two key points corresponding to the plug, extracting key points from the edge of the socket in the image, obtaining first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear. According to the first coordinate information of at least four key points and the standard coordinate information of at least four key points, the first coordinate information of at least two key points and the first coordinate information of at least four key points are corrected for perspective distortion, and corrected coordinate information of at least two key points and corrected coordinate information of at least four key points can be obtained; according to the corrected coordinate information of at least two key points and the corrected coordinate information of at least four key points, the state of the connector is determined. In an embodiment of the present application, by correcting the perspective distortion of the first coordinate information of the key points, the influence of perspective distortion on the coordinate information of the key points is avoided, and the accuracy of the comparison result is guaranteed. The process of determining the state of the connector based on the accurate comparison result can effectively improve the accuracy of the detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A schematic diagram of extracting edge segments;

[0053] Figure 2 A flowchart of a connector status detection method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of connector key point extraction provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of perspective distortion correction provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of calculating a first distance provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of another method for calculating a first distance provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of calculating a mean value provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of another method for calculating the mean value provided in an embodiment of the present application;

[0060] Fig. 9 A schematic diagram of a connector status detection method provided in an embodiment of the present application;

[0061] Fig.10 A schematic structural diagram of a connector status detection device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] As described above, currently, in order to ensure that the connector can effectively transmit signals, the connector can be photographed to obtain an image including the connector, and the connection status between the connector plug and the socket can be further determined based on the image.

[0064] like Figure 1 As shown in the figure, this figure is a schematic diagram of extracting edge segments. The edge segment 1001 on the outside of the plug of the connector in the image and the edge segment 1002 on the outside of the socket can be extracted through the edge extraction algorithm. The distance (d) between the two edge segments is compared with the standard distance to determine the connection state between the connector plug and the socket, wherein: the connection state between the plug and the socket includes a plugged state or a loose plugged state.

[0065] However, due to the influence of shooting angle and distance, the connector in the image may have perspective distortion. In the case of perspective distortion, the calculated distance between the two edge segments will deviate from the actual distance, thereby reducing the accuracy of detection.

[0066] At the same time, edge extraction methods are easily affected by a variety of interference factors, such as image shooting angle, lighting, occlusion and other factors, which may cause extraction failure and reduce detection efficiency.

[0067] Based on this, an embodiment of the present application provides a connector status detection method, which extracts key points from the edge of the plug in the image to obtain the first coordinate information of at least two key points corresponding to the plug, and extracts key points from the edge of the socket in the image to obtain the first coordinate information of at least four key points corresponding to the socket. On this basis, the first coordinate information of at least two key points corresponding to the plug and the first coordinate information of at least four key points corresponding to the socket are corrected for perspective distortion, thereby avoiding the influence of perspective distortion on the coordinate information of the key points and ensuring the accuracy of the comparison results. The process of determining the connector status based on accurate comparison results can effectively improve the accuracy of detection.

[0068] The method provided in the embodiment of the present application is introduced below with reference to the accompanying drawings.

[0069] like Figure 2 As shown, this figure is a flowchart of a connector status detection method provided in an embodiment of the present application, including S101-S104.

[0070] S101. Acquire an image including a connector.

[0071] The connector includes a plug and a socket, and the connection state between the plug and the socket is an important factor in determining whether the signal can be transmitted normally.

[0072] In the embodiment of the present application, the connector can be photographed by an image acquisition device (such as a camera) to obtain an image including the connector.

[0073] In one example, a single connector can be photographed by an image acquisition device to obtain multiple images including a connector. When the connector status in the image is subsequently detected, the connector status in multiple images can be detected one by one, or multiple images can be detected in batches to improve the detection efficiency of the connector status.

[0074] In another example, an image including multiple connectors can be captured by an image acquisition device, so that when the status of the connectors is subsequently detected, the status of multiple connectors in one image can be detected simultaneously, thereby improving detection efficiency.

[0075] It is understandable that the number of connectors included in an image is not specifically limited in the embodiments of the present application. For ease of understanding, the following description will only take the detection of the state of a connector in an image as an example.

[0076] S102, extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket.

[0077] Among them, any three key points among the at least four key points corresponding to the socket are not collinear.

[0078] In a possible implementation, key points may be extracted from the edge of the plug in the image based on a key point detection model to obtain first coordinate information of at least two key points corresponding to the plug. Key points may be extracted from the edge of the socket in the image based on the key point detection model to obtain first coordinate information of at least four key points corresponding to the socket.

[0079] The key point detection model may be trained based on labeled data, wherein the labeled data is an image including connectors after the key points are labeled.

[0080] Exemplarily, to extract key points of a connector in an image, at least two key points may be defined on an edge of a plug of the connector, and at least four key points may be defined on an edge of a socket of the connector.

[0081] like Figure 3As shown in the figure, this figure is a schematic diagram of a connector key point extraction provided by an embodiment of the present application. In the figure, two key points defined on the edge of the connector plug are P1 and P2, which are located at the edge corners of the plug; four key points defined on the edge of the connector socket are P3, P4, P5, and P6, which are located at the four edge corners of the socket respectively.

[0082] Based on the definition method of the six key points, a large number of images including connectors are annotated with key points to obtain a large amount of annotated data. A deep learning model is trained based on the large amount of annotated data to obtain a key point detection model.

[0083] The image including the connector is input into the key point detection model, and the key point detection model can automatically detect the positions of the six key points determined based on the above definition to obtain the coordinate information of the six key points.

[0084] It can be understood that in the embodiment of the present application, there is no specific limitation on the positions of at least two key points corresponding to the plug and at least four key points corresponding to the socket. Their specific positions can be defined according to actual needs when marking the key points. For example, connectors of different types and positions can correspond to different key points. The above is only an example.

[0085] In an embodiment of the present application, key points of an image are extracted based on a deep learning model, which can automatically locate the position and category of the connector. There is no need to set the area of ​​interest in advance, and there is no need to set the relevant complex parameters corresponding to the edge extraction algorithm. It is less affected by factors such as image shooting angle, lighting, and occlusion, and can effectively improve the extraction speed and accuracy of key points.

[0086] S103. Perform perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points.

[0087] The first coordinate information is the coordinate information of the key points directly obtained by extracting the key points from the image; the corrected coordinate information is the coordinate information obtained by correcting the perspective distortion of the first coordinate information, and is the coordinate information of the key points in the image after eliminating the perspective distortion.

[0088] Perspective distortion refers to the distortion problem caused by perspective relationship when an image presented in three-dimensional space is mapped onto a two-dimensional plane after being imaged by an image acquisition device. This distortion will cause the straight lines in the image to no longer be straight lines, but to become trapezoidal or inclined shapes. In this case, the accuracy of the detection result will be reduced when the state of the connector is detected based on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points.

[0089] To this end, in the embodiment of the present application, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points.

[0090] The principle of perspective distortion correction is to restore the image to its original appearance in three-dimensional space by solving the inverse transformation of camera imaging. This process usually involves determining the four vertices of a rectangle in the image and calculating the inverse matrix of the camera imaging matrix based on the positions of these four vertices in the real world. Then, the image is transformed using this inverse matrix to eliminate perspective distortion.

[0091] In a possible implementation manner, in the embodiment of the present application, a perspective transformation matrix may be calculated according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points.

[0092] The standard coordinate information is a standard value corresponding to the first coordinate information, and is coordinate information without perspective distortion.

[0093] In one example, an image of a connector without perspective distortion and with the plug and the socket in a tightly plugged state can be taken in advance and used as a standard image of this type of connector. Key point extraction is performed on the standard image including the connector to obtain standard coordinate information of at least two key points corresponding to the plug and standard coordinate information of at least four key points corresponding to the socket.

[0094] For example, Figure 3 As shown, the socket of the connector in the embodiment of the present application can be regarded as a rigid body, and a rectangle can be determined based on the points of its four edge corners. By extracting key points from the edge of the socket in the image, the first coordinate information of at least four key points corresponding to the socket can be obtained. Among them, the at least four key points may include the points of the four edge corners of the socket, such as Figure 3 Based on the first coordinate information of P3, P4, P5, P6 and the standard coordinate information of P3, P4, P5, P6, the perspective transformation matrix can be calculated.

[0095] Based on the perspective transformation matrix, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points, so that corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points can be obtained.

[0096] For example, Figure 4 As shown in FIG. 1 , this figure is a schematic diagram of perspective distortion correction provided by an embodiment of the present application. When the key points corresponding to the plug are P1 and P2, and the key points corresponding to the socket are P3, P4, P5, and P6, after obtaining the perspective transformation matrix, the first coordinate information of the above 6 key points can be corrected for perspective distortion according to formula (1) to obtain the corrected coordinate information of the 6 key points.

[0097]

[0098] In formula (1), a0, a1, a2, a3, b0, b1, c0, c1, 1 are the parameters constituting the perspective transformation matrix; X, Y represent the first coordinate information of the key point; X0, Y0 represent the corrected coordinate information of the key point.

[0099] It can be understood that in the embodiment of the present application, based on the perspective transformation matrix, the first coordinate information of any key point can be corrected for perspective distortion to obtain corresponding corrected coordinate information.

[0100] After obtaining the corrected coordinate information of at least two key points corresponding to the plug and the corrected coordinate information of at least four key points corresponding to the socket, S104 may be executed.

[0101] S104: Determine the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points.

[0102] The state of the connector includes that the plug and the socket are in a plugged-in state, or the plug and the socket are in a loosely plugged-in state.

[0103] In one possible implementation, the state of the connector can be determined based on the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points.

[0104] In one example, the edge line segment corresponding to the plug can be determined based on the corrected coordinate information of the at least two key points, and the edge line segment corresponding to the socket can be determined based on the corrected coordinate information of at least two key points among the at least four key points, wherein the edge line segment corresponding to the socket does not intersect with the edge line segment corresponding to the plug.

[0105] like Figure 4 As shown, taking two key points corresponding to a plug as an example, the two key points corresponding to the plug may be P1 and P2. Based on the corrected coordinate information of P1 and P2, the edge line segment 1201 corresponding to the plug may be determined.

[0106] Taking two key points among the at least four key points corresponding to the socket as an example, the two key points among the at least four key points corresponding to the socket may be P3 and P4, or P5 and P6.

[0107] When the two key points corresponding to the socket are P3 and P4, the edge segment determined by P1 and P2 does not intersect with the edge segment determined by P3 and P4. Similarly, when the two key points corresponding to the socket are P5 and P6, the edge segment determined by P1 and P2 does not intersect with the edge segment determined by P5 and P6.

[0108] After the edge line segment corresponding to the plug and the edge line segment corresponding to the socket are determined, the distance between the edge line segment corresponding to the plug and the corrected edge line segment corresponding to the socket may be calculated to obtain a first distance.

[0109] For example, Figure 5 As shown, this figure is a schematic diagram of calculating a first distance provided by an embodiment of the present application. When the two key points corresponding to the plug are P1 and P2, and the two key points corresponding to the socket are P3 and P4, the distance between the edge line segment 1201 corresponding to the plug and the edge line segment 1202 corresponding to the socket is calculated to obtain the first distance (d1).

[0110] like Figure 6 As shown, this figure is another schematic diagram of calculating the first distance provided by an embodiment of the present application. When the two key points corresponding to the plug are P1 and P2, and the two key points corresponding to the socket are P5 and P6, the distance between the edge line segment 1201 corresponding to the plug and the edge line segment 1203 corresponding to the socket is calculated to obtain the first distance (d2).

[0111] After the first distance is obtained, the state of the connector is determined according to a comparison result obtained by comparing the first distance with a standard value of the first distance.

[0112] The standard value of the first distance is calculated based on the standard coordinate information of the at least two key points and the standard coordinate information of at least two key points among the at least four key points.

[0113] It can be understood that the standard value of the first distance can be obtained by real-time calculation or pre-calculation, and this is not specifically limited in the embodiments of the present application.

[0114] In a possible implementation, based on a comparison result obtained by comparing the first distance with the standard value of the first distance, when the comparison result is that the difference between the first distance and the standard value of the first distance is less than or equal to a first tolerance threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the comparison result is that the difference between the first distance and the standard value of the first distance is greater than the first tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0115] It is understandable that the size of the first tolerance threshold is not specifically limited in the embodiment of the present application, and it can be set according to actual conditions. Exemplarily, the above d1 and d2 can correspond to first tolerance thresholds of different sizes, or can correspond to first tolerance thresholds of the same size.

[0116] In a possible implementation, the at least two key points corresponding to the plug include the first key point and the second key point, and the at least four key points corresponding to the socket include the third key point and the fourth key point. An edge line segment with the first key point and the second key point as endpoints does not intersect with an edge line segment with the third key point and the fourth key point as endpoints; an edge line segment with the first key point and the third key point as endpoints does not intersect with an edge line segment with the second key point and the fourth key point as endpoints.

[0117] like Figure 7 As shown, this figure is a schematic diagram of calculating the mean provided by an embodiment of the present application. The first key point may be P1, the second key point may be P2, the third key point may be P3, and the fourth key point may be P4. The edge line segment with P1 and P2 as endpoints does not intersect with the edge line segment with P3 and P4 as endpoints; the edge line segment with P1 and P3 as endpoints does not intersect with the edge line segment with P2 and P4 as endpoints.

[0118] like Figure 8 As shown, this figure is a schematic diagram of another mean value calculation provided by an embodiment of the present application. The first key point may be P1, the second key point may be P2, the third key point may be P5, and the fourth key point may be P6. The edge line segment with P1 and P2 as endpoints does not intersect with the edge line segment with P5 and P6 as endpoints; the edge line segment with P1 and P5 as endpoints does not intersect with the edge line segment with P2 and P6 as endpoints.

[0119] According to the corrected coordinate information of the first key point and the corrected coordinate information of the third key point, the second distance between the first key point and the third key point is calculated; according to the corrected coordinate information of the second key point and the corrected coordinate information of the fourth key point, the third distance between the second key point and the fourth key point is calculated. After obtaining the second distance and the third distance, the average of the second distance and the third distance can be calculated.

[0120] For example, when the third key point is P3 and the fourth key point is P4, the distance between P1 and P3 can be calculated as the second distance (d3), and the distance between P2 and P4 can be calculated as the third distance (d4). Thus, the average of the second distance and the third distance is calculated to be (d3+d4) / 2.

[0121] When the third key point is P5 and the fourth key point is P6, the distance between P1 and P5 can be calculated as the second distance (d5), and the distance between P2 and P6 can be calculated as the third distance (d6). Thus, the average of the second distance and the third distance is calculated to be (d5+d6) / 2.

[0122] The state of the connector is determined according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point.

[0123] Exemplarily, the standard value of the mean may be determined based on the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point.

[0124] By comparing the mean value with the standard value of the mean value, a comparison result can be obtained. When the comparison result is that the difference between the mean value and the standard value of the mean value is less than or equal to the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the comparison result is that the difference between the mean value and the standard value of the mean value is greater than the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0125] It is understandable that in the embodiment of the present application, there is no specific limitation on the size of the second tolerance threshold, which can be set according to actual conditions.

[0126] In a possible implementation, the state of the connector is determined based on the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points, including: generating a first feature vector based on the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; generating a second feature vector based on the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points.

[0127] Among them, the first feature vector includes the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; the second feature vector includes the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points.

[0128] Exemplarily, at least two key points include P1 and P2, and at least four key points include P3, P4, P5, and P6. Among them, the corrected coordinate information of P1, P2, P3, P4, P5, and P6 are (X1, Y1), (X2, Y2), (X3, Y3), (X4, Y4), (X5, Y5), and (X6, Y6), respectively. Based on the corrected coordinate information of these six key points, a 12-dimensional first eigenvector can be formed, and the first eigenvector can be expressed as A = [X1, Y1, X2, Y2, X3, Y3, X4, Y4, X5, Y5, X6, Y6].

[0129] Similarly, the standard coordinate information of P1, P2, P3, P4, P5, and P6 are (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5), and (x6, y6), respectively. Based on the standard coordinate information of these six key points, a 12-dimensional first eigenvector can be formed. The first eigenvector can be expressed as B = [x1, y1, x2, y2, x3, y3, x4, y4, x5, y5, x6, y6].

[0130] After the first feature vector and the second feature vector are generated, the cosine similarity between the first feature vector and the second feature vector may be calculated.

[0131] Exemplarily, for the first eigenvector A and the second eigenvector B, the cosine similarity COS(A, B)=A·B / (||A||||B||).

[0132] In an embodiment of the present application, when the cosine similarity is greater than a similarity threshold, it can be determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the cosine similarity is less than or equal to the similarity threshold, it can be determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0133] It is understandable that the embodiment of the present application does not specifically limit the size of the similarity threshold, which can be set according to actual conditions.

[0134] In summary, the method provided in the embodiment of the present application extracts key points from the edge of the plug in the image to obtain the first coordinate information of at least two key points corresponding to the plug, and extracts key points from the edge of the socket in the image to obtain the first coordinate information of at least four key points corresponding to the socket. On this basis, perspective distortion correction is performed on the first coordinate information of at least four key points to avoid the influence of perspective distortion on the coordinate information of the key points and ensure the accuracy of the comparison results. The process of determining the connector status based on accurate comparison results can effectively improve the accuracy of detection.

[0135] At the same time, key point extraction of images based on the deep learning model can automatically locate the position and category of the connector. There is no need to set the region of interest (ROI) in advance, no need to set complex edge extraction related parameters, and it is less affected by factors such as image shooting angle, lighting, and occlusion, which can effectively improve the extraction efficiency.

[0136] For ease of understanding, the following Fig. 9 An overall introduction is given to the connector status detection method provided in an embodiment of the present application. The figure is a schematic diagram of a connector status detection method provided in an embodiment of the present application.

[0137] Key point extraction is performed on the edge of the plug in the image including the connector to obtain first coordinate information of two key points corresponding to the plug, namely, first coordinate information of P1 and first coordinate information of P2. Key point extraction is performed on the edge of the socket in the image to obtain first coordinate information of four key points corresponding to the socket, namely, first coordinate information of P3, first coordinate information of P4, first coordinate information of P5 and first coordinate information of P6.

[0138] The coordinate information of P3, P4, P5 and P6 in the standard image corresponding to the connector is used as the standard coordinate information corresponding to P3, P4, P5 and P6. The perspective transformation matrix is ​​calculated based on the first coordinate information and the standard coordinate information corresponding to P3, P4, P5 and P6.

[0139] Based on the perspective transformation matrix, perspective distortion correction is performed on the first coordinate information of P1, P2, P3, P4, P5 and P6 to obtain corrected coordinate information corresponding to each of P1, P2, P3, P4, P5 and P6.

[0140] As described in the method of S104 above, the state of the connector can be determined according to the corrected coordinate information and the standard coordinate information corresponding to P1, P2, P3, P4, P5 and P6 respectively.

[0141] In summary, the method provided in the embodiment of the present application extracts key points from the edge of the plug in the image to obtain the first coordinate information of at least two key points corresponding to the plug, and extracts key points from the edge of the socket in the image to obtain the first coordinate information of at least four key points corresponding to the socket. On this basis, perspective distortion correction is performed on the first coordinate information of at least four key points to avoid the influence of perspective distortion on the coordinate information of the key points and ensure the accuracy of the comparison results. The process of determining the connector status based on accurate comparison results can effectively improve the accuracy of detection.

[0142] This application provides a connector status detection device, see Fig.10 , this figure is a structural schematic diagram of a connector status detection device provided in an embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above method, and some contents will not be repeated here.

[0143] The embodiment of the present application provides a connector state detection device 1300, including:

[0144] An acquisition unit 1301, an extraction unit 1302, a correction unit 1303 and a determination unit 1304;

[0145] The acquisition unit 1301 is used to acquire an image including a connector, where the connector includes a plug and a socket;

[0146] The extraction unit 1302 is configured to extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and to extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear;

[0147] The correction unit 1303 is used to perform perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points, so as to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points;

[0148] The determination unit 1304 is used to determine the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, wherein the state includes whether the plug and the socket are in a plugged in state or the plug and the socket are in a loosely plugged in state.

[0149] In a possible implementation, the correction unit is specifically configured to:

[0150] Calculating a perspective transformation matrix according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points;

[0151] Based on the perspective transformation matrix, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points.

[0152] In a possible implementation manner, the determining unit is specifically configured to:

[0153] The state of the connector is determined according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points.

[0154] In a possible implementation manner, the determining unit is specifically configured to:

[0155] Determine the edge line segment corresponding to the plug according to the corrected coordinate information of the at least two key points; determine the edge line segment corresponding to the socket according to the corrected coordinate information of at least two key points among the at least four key points, wherein the edge line segment corresponding to the socket does not intersect with the edge line segment corresponding to the plug;

[0156] Calculating the distance between the edge line segment corresponding to the plug and the corrected edge line segment corresponding to the socket to obtain a first distance;

[0157] The state of the connector is determined based on a comparison result obtained by comparing the first distance with a standard value of the first distance; the standard value of the first distance is calculated based on standard coordinate information of the at least two key points and standard coordinate information of at least two key points among the at least four key points.

[0158] In a possible implementation manner, the determining unit is specifically configured to:

[0159] When the comparison result is that the difference between the first distance and the standard value of the first distance is less than or equal to a first tolerance threshold, determining that the state of the connector is that the plug and the socket are in a tightly plugged state;

[0160] When the comparison result is that the difference between the first distance and the standard value of the first distance is greater than the first tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged together.

[0161] In a possible implementation manner, the determining unit is specifically configured to:

[0162] According to the corrected coordinate information of the first key point and the corrected coordinate information of the third key point, a second distance between the first key point and the third key point is calculated; according to the corrected coordinate information of the second key point and the corrected coordinate information of the fourth key point, a third distance between the second key point and the fourth key point is calculated; the at least two key points corresponding to the plug include the first key point and the second key point, and the at least four key points corresponding to the socket include the third key point and the fourth key point;

[0163] An edge line segment with the first key point and the second key point as endpoints does not intersect with an edge line segment with the third key point and the fourth key point as endpoints; an edge line segment with the first key point and the third key point as endpoints does not intersect with an edge line segment with the second key point and the fourth key point as endpoints;

[0164] Calculating a mean of the second distance and the third distance;

[0165] The state of the connector is determined according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point.

[0166] In a possible implementation manner, the determining unit is specifically configured to:

[0167] Determining a standard value of the mean according to the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point;

[0168] Comparing the mean with the standard value of the mean to obtain a comparison result;

[0169] When the comparison result is that the difference between the mean and the standard value of the mean is less than or equal to the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the comparison result is that the difference between the mean and the standard value of the mean is greater than the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0170] In a possible implementation manner, the determining unit is specifically configured to:

[0171] Generate a first feature vector based on the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; generate a second feature vector based on the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points; the first feature vector includes the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; the second feature vector includes the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points;

[0172] Calculating the cosine similarity between the first feature vector and the second feature vector;

[0173] When the cosine similarity is greater than a similarity threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the cosine similarity is less than or equal to the similarity threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

[0174] In a possible implementation, the extraction unit is specifically configured to:

[0175] Based on the key point detection model, key points are extracted from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and key points are extracted from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket; the key point detection model is trained based on the labeled data, and the labeled data is an image including the connector after the key points are labeled.

[0176] In summary, the device provided in the embodiment of the present application extracts key points from the edge of the plug in the image to obtain the first coordinate information of at least two key points corresponding to the plug, and extracts key points from the edge of the socket in the image to obtain the first coordinate information of at least four key points corresponding to the socket. On this basis, the first coordinate information of at least four key points is corrected for perspective distortion, thereby avoiding the influence of perspective distortion on the coordinate information of the key points and ensuring the accuracy of the comparison results. The process of determining the connector status based on accurate comparison results can effectively improve the accuracy of detection.

[0177] At the same time, key point extraction of images based on deep learning models can automatically locate the position and category of connectors. There is no need to set the area of ​​interest in advance, no need to set complex edge extraction related parameters, and it is less affected by factors such as image shooting angle, lighting, and occlusion, which can effectively improve the extraction efficiency.

[0178] An embodiment of the present application provides a computer device, the computer device comprising: a processor and a memory;

[0179] The memory is used to store program code and transmit the program code to the processor;

[0180] The processor is used to execute the steps of the connector status detection method as described above according to the instructions in the program code.

[0181] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the connector status detection method described above are implemented.

[0182] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A connector status detection method, characterized in that: include: acquiring an image including a connector, the connector including a plug and a socket; Extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug; extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear; According to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points; According to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, the state of the connector is determined, the state including that the plug and the socket are in a plugged in state, or that the plug and the socket are in a loosely plugged in state.

2. The method according to claim 1, characterized in that The step of performing perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points includes: Calculating a perspective transformation matrix according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points; Based on the perspective transformation matrix, perspective distortion correction is performed on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points.

3. The method according to claim 1, characterized in that The determining the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points includes: The state of the connector is determined according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points.

4. The method according to claim 3, characterized in that The determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes: Determine the edge line segment corresponding to the plug according to the corrected coordinate information of the at least two key points; determine the edge line segment corresponding to the socket according to the corrected coordinate information of at least two key points among the at least four key points, wherein the edge line segment corresponding to the socket does not intersect with the edge line segment corresponding to the plug; Calculating the distance between the edge line segment corresponding to the plug and the corrected edge line segment corresponding to the socket to obtain a first distance; The state of the connector is determined based on a comparison result obtained by comparing the first distance with a standard value of the first distance; the standard value of the first distance is calculated based on standard coordinate information of the at least two key points and standard coordinate information of at least two key points among the at least four key points.

5. The method according to claim 4, characterized in that The determining the state of the connector according to a comparison result obtained by comparing the first distance with a standard value of the first distance includes: When the comparison result is that the difference between the first distance and the standard value of the first distance is less than or equal to a first tolerance threshold, determining that the state of the connector is that the plug and the socket are in a tightly plugged state; When the comparison result is that the difference between the first distance and the standard value of the first distance is greater than the first tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged together.

6. The method according to claim 3, characterized in that The determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes: According to the corrected coordinate information of the first key point and the corrected coordinate information of the third key point, a second distance between the first key point and the third key point is calculated; according to the corrected coordinate information of the second key point and the corrected coordinate information of the fourth key point, a third distance between the second key point and the fourth key point is calculated; the at least two key points corresponding to the plug include the first key point and the second key point, and the at least four key points corresponding to the socket include the third key point and the fourth key point; An edge line segment with the first key point and the second key point as endpoints does not intersect with an edge line segment with the third key point and the fourth key point as endpoints; an edge line segment with the first key point and the third key point as endpoints does not intersect with an edge line segment with the second key point and the fourth key point as endpoints; Calculating a mean of the second distance and the third distance; The state of the connector is determined according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point.

7. The method according to claim 6, characterized in that Determining the state of the connector according to the mean, the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point includes: Determining a standard value of the mean according to the standard coordinate information of the first key point, the standard coordinate information of the second key point, the standard coordinate information of the third key point, and the standard coordinate information of the fourth key point; Comparing the mean with the standard value of the mean to obtain a comparison result; When the comparison result is that the difference between the mean and the standard value of the mean is less than or equal to the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the comparison result is that the difference between the mean and the standard value of the mean is greater than the second tolerance threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

8. The method according to claim 3, characterized in that The determining the state of the connector according to the corrected coordinate information of the at least two key points, the corrected coordinate information of the at least four key points, the standard coordinate information of the at least two key points, and the standard coordinate information of the at least four key points includes: Generate a first feature vector based on the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; generate a second feature vector based on the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points; the first feature vector includes the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points; the second feature vector includes the standard coordinate information of the at least two key points and the standard coordinate information of the at least four key points; Calculating the cosine similarity between the first feature vector and the second feature vector; When the cosine similarity is greater than a similarity threshold, it is determined that the state of the connector is that the plug and the socket are in a tightly plugged state; when the cosine similarity is less than or equal to the similarity threshold, it is determined that the state of the connector is that the plug and the socket are not tightly plugged.

9. The method according to any one of claims 1 to 8, characterized in that: The step of extracting key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and extracting key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket includes: Based on the key point detection model, key points are extracted from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and key points are extracted from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket; the key point detection model is trained based on the labeled data, and the labeled data is an image including the connector after the key points are labeled.

10. A connector status detection device, characterized in that: include: an acquisition unit, an extraction unit, a correction unit, and a determination unit; The acquisition unit is used to acquire an image including a connector, wherein the connector includes a plug and a socket; The extraction unit is used to extract key points from the edge of the plug in the image to obtain first coordinate information of at least two key points corresponding to the plug, and to extract key points from the edge of the socket in the image to obtain first coordinate information of at least four key points corresponding to the socket, wherein any three of the at least four key points are not collinear; The correction unit is used to perform perspective distortion correction on the first coordinate information of the at least two key points and the first coordinate information of the at least four key points according to the first coordinate information of the at least four key points and the standard coordinate information of the at least four key points, so as to obtain corrected coordinate information of the at least two key points and corrected coordinate information of the at least four key points; The determination unit is used to determine the state of the connector according to the corrected coordinate information of the at least two key points and the corrected coordinate information of the at least four key points, wherein the state includes whether the plug and the socket are in a plugged in state or the plug and the socket are in a loosely plugged in state.

11. A computer device, characterized in that: The computer device comprises: a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of a connector status detection method according to any one of claims 1 to 9 according to the instructions in the program code.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a connector status detection method according to any one of claims 1 to 9 are implemented.

Images