Computer vision-based device inspection method, device and apparatus

Abstract

The present specification relates to the technical field of equipment inspection, and provides an equipment inspection method and device based on computer vision, and equipment, the method comprising: acquiring a target image of a target equipment; performing common corner point detection on the target image to obtain a common corner point coordinate set in the target image; determining whether the target image meets a preset condition according to the common corner point coordinate set; when the target image meets the preset condition, performing perspective transformation on the target image according to a pre-constructed perspective transformation matrix and the common corner point coordinate set to obtain a corrected target image; identifying an interest region in the corrected target image; the interest region contains an indicator for indicating a running state of the target equipment by color; performing similarity matching of a color of the interest region with a preset color, and determining an inspection result of the target equipment according to a matching result. The present specification can improve the inspection efficiency and application range of equipment inspection.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of equipment inspection, and in particular to an equipment inspection method and device based on computer vision. BACKGROUND

[0002] Equipment inspection (such as electrical equipment inspection) can play a crucial role in timely understanding the running state of the equipment and ensuring the safety of the equipment. The traditional inspection mode needs to be found one by one by manual, and the inspection records report is copied by hand, which consumes a large amount of manual work, has low efficiency, and the standardization of the inspection action cannot be monitored, and the accuracy of the inspection result based on the subjective consciousness of the person is also difficult to guarantee. In order to solve the problems existing in the manual inspection, the current equipment automatic inspection scheme has appeared.

[0003] The equipment automatic inspection scheme based on the deep learning algorithm such as the convolutional neural network is one of the current mainstream equipment automatic inspection schemes, which can guarantee a high accuracy in the real-time monitoring scene, but also has problems such as high requirements for hardware, network, large resource consumption, and slow recognition rate, which is difficult to be applied on a large scale. Another type of method based on the visual system uses the Hough circle detection method and the V value of the target region HSV color space to determine the state of the indicator light, which has a high detection accuracy in a specific scene. However, the recognition method has high requirements for the circular regularity degree of the collected image and the image quality of the indicator light in the actual operation process, so it is also difficult to be applied on a large scale in the actual scene. Therefore, the current intelligent inspection method mostly stays in the laboratory research stage, and cannot be effectively applied to the actual scene. Therefore, how to improve the inspection efficiency and the application range of the equipment inspection has become a technical problem to be solved at present. SUMMARY

[0004] The purpose of the embodiments of the present specification is to provide an equipment inspection method and device based on computer vision to improve the inspection efficiency and the application range of the equipment inspection.

[0005] To achieve the above purpose, on the one hand, the embodiments of the present specification provide an equipment inspection method based on computer vision, comprising:

[0006] obtaining a target image of a target equipment;

[0007] performing common corner point detection on the target image to obtain a common corner point coordinate set in the target image;

[0008] confirming whether the target image meets a preset condition according to the common corner point coordinate set;

[0009] when the target image meets the preset condition, performing perspective transformation on the target image according to a pre-constructed perspective transformation matrix to obtain a corrected target image;

[0010] identifying an interest region in the corrected target image; the interest region containing an indicator for indicating a running state of the target device by color;

[0011] matching the color of the interest region with a preset color, and determining an inspection result of the target device according to a matching result.

[0012] In the device inspection method based on computer vision, the common corner point detection on the target image comprises:

[0013] The target image is divided into four sub-images according to a four-angle equal-division division rule; the four sub-images comprise a top-left sub-image, a top-right sub-image, a bottom-left sub-image, and a bottom-right sub-image;

[0014] Taking a specified sub-image in the four sub-images as a reference, the remaining sub-images in the four sub-images are flipped to make each flipped sub-image have the same position layout of black-and-white square chessboards as the reference;

[0015] Based on a black-and-white grid corner point detection method, common corner point detection is performed on the specified sub-image and each flipped sub-image to correspondingly obtain a first common corner point coordinate set of the specified sub-image and each flipped sub-image;

[0016] Each flipped sub-image is flipped and reset, and a second common corner point coordinate set of each flipped and reset sub-image is determined according to the first common corner point coordinate set of each flipped sub-image.

[0017] In the device inspection method based on computer vision, the specified sub-image comprises the top-left sub-image.

[0018] In the device inspection method based on computer vision, the common corner point detection logic of the black-and-white grid corner point detection method comprises:

[0019] The target sub-image is subjected to image binarization processing to form a first sub-image;

[0020] The first sub-image is subjected to image dilation processing to separate the connection between each black square in the first sub-image, to form a second sub-image;

[0021] The contours of all black squares in the second sub-image are extracted to form a third sub-image;

[0022] For each pair of diagonally adjacent square contours in the third sub-image, the midpoint coordinates of the connecting line of the two points of the specified diagonal of the pair of square contours are taken as common corner point coordinates to form a first common corner point coordinate set of the target sub-image.

[0023] In the computer vision-based device inspection method of the embodiments of this specification, the common corner detection logic of the black and white grid corner detection method further includes:

[0024] Interfering square contours in the extracted contours are removed by removing them by perimeter or area, so that the third sub-image does not contain the interfering square contours.

[0025] In the computer vision-based device inspection method of this specification embodiment, the designated diagonal includes the farthest diagonal of the grid outline.

[0026] In the computer vision-based device inspection method of this specification, the step of confirming whether the target image meets the preset conditions based on the common corner point coordinate set includes:

[0027] For each sub-image of the target image, all grid contours corresponding to the sub-image are traversed, and isolated grid contours are removed according to the set of common corner coordinates of the sub-image; the isolated grid contours are grid contours that do not contain common corners.

[0028] Based on the set of common corner point coordinates corresponding to the subgraph and its correspondence with all the remaining squares after removing the isolated square outlines of the subgraph, determine the overall structural layout of all the remaining squares after removing the isolated square outlines of the subgraph.

[0029] Determine whether the overall structural layout matches the preset structural layout;

[0030] If the overall structural layout matches the preset structural layout, then it is confirmed that the sub-graph contains the expected black and white checkerboard.

[0031] When all sub-images of the target image contain the expected black and white checkerboard pattern, the target image is confirmed to meet the preset conditions.

[0032] In the computer vision-based equipment inspection method of this specification, the step of performing perspective transformation on the target image according to a pre-constructed perspective transformation matrix includes:

[0033] The target image is then subjected to perspective transformation according to the following formula:

[0034]

[0035] Determine the outer contour of the black and white checkerboard in the target image after perspective transformation;

[0036] Cropping out the edge portions of the target image that are outside the outer contour after perspective transformation;

[0037] Where x, y, z are the original coordinates of the cropped target image, and x', y', z' are the coordinates of the cropped target image after perspective transformation. Let a be the perspective transformation matrix. 11 ~a 33 These are elements in the perspective transformation matrix.

[0038] In the computer vision-based device inspection method of this specification embodiment, the similarity matching of the color of the region of interest with a preset color includes:

[0039] Randomly select a specified number of points from the region of interest;

[0040] Calculate the color similarity between the color of each selected point and the preset color to form a color similarity set;

[0041] Determine the percentage of colors in the color similarity set that reach the color similarity threshold;

[0042] The color similarity percentage is compared with the percentage threshold, and the matching result is determined based on the comparison result.

[0043] In the computer vision-based equipment inspection method of this specification, the step of determining the inspection result of the target equipment based on the matching result includes:

[0044] When the color similarity ratio reaches the ratio threshold, it is confirmed that the target device is in a normal state;

[0045] When the color similarity ratio does not reach the ratio threshold, the target device is confirmed to be in an abnormal state.

[0046] In the computer vision-based device inspection method of this specification, the preset color is determined in advance by the following method:

[0047] Pre-acquire images of the target device and perform image correction on those images;

[0048] A region of interest is selected from the image-corrected target image using a randomly chosen selection box; the region of interest contains an image of an indicator.

[0049] Cluster all colors within the region of interest based on a pre-defined clustering method;

[0050] Calculate the number of points contained in each color category within the region of interest;

[0051] The number of points contained in each color category in the region of interest is sorted, and the primary and secondary colors in the region of interest are determined based on the sorting results.

[0052] When the main color in the region of interest is the same as the color corresponding to the indicator in the specified state, the main color is used as the preset color, and the selection box is used as the selection box for identifying the region of interest during subsequent inspections.

[0053] When the secondary color in the region of interest is the same as the color corresponding to the indicator in a specified state, the corresponding secondary color is used as the preset color, and the selection box is used as the selection box for identifying the region of interest during subsequent inspections.

[0054] On the other hand, embodiments of this specification also provide a computer vision-based equipment inspection device, including:

[0055] The image acquisition module is used to acquire target images from the target device.

[0056] The corner detection module is used to detect common corners in the target image to obtain a set of coordinates of common corners in the target image.

[0057] The condition judgment module is used to determine whether the target image meets preset conditions based on the common corner point coordinate set;

[0058] A perspective transformation module is used to perform perspective transformation on the target image according to a pre-constructed perspective transformation matrix when the target image meets the preset conditions, so as to obtain a corrected target image.

[0059] A region recognition module is used to identify regions of interest in the corrected target image; the regions of interest contain indicators for indicating the operating status of the target device by means of color;

[0060] The similarity matching module is used to match the color of the region of interest with a preset color based on similarity, and determine the inspection result of the target device based on the matching result.

[0061] On the other hand, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when run by the processor, executes instructions for the above-described method.

[0062] On the other hand, embodiments of this specification also provide a computer storage medium storing a computer program thereon, which, when run by the processor of a computer device, executes instructions for the above-described method.

[0063] On the other hand, embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the above-described method.

[0064] As can be seen from the technical solutions provided in the embodiments of this specification above, the embodiments of this specification utilize image correction technology and color recognition technology in computer vision to achieve rapid identification of the operating status of equipment. Compared with automatic equipment inspection schemes based on deep learning algorithms such as convolutional neural networks, the embodiments of this specification do not require the high computational load in convolutional neural network scenarios, have lower requirements for hardware, network and other resources, and have a faster recognition speed. Compared with equipment inspection methods such as the Hough circle detection method, the embodiments of this specification have no special requirements for the shape of the equipment's indicators. Therefore, the embodiments of this specification can be applied to most practical inspection application scenarios, thereby improving the inspection efficiency and applicability of equipment inspection. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0066] Figure 1 This specification shows schematic diagrams of computer vision-based equipment inspection systems in some embodiments;

[0067] Figure 2 Flowcharts of computer vision-based equipment inspection methods in some embodiments of this specification are shown;

[0068] Figure 3 It shows Figure 2 The flowchart shown illustrates the method for detecting common corner points in the target image.

[0069] Figure 4 It shows Figure 3 The flowchart of the common corner detection logic of the black and white grid corner detection method shown is as follows;

[0070] Figure 5 It shows Figure 2 The flowchart shown illustrates the method for determining whether a target image meets preset conditions based on a set of common corner coordinates.

[0071] Figure 6 It shows Figure 2 The flowchart shown illustrates the process of matching the color of the region of interest with a preset color based on similarity, and determining the inspection result of the target device based on the matching result.

[0072] Figure 7 A target image of the device obtained in an exemplary embodiment of this specification is shown;

[0073] Figure 8 It shows Figure 7 The diagram shows the segmentation of the target image.

[0074] Figure 9 It shows Figure 7 The image shown is a schematic diagram of a sub-image in the binary-processed black and white checkerboard area of ​​the target image.

[0075] Figure 10 It shows Figure 9 The diagram shows a black and white checkerboard after image inflation.

[0076] Figure 11 It shows from Figure 10 A schematic diagram showing the outlines of multiple squares extracted from the black and white checkerboard shown;

[0077] Figure 12 It shows from Figure 11 A schematic diagram showing the determination of common corner points in multiple grid outlines;

[0078] Figure 13 It shows Figure 7 The target image shown is a front view after perspective transformation;

[0079] Figure 14 It shows Figure 13 A schematic diagram showing the front view after the edges have been cropped.

[0080] Figure 15 This diagram illustrates a plurality of square outlines (including interfering square outlines) extracted from an image-dilated black-and-white checkerboard in an exemplary embodiment of this specification.

[0081] Figure 16 This specification shows a structural block diagram of a computer vision-based equipment inspection device in some embodiments;

[0082] Figure 17 A structural block diagram of a computer device in some embodiments of this specification is shown.

[0083] [Explanation of Labels in the Attached Image]

[0084] 10. Client-side application;

[0085] 20. Server-side;

[0086] 30. A black and white checkerboard;

[0087] 40. Indicator lights;

[0088] 50. Common corner points;

[0089] 161. Image acquisition module;

[0090] 162. Corner Detection Module;

[0091] 163. Conditional Judgment Module;

[0092] 164. Perspective Transformation Module;

[0093] 165. Region Identification Module;

[0094] 166. Similarity matching module;

[0095] 1702. Computer equipment;

[0096] 1704, Processor;

[0097] 1706. Memory;

[0098] 1708. Drive mechanism;

[0099] 1710. Input / output interface;

[0100] 1712. Input devices;

[0101] 1714. Output devices;

[0102] 1716. Presentation equipment;

[0103] 1718. Graphical User Interface;

[0104] 1720. Network interface;

[0105] 1722. Communication link;

[0106] 1724. Communication bus. Detailed Implementation

[0107] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0108] The embodiments in this specification relate to computer vision-based automatic equipment inspection technology to improve the inspection efficiency and applicability of equipment inspection. It can be applied to any suitable equipment (e.g., electrical equipment, power equipment, communication equipment, etc.) inspection scenario.

[0109] Figure 1This document illustrates schematic diagrams of a computer vision-based equipment inspection system according to some embodiments of this specification. The system may include a client 10 and a server 20. The server 20 can acquire target images of the target equipment from the client 10. Utilizing image correction and color recognition technologies in computer vision, it achieves rapid identification of the equipment's operating status. It has low requirements for hardware, network, and other resources, and does not have special requirements for the shape of the indicators on the equipment, making it suitable for most practical inspection application scenarios, thereby improving the efficiency and applicability of equipment inspection. The indicators are used to indicate the operating status of the equipment through color (e.g., green indicates normal status, red indicates abnormal status or malfunction). For example, in some embodiments, the indicators may be color indicator lights (hereinafter referred to as indicator lights), but this specification is not limited to this, and other indicators (e.g., displays) may be selected as needed.

[0110] In some embodiments, the client 10 can be an image acquisition device (such as a digital camera, webcam, etc.), a mobile terminal (i.e., a smartphone), a tablet computer, a laptop computer, a digital assistant, or a smart wearable device. Smart wearable devices can include smart bracelets, smartwatches, smart glasses, or smart helmets. Of course, the client is not limited to the aforementioned physical electronic devices; it can also be software running on such electronic devices. The server 20 can be an electronic device with computing and network interaction functions; it can also be software running on that electronic device that provides business logic for data processing and network interaction.

[0111] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided in this specification. In actual applications, there can be multiple clients and multiple servers, and this specification does not impose any restrictions.

[0112] This specification provides a computer vision-based device inspection method, which can be applied to the aforementioned server-side. (Refer to...) Figure 2 As shown, in some embodiments, the computer vision-based device inspection method may include the following steps:

[0113] Step 201: Obtain the target image of the target device.

[0114] Step 202: Perform common corner detection on the target image to obtain the coordinate set of common corner points in the target image.

[0115] Step 203: Confirm whether the target image meets the preset conditions based on the common corner point coordinate set.

[0116] Step 204: When the target image meets the preset conditions, perform perspective transformation on the target image according to the pre-constructed perspective transformation matrix to obtain the corrected target image.

[0117] Step 205: Identify the region of interest in the corrected target image; the region of interest contains an indicator light for indicating the operating status of the target device by means of color.

[0118] Step 206: Match the color of the region of interest with a preset color based on similarity, and determine the inspection result of the target device based on the matching result.

[0119] This specification's embodiments employ image correction techniques based on common corner detection and perspective transformation, along with color recognition techniques based on similarity calculation, to achieve automatic equipment inspection. Since the overall computational load and complexity of common corner detection, perspective transformation, and similarity calculation are less than those of deep learning algorithms, this specification's embodiments reduce resource consumption and improve inspection efficiency. Furthermore, the lower computational load and complexity also mean lower requirements for hardware and network requirements, making this specification's embodiments applicable even in equipment inspection scenarios with limited hardware and network configurations, thus expanding the applicability of equipment inspection. Moreover, in this specification's embodiments, regardless of the shape of the indicator, the region of interest containing the indicator can be selected from the corrected target image. Therefore, compared to using the Hough circle detection method and the V value of the target region's HSV color space to determine the indicator's state, this specification's embodiments are applicable to equipment inspection scenarios with indicators of arbitrary shapes, thereby expanding the applicability of equipment inspection. The target equipment is the equipment to be inspected (e.g., electrical equipment to be inspected). In some embodiments, there may be one or more devices that need to be inspected. When there are multiple devices that need to be inspected, the server can identify the status of each device that needs to be inspected one by one based on the image corresponding to each device. When the server has parallel processing capabilities (e.g., the server is a server cluster), the server can identify the status of each device that needs to be inspected concurrently.

[0120] The target image is an image that includes the indicator light portion of the target device. To improve image acquisition efficiency and the accuracy of device status recognition, a black and white checkerboard pattern can be pre-marked around the four corners of the indicator light of each device to be inspected; the black and white checkerboard pattern serves as a marker to indicate the area to be photographed; if the acquired indicator light image includes the area to be photographed, it indicates that the indicator light image meets the requirements.

[0121] It should be noted that the black and white checkerboard pattern has significant recognizability for most devices (i.e., it is easy to identify and not easily confused with other devices); however, as an exemplary illustration of an embodiment of this specification, the black and white checkerboard pattern is not mandatory as a marker. In other embodiments, the black and white checkerboard pattern can be replaced with any other marker that can have a similar function, and this specification does not limit this to a single one. Of course, when using other markers, the technical means of detecting other markers should also be adapted accordingly; however, regardless of the marker used, image correction and color recognition should be included.

[0122] like Figure 7 As shown, in a black and white checkerboard, black and white squares are arranged alternately, and each black square is connected to at least one other black square through a corner. Therefore, for any two adjacent black squares, they share a common corner position at their connection point; this common corner position is called a common corner point. By detecting common corner points in a target image, it can be used to identify whether the target image contains a black and white checkerboard, and thus to evaluate whether the image meets preset conditions.

[0123] refer to Figure 3 As shown, in some embodiments, detecting common corners of a target image may include the following steps:

[0124] Step 301: Divide the target image into four sub-images according to the four-corner equal division rule; the four sub-images include the upper left sub-image, the upper right sub-image, the lower left sub-image, and the lower right sub-image.

[0125] The four-corner equal division rule refers to dividing an image equally using two dividing lines, one horizontal and one vertical, passing through the center point of the image. For example, using... Figure 7 Taking the target image shown as an example, dividing it according to the four-corner equal division rule can yield the following results: Figure 8 The four sub-graphs shown are: Figure 8 The top-left subgraph, top-right subgraph, bottom-left subgraph, and bottom-right subgraph in the image. Figure 7 and Figure 8 In the center, there is an indicator light 40 for the device, and the four corner sections are a black and white checkerboard 30.

[0126] Step 302: Using a designated subgraph among the four subgraphs as a reference, flip the remaining subgraphs among the four subgraphs so that each flipped subgraph has the same black and white checkerboard layout as the reference.

[0127] By using a specified subgraph as a reference and flipping the other three subgraphs (i.e., symmetrical image flipping), each flipped subgraph can have a black and white checkerboard layout with the same position as the reference. This makes it easier to repeatedly call the same function for calculation, thereby reducing code duplication and improving code utilization.

[0128] In some embodiments, the designated subgraph can be the top-left subgraph. When the tilt angle of the image is different, the order of the subsequently calculated common corner point coordinate set will differ; for example, it can be arranged in row order or column order. Regardless of whether it is arranged in row order or column order, the top-left subgraph is always the first subgraph. Therefore, using the top-left subgraph as the reference can reduce the amount of computation. In other embodiments, the designated subgraph can also be other subgraphs selected as the reference as needed, and this specification does not limit this to a single one.

[0129] by Figure 8 Taking the illustrated embodiment as an example, when the specified subgraph is the top-left subgraph (i.e., based on the top-left subgraph), Figure 8 The bottom left image can be flipped upwards. Figure 8 The top right image in the image can be flipped to the left. Figure 8 The bottom right image can be flipped to the left and then to the top (or flipped to the top and then to the left). After flipping, all sub-images have a black and white checkered board with the same layout (i.e., the black and white checkered board in all sub-images is located in the upper left of the sub-image).

[0130] Step 303: Based on the black and white grid corner detection method, perform common corner detection on the specified sub-image and each of the flipped sub-images respectively, so as to obtain the first common corner coordinate set of the specified sub-image and each of the flipped sub-images.

[0131] Step 304: Reset each flipped subgraph, and determine the second common corner coordinate set of each flipped subgraph based on the first common corner coordinate set of each flipped subgraph.

[0132] In step 303, since the specified subgraph is not flipped, the first common corner coordinate set of the specified subgraph is the common corner coordinate set of the subgraph. However, for the other three subgraphs that have been flipped, since their first common corner coordinate sets are calculated in the flipped state, they cannot be directly used as the common corner coordinate sets of the subgraphs. Therefore, it is necessary to flip and reset each subgraph after flipping (i.e., restore it to the state before flipping), and determine the second common corner coordinate set of each subgraph after flipping and resetting based on the first common corner coordinate set of each subgraph after flipping. Then, the second common corner coordinate set of each subgraph after flipping and resetting can be used as the common corner coordinate set of the corresponding subgraph.

[0133] In the embodiments of this specification, the set of common corner point coordinates corresponds one-to-one with the black and white square chessboard. The set of common corner point coordinates is the set of coordinates of multiple common corner points under the same black and white square chessboard (a black and white square chessboard generally has multiple common corner points).

[0134] refer to Figure 4 As shown, in some embodiments, the common corner detection logic of the above-described black and white grid corner detection method may include:

[0135] Step 401: Perform image binarization on the target sub-image to form the first sub-image.

[0136] The first sub-image is a binary image.

[0137] Step 402: Perform image dilation processing on the first sub-image to separate the connections between each black square in the first sub-image, forming a second sub-image.

[0138] In the embodiments of this specification, image dilation refers to expanding the boundary points of the white squares in the first sub-image, thereby forcing a reduction in the boundary points of the black squares in the first sub-image, and thus separating the connections between each black square in the first sub-image. In an exemplary embodiment, an omnidirectional dilation method can be used to dilate the white squares in the first sub-image. Omnidirectional dilation means that when a boundary point is a white pixel, all surrounding pixels adjacent to it are changed to white pixels.

[0139] For example, with Figure 9 Taking the image shown as an example, by performing image dilation processing, we can obtain the following: Figure 10 The image shown; comparison Figure 9 and Figure 10 As can be seen, after image dilation, the connection between each black square is separated, making each black square separate from each other (of course, the actual separation distance is relatively small, possibly only one or a few pixels apart); thus, the overall layout of the black and white checkerboard is not changed, and the accuracy of contour extraction is improved.

[0140] Step 403: Extract the outlines of all black squares in the second sub-image to form the third sub-image.

[0141] Contour extraction can be performed using any suitable contour extraction method. For example, the cvFindContours function in the OpenCV library (an open-source image processing library) can be called to extract the contours of all black squares in the second sub-image. In an exemplary embodiment, using... Figure 10 Taking the image shown as an example, by performing contour extraction processing, we can obtain the following: Figure 11 The outline image shown.

[0142] Step 404: Remove interfering square contours from the extracted contours by perimeter or area, so that the third sub-image does not contain the interfering square contours.

[0143] After contour extraction, the contour of each square can be obtained, but there are also many interfering contours (e.g., Figure 15 The extracted contours contain some interfering contours. To avoid misjudging the overall structure and layout of the corresponding black and white checkerboard due to interfering contours, interfering square contours in the extracted contours can be removed by perimeter or area. Specifically, in an exemplary embodiment, the cvApproxPoly function of the OpenCV library can be called first to perform polygon fitting on the contours to exclude contours that are not square, and then interfering square contours in the extracted contours can be removed by perimeter or area.

[0144] It should be noted that step 404 is an optional step; in other embodiments, step 404 may be omitted as needed.

[0145] Step 405: For each diagonally adjacent square grid contour in the third sub-image, take the midpoint coordinates of the line connecting two points at a specified diagonal of the square grid contour as the common corner coordinates to form the first common corner coordinate set of the target sub-image.

[0146] Due to the third sub-image (e.g.) Figure 11 In the diagram, the spacing between diagonally adjacent square outlines is very small, possibly only one or a few pixels. To improve the accuracy of identifying common corner points, the diagonal can be specified as the farthest diagonal of that square outline.

[0147] For example, in Figure 12 In the exemplary embodiment shown, grid outline b1 and grid outline b4 are diagonally adjacent. Taking the specified diagonal as the farthest diagonal as an example, the line connecting the two points of the specified diagonal (see...) Figure 12 The midpoint of the dashed line in the image can be used as a common corner point 50 for identification; correspondingly, the coordinates of this midpoint are the coordinates of the identified common corner point. Thus, through the processing in step 405, the coordinates of all common corner points in the third sub-image can be identified, and these common corner point coordinates form the first common corner point coordinate set of the target sub-image.

[0148] refer to Figure 5 As shown, in some embodiments, confirming whether the target image meets preset conditions based on a common corner coordinate set may include the following steps:

[0149] Step 501: For each sub-image of the target image, traverse all the grid contours corresponding to that sub-image and remove isolated grid contours based on the set of common corner coordinates of the sub-image; the isolated grid contour is a grid contour that does not contain common corners. In other words, among all the grid contours corresponding to the same sub-image, if a grid contour has no intersection with other grid contours, then that grid contour is an isolated grid contour. By removing isolated grid contours, the recognition accuracy of the overall grid structure layout of the sub-image can be improved.

[0150] Step 502: Based on the coordinate set of the common corner points corresponding to the subgraph and its correspondence with all the remaining squares after removing the isolated square outlines of the subgraph, determine the overall structural layout of all the remaining squares after removing the isolated square outlines of the subgraph.

[0151] like Figure 12 As shown, based on the set of coordinates of the common corner points corresponding to a subgraph (i.e. Figure 12 The layout of these common corner points (i.e., planar arrangement) can be determined by the black dots in the graph. Based on this, the overall structural layout of the grid outline in the subgraph can be determined by the relationship between the coordinates of each common corner point in the coordinate set and the grid outline (for example, grid outlines b1 and b4 have a common corner point A; grid outlines b2 and b4 have a common corner point B).

[0152] Step 503: Determine whether the overall structural layout matches the preset structural layout. If the overall structural layout does not match the preset structural layout, proceed to step 504; otherwise, proceed to step 505.

[0153] The preset structural layout refers to the overall structural layout of the black squares corresponding to the black squares on the black and white checkerboard around the four corners of the indicator lights of the equipment to be inspected. For example, if the preset structural layout is a 3×3 square matrix, and the overall structural layout obtained in step 502 is also a 3×3 square matrix, then it is confirmed that the overall structural layout matches the preset structural layout.

[0154] Step 504: Confirm that the target image does not meet the preset conditions.

[0155] If the black and white checkerboard pattern pre-set around the four corners of the indicator light is not fully and completely included in the shooting range when capturing the target image, at least one sub-image will have a missing or incomplete black and white checkerboard pattern. This will cause the overall structural layout of the checkerboard pattern of the sub-image to not match the preset structural layout. In this case, the target image will not meet the preset conditions.

[0156] In some embodiments, when it is confirmed that the target image does not meet the preset conditions, response processing logic (such as outputting an alarm prompt) can also be executed to facilitate subsequent image re-capture.

[0157] Step 505: Confirm that the subgraph contains the expected black and white checkerboard.

[0158] Step 506: Determine whether all sub-images of the target image have been processed; if all sub-images of the target image have been processed, proceed to step 507; otherwise, proceed to step 501 to continue processing the next sub-image of the target image.

[0159] Step 507: Confirm that the target image meets the preset conditions.

[0160] When all sub-images of the target image contain the expected black and white checkerboard pattern, the target image is confirmed to meet the preset conditions, thereby triggering subsequent correction processing.

[0161] In some embodiments, performing perspective transformation on the target image based on a pre-constructed perspective transformation matrix may include:

[0162] (1) Perform perspective transformation on the target image according to the following formula:

[0163]

[0164] Where x, y, z are the original coordinates of the cropped target image, and x', y', z' are the coordinates of the cropped target image after perspective transformation. Let a be the perspective transformation matrix. 11 ~a 33 These are elements in the perspective transformation matrix.

[0165] Perspective transformation refers to the transformation that utilizes the collinearity of the perspective center, image point, and target point, and rotates the projection plane (perspective plane) around the trace line (perspective axis) by a certain angle according to the laws of perspective rotation. This disrupts the original projection ray beams while maintaining the geometric shape of the projected image on the projection plane. Essentially, it projects an image onto a new viewpoint or plane, altering the apparent perspective by distorting the image to form a two-dimensional orthographic view.

[0166] The pre-constructed perspective transformation matrix refers to mapping the common corner coordinate set x of any sub-image in the target image to the common corner coordinate set y of the corresponding sub-image in the preset standard image. This mapping determines the transformation matrix z required to transform from the common corner coordinate set x to the common corner coordinate set y. This transformation matrix is ​​the perspective transformation matrix. Since the perspective transformation matrix needs to be determined before performing the perspective transformation, it is called the pre-constructed perspective transformation matrix.

[0167] For example, with Figure 7Taking the target image shown as an example, through perspective transformation, we can obtain the following: Figure 13 The target image after perspective transformation is shown.

[0168] (2) Determine the outer contour of the black and white checkerboard in the target image after perspective transformation.

[0169] exist Figure 13 In the exemplary embodiment shown, a perspective-transformed target image is illustrated. The outer contour enclosed by the black and white checkerboard in the perspective-transformed target image can be as follows: Figure 13 As shown in the dashed box in the image.

[0170] (3) Crop the edge portion of the target image after perspective transformation that is outside the outer contour.

[0171] After perspective transformation and cropping, the corrected target image is obtained. For example, with... Figure 13 Taking the perspective-transformed target image as an example, through cropping, we can obtain the following: Figure 14 The corrected target image is shown. Figure 13 and Figure 14 In the center, there is an indicator light 40 for the device, and the four corner sections are a black and white checkerboard 30.

[0172] The above-mentioned computer vision-based image correction processing can reduce or eliminate image distortion caused by aberrations and distortions in the imaging system, thereby helping to reduce the increase in color recognition and matching errors caused by image distortion.

[0173] refer to Figure 6 As shown, in some embodiments, matching the color of the region of interest with a preset color based on similarity may include the following steps:

[0174] Step 601: Randomly select a specified number of points from the region of interest.

[0175] The region of interest (ROI) is the area covered by the indicator lights in the corrected target image. When identifying the ROI in the corrected target image, a box (or other shaped selection box) can be used to select the indicator lights in the corrected target image; the area within the box is then the ROI. For example, using... Figure 14 Taking the corrected target image shown as an example, by selecting the area, one can obtain the following: Figure 14 The region of interest is indicated by the dashed box in the image. The extent of the region of interest can be preset. In some embodiments, the region of interest in the corrected target image can be identified using the OpenCV mouse click method or the AirCV coordinate matching method.

[0176] Compared to selecting all points within a region of interest, randomly selecting a subset of points can reduce computational load. Of course, to ensure recognition accuracy, the number of randomly selected points should not be too small, and can be appropriately set according to the actual application scenario.

[0177] Step 602: Calculate the color similarity between the color of each selected point and the preset color to form a color similarity set.

[0178] By comparing the color of each selected point with a preset color, the color similarity value between the selected point and the preset color can be obtained.

[0179] The preset color can be the color corresponding to the indicator light in a specified state. For example, in some embodiments, the preset color can be the color corresponding to the indicator light in a normal state. Obviously, in other embodiments, the preset color can also be set to the color corresponding to the indicator light in an abnormal state.

[0180] In some embodiments, taking the preset color as the color corresponding to the indicator light being in its normal state as an example, all colors within the region of interest in the pre-acquired and corrected image can be clustered based on clustering methods (such as K-Means clustering, spectral clustering, DBSCAN clustering, or BIRCH clustering, etc.), and the primary and secondary colors within the region of interest can be determined based on the clustering results. Specifically, this may include the following steps:

[0181] (1) Pre-acquire images of the equipment to be inspected and perform image correction on the images.

[0182] (2) Use a randomly selected selection box to select a region of interest from the image-corrected target image; the region of interest contains an indicator light image.

[0183] (3) Cluster all colors within the region of interest based on the preset clustering method.

[0184] (4) Calculate the number of points contained in each color category within the region of interest. The number of points can be the number of pixels.

[0185] (5) Sort the number of points contained in each color category in the region of interest, and determine the primary and secondary colors in the region of interest based on the sorting results.

[0186] (6) When the primary color in the area of ​​interest is the same as the color corresponding to the indicator light in normal state, the primary color is used as the preset color, and the selection box is used as the selection box for identifying the area of ​​interest during subsequent inspections; when the secondary color in the area of ​​interest is the same as the color corresponding to the indicator light in normal state, the corresponding secondary color is used as the preset color, and the selection box is used as the selection box for identifying the area of ​​interest during subsequent inspections.

[0187] Clearly, when the randomly selected selection box is large, the background portion of the area within the region of interest occupies the majority of the area (i.e., the background color is the primary color), while the illuminated portion of the indicator light occupies the minority of the area (the indicator light color is the secondary color). Therefore, the secondary color can be used as the preset color. When the randomly selected selection box is small, the background portion of the area within the region of interest occupies the minority of the area (i.e., the background color is the secondary color), while the illuminated portion of the indicator light occupies the majority of the area (the indicator light color is the primary color). Therefore, the primary color can be used as the preset color. This not only avoids manually defining the color for each area, reducing human workload and human-induced deviations, but also lowers the requirements for the selection box of the region of interest.

[0188] For example, in an exemplary embodiment, when the selection box [x,y,w,h] selects a region of interest from the corrected image containing three colors: red, yellow, and green, and the number of points contained in the region of interest for each color is ordered from highest to lowest as follows: green → yellow → red (i.e., green is the primary color, and yellow and red are secondary colors); if the indicator light is green when it is in a normal state, then green can be used as the preset color, and the selection box [x,y,w,h] can be used as the selection box for identifying the region of interest during subsequent inspections. Here, x and y are the coordinates of the center point in the region of interest, and w and h are the length and width of the region of interest.

[0189] In some embodiments, the preset color and the calculated color can be RGB values, Lab values, HSV values, or CIEDE2000 values, etc. For example, both the preset color and the calculated color can use RGB values.

[0190] In some other embodiments, since the ambient light intensity of the equipment to be inspected may vary (e.g., strong light at midday and weak light at night), even if the indicator light displays the same color, the color may not be exactly the same under different light intensities. Therefore, multiple approximate preset colors are set for the same indicator light, forming a preset color set. During calculation, the color of a certain point can be compared with each preset color in the preset color set; when the similarity between the color of that point and one of the preset colors reaches a color similarity threshold, the two colors are considered to match.

[0191] In other embodiments, when there are multiple devices that need to be inspected, or when a device that needs to be inspected has multiple indicator lights for indicating the operating status of different functional units, each device that needs to be inspected or each indicator light can be pre-configured with a corresponding preset color. For example, the preset color settings (including selection box settings) for each indicator light can be shown in the following matrix:

[0192]

[0193] Among them, color i This represents the color identifier set (color cluster) for the i-th device that needs to be inspected; [x i ,y i ,w i ,h i [x] represents the region of interest (ROI) range corresponding to the target image of the i-th device that needs to be inspected. i ,y i Let w be the coordinates of the center point in the region of interest corresponding to the target image of the i-th device that needs to be inspected. i ,h i c represents the length and width of the region of interest corresponding to the target image of the i-th device that needs to be inspected; i For color i The number of colors in (i.e., color) i (number of elements); pid i For color i The selected preset color corresponds to the color category number; where 1≤i≤n.

[0194] Step 603: Determine the percentage of color similarities in the color similarity set that reach the color similarity threshold.

[0195] The color similarity threshold is a preset value. When the color of a point reaches the preset color similarity value, the color of the point can be considered as the preset color; otherwise, the color of the point can be considered as not being the preset color. Based on this, the proportion of color similarity values ​​that reach the color similarity threshold in the color similarity set can be determined.

[0196] In some implementations, the color similarity threshold can be calculated in advance according to the following formula.

[0197] ΔV=K E -1 ΔE

[0198]

[0199] Where ΔV is the acceptable color difference (i.e., the color similarity threshold), and K E -1It represents visual sensitivity; ΔL', ΔC', and ΔH' represent the difference in brightness, chromaticity, and hue, respectively; K L K C K H These are the lightness weighting factor, chroma weighting factor, and hue weighting factor, which are related to the experimental conditions; S L S C S H These are the factors for correcting lightness, chroma, and hue, respectively; R T It is a rotation function used to correct the deflection of the blue area along the elliptical axis for visual perception.

[0200] Step 604: Determine whether the color similarity percentage reaches the percentage threshold. If the color similarity percentage reaches the percentage threshold, proceed to step 605; otherwise, proceed to step 606.

[0201] Step 605: Confirm that the target device is in normal condition.

[0202] When the color similarity ratio reaches the ratio threshold, the target device is confirmed to be in normal condition; thereafter, the next device (if any) can be inspected.

[0203] Step 606: Confirm that the target device is in an abnormal state.

[0204] When the color similarity ratio does not reach the specified threshold, the target device is confirmed to be in an abnormal state. An alarm can then be output to alert relevant personnel for appropriate handling. For example, in some embodiments, if the detection result is abnormal, an alarm can be output, and a corresponding solution can be automatically retrieved from the expert database, quickly notifying relevant personnel for processing. After the abnormal alarm is handled, the processing method and result are recorded and stored in the expert database.

[0205] In other instances, the server can also generate inspection reports based on the equipment inspection results.

[0206] Although the process described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, which may be executed sequentially or in parallel (e.g., using parallel processors or a multithreaded environment).

[0207] Corresponding to the above-described computer vision-based equipment inspection method, this specification also provides a computer vision-based equipment inspection device, see reference. Figure 16 As shown, in some embodiments, the computer vision-based equipment inspection device may include:

[0208] Image acquisition module 161 is used to acquire target images of the target device;

[0209] Corner detection module 162 is used to perform common corner detection on the target image to obtain a set of common corner coordinates in the target image;

[0210] Condition judgment module 163 is used to determine whether the target image meets preset conditions based on the common corner point coordinate set;

[0211] The perspective transformation module 164 is used to perform perspective transformation on the target image according to a pre-constructed perspective transformation matrix when the target image meets the preset conditions, so as to obtain a corrected target image.

[0212] The region recognition module 165 is used to identify regions of interest in the corrected target image; the regions of interest include indicator lights for indicating the operating status of the target device by means of color;

[0213] The similarity matching module 166 is used to match the color of the region of interest with a preset color based on similarity, and to determine the inspection result of the target device based on the matching result.

[0214] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) that may be involved in the embodiments of this specification are all information and data that have been authorized and agreed to by the user and have been fully authorized by all parties.

[0216] Embodiments of this specification also provide a computer device. For example... Figure 17As shown, in some embodiments of this specification, the computer device 1702 may include one or more processors 1704, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each of which may implement one or more hardware threads. The computer device 1702 may also include any memory 1706 for storing information of any kind, such as code, settings, data, etc. In one specific embodiment, a computer program on the memory 1706 and executable on the processor 1704, when run by the processor 1704, can execute instructions of the computer vision-based device inspection method described in any of the above embodiments. Without limitation, for example, the memory 1706 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the computer device 1702. In one scenario, when processor 1704 executes associated instructions stored in any memory or combination of memories, computer device 1702 can perform any operation of the associated instructions. Computer device 1702 also includes one or more drive mechanisms 1708 for interacting with any memory, such as a hard disk drive, an optical disk drive, etc.

[0217] Computer device 1702 may also include an input / output interface 1710 (I / O) for receiving various inputs (via input device 1712) and providing various outputs (via output device 1714). A specific output mechanism may include a presentation device 1716 and an associated graphical user interface 1718 (GUI). In other embodiments, the input / output interface 1710 (I / O), input device 1712, and output device 1714 may be omitted, and the device may function solely as a computer device within a network. Computer device 1702 may also include one or more network interfaces 1720 for exchanging data with other devices via one or more communication links 1722. One or more communication buses 1724 couple the components described above together.

[0218] Communication link 1722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0219] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0220] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0221] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0222] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0223] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0224] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0225] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0227] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0228] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0229] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0230] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A computer vision-based equipment inspection method, characterized in that, include: Acquire the target image of the target device; The target image is an image of the indicator light portion of the target device; The target image is subjected to common corner detection to obtain a set of common corner coordinates in the target image; a black and white checkerboard is set at each of the four corners of the target image; the set of common corner coordinates corresponds one-to-one with the black and white checkerboard, and each set of common corner coordinates is a set of coordinates of multiple common corners under the corresponding black and white checkerboard. Confirm whether the target image meets the preset conditions based on the set of common corner coordinates; When the target image meets the preset conditions, the target image is subjected to perspective transformation according to the pre-constructed perspective transformation matrix to obtain the corrected target image; Identify regions of interest in the corrected target image; the regions of interest contain indicators for indicating the operating status of the target device by means of color; The color of the region of interest is matched with a preset color based on similarity, and the inspection result of the target device is determined based on the matching result. The step of detecting common corner points in the target image includes: The target image is divided into four sub-images according to the four-corner equal division rule; the four sub-images include the top left sub-image, the top right sub-image, the bottom left sub-image, and the bottom right sub-image. Using a designated subgraph among the four subgraphs as a reference, the remaining subgraphs in the four subgraphs are flipped so that each flipped subgraph has the same black and white checkerboard layout as the reference. Based on the black and white grid corner detection method, common corner points are detected for the specified sub-image and each of the flipped sub-images respectively, so as to obtain the first common corner point coordinate set of the specified sub-image and each of the flipped sub-images. Each subgraph after being flipped is flipped and reset, and the second common corner point coordinate set of each subgraph after being flipped and reset is determined based on the first common corner point coordinate set of each subgraph after being flipped.

2. The equipment inspection method based on computer vision as described in claim 1, characterized in that, The designated subgraph includes the top-left subgraph.

3. The equipment inspection method based on computer vision as described in claim 1, characterized in that, The common corner detection logic of the black-and-white grid corner detection method includes: The target sub-image is binarized to form the first sub-image; The first sub-image is subjected to image dilation processing to separate the connections between each black square in the first sub-image, forming the second sub-image; Extract the outlines of all black squares in the second sub-image to form the third sub-image; For each diagonally adjacent square grid contour in the third sub-image, the coordinates of the midpoint of the line connecting two points at a specified diagonal of the square grid contour are taken as the coordinates of the common corner point, forming the first common corner point coordinate set of the target sub-image.

4. The equipment inspection method based on computer vision as described in claim 3, characterized in that, The common corner detection logic of the black-and-white grid corner detection method also includes: Interference square contours in the extracted contours are removed by removing them by perimeter or area, so that the third sub-image does not contain the interference square contours.

5. The equipment inspection method based on computer vision as described in claim 3, characterized in that, The designated diagonal includes the farthest diagonal of the square grid outline.

6. The equipment inspection method based on computer vision as described in claim 3, characterized in that, The step of confirming whether the target image meets the preset conditions based on the common corner coordinate set includes: For each sub-image of the target image, all grid contours corresponding to the sub-image are traversed, and isolated grid contours are removed according to the set of common corner coordinates of the sub-image; the isolated grid contours are grid contours that do not contain common corners. Based on the set of common corner point coordinates corresponding to the subgraph and its correspondence with all the remaining squares after removing the isolated square outlines of the subgraph, determine the overall structural layout of all the remaining squares after removing the isolated square outlines of the subgraph. Determine whether the overall structural layout matches the preset structural layout; If the overall structural layout matches the preset structural layout, then it is confirmed that the sub-graph contains the expected black and white checkerboard. When all sub-images of the target image contain the expected black and white checkerboard, the target image is confirmed to meet the preset conditions.

7. The equipment inspection method based on computer vision as described in claim 6, characterized in that, The step of performing perspective transformation on the target image according to the pre-constructed perspective transformation matrix includes: The target image is then subjected to perspective transformation according to the following formula: Determine the outer contour of the black and white checkerboard in the target image after perspective transformation; Cropping out the edge portions of the target image that are outside the outer contour after perspective transformation; in, The original coordinates of the cropped target image. The coordinates of the cropped target image after perspective transformation. This is the perspective transformation matrix. ~ These are elements in the perspective transformation matrix.

8. The equipment inspection method based on computer vision as described in claim 1, characterized in that, Matching the color of the region of interest with a preset color based on similarity includes: Randomly select a specified number of points from the region of interest; Calculate the color similarity between the color of each selected point and the preset color to form a color similarity set; Determine the percentage of colors in the color similarity set that reach the color similarity threshold; The color similarity percentage is compared with the percentage threshold, and the matching result is determined based on the comparison result.

9. The equipment inspection method based on computer vision as described in claim 8, characterized in that, The step of determining the inspection results of the target equipment based on the matching results includes: When the color similarity ratio reaches the ratio threshold, it is confirmed that the target device is in a normal state; When the color similarity ratio does not reach the ratio threshold, the target device is confirmed to be in an abnormal state.

10. The equipment inspection method based on computer vision as described in claim 1, characterized in that, The preset color is determined in advance through the following method: Pre-acquire images of the target device and perform image correction on those images; A region of interest is selected from the image-corrected target image using a randomly chosen selection box; the region of interest contains an image of an indicator. Cluster all colors within the region of interest based on a pre-defined clustering method; Calculate the number of points contained in each color category within the region of interest; The number of points contained in each color category in the region of interest is sorted, and the primary and secondary colors in the region of interest are determined based on the sorting results. When the main color in the region of interest is the same as the color corresponding to the indicator in the specified state, the main color is used as the preset color, and the selection box is used as the selection box for identifying the region of interest during subsequent inspections. When the secondary color in the region of interest is the same as the color corresponding to the indicator in a specified state, the corresponding secondary color is used as the preset color, and the selection box is used as the selection box for identifying the region of interest during subsequent inspections.

11. A computer vision-based equipment inspection device, characterized in that, include: The image acquisition module is used to acquire target images from the target device. The target image is an image of the indicator light portion of the target device; A corner detection module is used to detect common corners in the target image to obtain a set of coordinates for common corners in the target image. A black and white checkerboard is provided at each of the four corners of the target image. The set of coordinates for common corners corresponds one-to-one with the checkerboard, and each set of coordinates for common corners is a set of coordinates for multiple common corners under the corresponding checkerboard. The detection of common corners in the target image includes: dividing the target image into four sub-images according to a four-corner equal division rule; the four sub-images include a top-left sub-image, a top-right sub-image, a bottom-left sub-image, and a bottom-right sub-image. Using a designated sub-image among the four sub-images as a reference, the remaining sub-images in the four sub-images are flipped so that each flipped sub-image has the same black and white checkerboard layout as the reference. Based on the black and white checkerboard corner detection method, common corner points are detected for the designated sub-image and each flipped sub-image respectively, so as to obtain the first common corner point coordinate set of the designated sub-image and each flipped sub-image. Each flipped sub-image is flipped and reset, and the second common corner point coordinate set of each flipped and reset sub-image is determined according to the first common corner point coordinate set of each flipped sub-image. The condition judgment module is used to determine whether the target image meets preset conditions based on the common corner point coordinate set; A perspective transformation module is used to perform perspective transformation on the target image according to a pre-constructed perspective transformation matrix when the target image meets the preset conditions, so as to obtain a corrected target image. A region recognition module is used to identify regions of interest in the corrected target image; the regions of interest contain indicators for indicating the operating status of the target device by means of color; The similarity matching module is used to match the color of the region of interest with a preset color based on similarity, and determine the inspection result of the target device based on the matching result.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-10.

13. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs instructions according to any one of claims 1-10.

Images