Camera field of view correction method, electronic device and storage medium

Abstract

The embodiment of the present invention relates to a camera field of view correction method, electronic device and storage medium. The embodiment of the present invention uses a camera to shoot a calibration plate image and preprocesses it to obtain a new calibration plate image, screens out valid dots of an original quasi-rectangular dot matrix in the new calibration plate image, obtains dot information of the valid dots, establishes a dot matrix coordinate system based on the identified dot information, rotates the original quasi-rectangular dot matrix positively and finds the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, obtains a maximum inscribed quasi-rectangular dot matrix based on the dot matrix coordinate system, generates an ideal rectangular dot matrix according to the width and height of the maximum inscribed quasi-rectangular dot matrix and the width and height of the calibration plate image, calculates a homography matrix based on the original quasi-rectangular dot matrix and the ideal rectangular dot matrix, and corrects the measured photos according to the homography matrix to correct the camera field of view. In this way, the embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of camera field of view correction, and in particular to a camera field of view correction method, electronic equipment and storage medium. Background Art

[0002] Automated Optical Inspection (AOI) is a device that uses optics, image processing, and computer vision technologies to inspect and analyze the surface, components, and structure of products. Optical machines and cameras are the optical machines that make up AOI equipment. Most current AOI equipment uses a single top camera to inspect the objects to be inspected on the bottom track. However, there are some scenarios where the top camera alone cannot meet the inspection requirements, such as requiring the side wall surface of the object to be vertical or tilted inward, and obstacles above the object. In these cases, another tilted camera is needed to shoot from the side to obtain the information of interest.

[0003] Since the imaging surface of a conventional lens (such as a perspective lens or a telecentric lens) is parallel to the camera's photosensitive chip, and considering that the side camera is arranged at an angle, its optical axis is not perpendicular to the surface of the object. This results in the side camera using a conventional lens only being able to focus on a line on the object. The area outside this line will be out of focus, which will greatly affect the detection effect.

[0004] In order to solve this problem, the industry generally adds a Schaum lens (scheimpflug) to the camera based on the conventional lens. The characteristic of the Schaum lens is that it can change the direction of the optical axis of the lens, so that the optical axis of the camera arranged diagonally is perpendicular to the surface of the object, so that the entire picture is in focus. But this brings a new problem. The pictures taken with the camera with a Schaum lens are distorted. More precisely, because it is shot sideways, the camera field of view (FOV) that was originally a rectangle will now be a trapezoid in the picture. Taking into account the error of mechanical processing, the original rectangular camera field of view may be distorted into a quadrilateral of arbitrary shape, so the camera field of view of this arbitrary quadrilateral needs to be corrected back to a rectangular camera field of view. Summary of the invention

[0005] In view of this, an object of an embodiment of the present invention is to provide a camera field of view correction method, electronic device and storage medium. The embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view, so as to accurately restore the surface information of the product structure.

[0006] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a camera field of view correction method, the method comprising:

[0008] The calibration plate image taken by the camera is preprocessed to obtain a new calibration plate image, wherein the new calibration plate image contains an original rectangular dot matrix, and the distribution of the original rectangular dot matrix corresponds to the distribution of the calibration blocks of the calibration plate;

[0009] Identify and filter out valid dots of the original rectangular dot matrix, and obtain dot information of the valid dots, including the center of mass of the dots;

[0010] Based on the identified dot information, a dot matrix coordinate system is established, and the original rectangular dot matrix is ​​rotated positively and the coordinates of each valid dot in the dot matrix coordinate system are found according to the dot information. The dot matrix coordinate system includes an X-axis and a Y-axis.

[0011] Based on the lattice coordinate system, obtain the centroids of the four vertices of the largest inscribed rectangular lattice, and find the largest inscribed rectangular lattice among all valid circular points based on the centroids of the four vertices;

[0012] Generate an ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image;

[0013] Based on the original rectangular lattice and the ideal rectangular lattice, the homography matrix is ​​calculated;

[0014] The measured photos are rectified according to the homography matrix to correct the camera field of view.

[0015] In some embodiments, the identifying and screening out valid dots of the original rectangular dot matrix and obtaining dot information of the valid dots includes:

[0016] Traverse all connected domains to perform connected domain analysis on the new calibration board map, and obtain the connected domains and connected domain information in the new calibration board map. The connected domain information includes the total number of connected domains, centroid coordinates and area, and the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain;

[0017] Based on the coordinates of the upper left corner, width and height of the circumscribed rectangle of each connected domain, incomplete connected domains are filtered out, where the incomplete connected domains are invalid dots;

[0018] Obtain the median area of ​​each connected domain according to its area, and set the area difference threshold based on the median;

[0019] The connected domains are filtered based on the area difference threshold to obtain qualified connected domains, and the connected domain information of the qualified connected domains is the dot information of the valid dots.

[0020] In some embodiments, the step of establishing a dot matrix coordinate system based on the identified dot information includes:

[0021] Calculate the real resolution of the camera based on the dot information and the actual size of the calibration block on the calibration plate. The real resolution includes the X-axis resolution and the Y-axis resolution.

[0022] A dot matrix coordinate system is established according to the X-axis resolution and the Y-axis resolution. The X-axis resolution corresponds to the X-axis of the dot matrix coordinate system, and the Y-axis resolution corresponds to the Y-axis of the dot matrix coordinate system.

[0023] In some embodiments, the dot information includes the actual dot distance of the dots, and the steps of rotating the original quasi-rectangular dot matrix and finding the coordinates of each valid dot in the dot matrix coordinate system according to the dot information include:

[0024] Calculate the rotation angle of the original rectangular dot matrix, and rotate the original rectangular dot matrix to the required angle according to the rotation angle to obtain a positive dot matrix, where the positive dot matrix includes an X-axis and a Y-axis;

[0025] Based on the actual dot distance and Y-axis resolution of each valid dot on the calibration plate, the centroid of the valid dots is traversed to identify the relative position of the valid dots on the Y-axis, and the relative position of the valid dots in the same row on the X-axis is identified based on the X-axis resolution to find the coordinates of each valid dot in the dot matrix coordinate system.

[0026] In some embodiments, the dot information includes the actual dot distance of the dots, and obtaining the centroid of four vertices of the largest inscribed rectangular dot matrix based on the dot matrix coordinate system includes:

[0027] Identify the leftmost and rightmost centroids of each row of all the centroids of valid dots;

[0028] Calculate the X-axis dot distance based on the X-axis resolution and the actual dot distance. The X-axis resolution and the X-axis dot distance are both in pixels.

[0029] The rotation direction of the original rectangular lattice is obtained, and according to different rotation directions, the maximum inscribed rectangular lattice is traversed to calculate the centroids of the four vertices of the maximum inscribed rectangular lattice. The centroids of the four vertices include the centroid of the upper left corner, the centroid of the upper right corner, the centroid of the lower left corner, and the centroid of the lower right corner.

[0030] In some embodiments, the method of finding the largest inscribed rectangular-like point matrix among all valid circular points based on the centroid of four vertices includes:

[0031] According to the coordinates of the upper left corner centroid and the upper right corner centroid, find the centroid of all valid circles in the first row of the largest inscribed rectangular lattice;

[0032] Starting from each centroid in the first row, set the jth row as the anchor point, search for the centroid closest to the anchor point in the j+1th row, and determine whether the centroid closest to the anchor point is in the same row as the lower left corner centroid or the lower right corner centroid;

[0033] When they are not in the same row, the centroid is set as the new anchor point;

[0034] When they are in the same row, the search ends and the current anchor point is recorded. At this time, all the centroids located in the maximum inscribed class rectangle have been found. All the found centroids are recorded as class rectangle centroids, and the dot matrix formed by the valid circular points where the class rectangle centroids are located is recorded as the maximum inscribed class rectangle dot matrix.

[0035] In some embodiments, generating an ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image includes:

[0036] Get the width and height of the calibration plate image;

[0037] Determine the width and height of the ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular-like dot matrix;

[0038] According to the width and height of the largest inscribed rectangular lattice and the width and height of the ideal rectangular lattice, coordinate system conversion is performed to obtain all coordinates of the ideal rectangular lattice;

[0039] An ideal rectangular lattice is established according to all the coordinates of the obtained ideal rectangular lattice.

[0040] In some embodiments, the step of calculating the homography matrix based on the original rectangular lattice and the ideal rectangular lattice includes:

[0041] Obtain the rotation angle and rotation center of the original rectangular lattice, and construct a rotation matrix according to the rotation angle, and rotate the rotated lattice based on the rotation matrix to rotate the maximum inscribed rectangular lattice back to the original angle;

[0042] The homography matrix is ​​calculated based on the ideal rectangular lattice and the rotated maximum inscribed rectangular lattice.

[0043] In a second aspect, an embodiment of the present invention provides an electronic device, including:

[0044] A processor and a memory communicatively connected to the processor;

[0045] The memory stores computer program instructions executable by the processor, and when the computer program instructions are called by the processor, the processor executes any one of the camera field of view correction methods proposed in the first aspect.

[0046] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, on which is stored computer program instructions executable by a processor, and when the computer program instructions are called by the processor, the processor executes any one of the camera field of view correction methods proposed in the first aspect.

[0047] The embodiments of the present invention have the following beneficial effects: Different from the prior art, the camera field of view correction method provided by the embodiments of the present invention includes: preprocessing the calibration plate image taken by the camera to obtain a new calibration plate image, the new calibration plate image includes an original rectangular dot matrix, the distribution of the original rectangular dot matrix corresponds to the distribution of the calibration block of the calibration plate, identifying and screening the valid dots of the original rectangular dot matrix, and obtaining the dot information of the valid dots, the dot information includes the center of mass of the dots, and based on the identified dot information, rotating the original rectangular dot matrix to establish the dot matrix coordinates. The dot coordinate system is constructed, and the coordinates of each valid dot in the dot coordinate system are found according to the dot information. The dot coordinate system includes an X-axis and a Y-axis. Based on the dot coordinate system, the centroids of the four vertices of the largest inscribed rectangular dot matrix are obtained, and the largest inscribed rectangular dot matrix is ​​found among all valid dots based on the centroids of the four vertices. An ideal rectangular dot matrix is ​​generated according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image. The homography matrix is ​​calculated based on the original rectangular dot matrix and the ideal rectangular dot matrix. The measured photos are corrected according to the homography matrix to correct the camera field of view.

[0048] The embodiment of the present invention uses a camera to take a calibration plate image, pre-processes the calibration plate image to obtain a new calibration plate image, screens out valid dots of the original rectangular dot matrix in the new calibration plate image, obtains dot information of the valid dots, establishes a dot matrix coordinate system based on the identified dot information, rotates the original rectangular dot matrix positively and finds the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, obtains the maximum inscribed rectangular dot matrix based on the dot matrix coordinate system, generates an ideal rectangular dot matrix according to the width and height of the maximum inscribed rectangular dot matrix and the width and height of the calibration plate image, calculates a homography matrix based on the original rectangular dot matrix and the ideal rectangular dot matrix, and corrects the measured photos according to the homography matrix to correct the camera field of view. In this way, the embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view, so that the surface information of the product structure can be accurately restored and the accuracy of product structure defect detection can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following is a brief introduction to the drawings required for the description of the prior art or the embodiments. Obviously, the drawings described below only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0050] Figure 1 is a schematic diagram of an application scenario of a camera field of view correction method provided by some embodiments of the present invention;

[0051] Figure 2 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present invention;

[0052] Figure 3 is a flowchart of a camera field of view correction method provided by some embodiments of the present invention;

[0053] Figure 4 is a new calibration plate image obtained by preprocessing the calibration plate image taken by the camera (without camera field of view correction) in some embodiments of the present invention, wherein the camera field of view is upward along the horizontal axis of the camera;

[0054] Figure 5 is a new calibration plate image obtained after preprocessing the calibration plate image taken by the camera (without camera field of view correction) in some embodiments of the present invention, wherein the camera field of view is tilted upward along the camera horizontal axis, turned clockwise along the camera vertical axis, and swung to the left along the camera vertical axis;

[0055] Figure 6 is a new calibration plate image obtained after preprocessing the calibration plate image taken by the camera (without camera field of view correction) in some embodiments of the present invention, wherein the camera field of view is tilted upward along the camera horizontal axis, turned counterclockwise along the camera vertical axis, and swung right along the camera vertical axis;

[0056] Figure 7 is a new calibration plate image obtained by preprocessing the calibration plate image taken by the camera and correcting the camera field of view in some embodiments of the present invention, wherein the camera field of view is a positive field of view without deflection;

[0057] Figure 8 yes Figure 3 A schematic diagram of a sub-process of step S200 in the camera field of view calibration method shown in the embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose and advantages of the embodiments of the present invention easier to understand, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The detailed description of the embodiments of the present invention in the drawings below does not limit the scope of protection claimed by the present invention, but only represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] It should be noted that, if no conflict is constituted, the various technical features involved in the embodiments of the present invention described below can be combined with each other, and are all within the scope of protection of the present invention. In addition, although the functional modules are divided in the device or structural schematic diagram, and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device, or in a different order than in the flow chart. In addition, the "first", "second", "third" and other similar expressions used herein do not limit the data and execution order, but are only for the purpose of convenience of explanation and to distinguish between the same or similar items with basically the same functions and effects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features.

[0060] Unless otherwise defined, the technical terms and scientific terms used in this specification have the same meanings as those generally understood by those of ordinary skill in the art of the present invention. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. It should be understood that the term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0061] In the semiconductor manufacturing industry, automated optical inspection equipment (AOI) is usually needed to detect and analyze product appearance, quality, and performance. Optical machines and cameras are the optical components that make up AOI equipment. Most current AOI equipment uses a top single camera configuration to inspect the objects to be inspected on the bottom track. However, there are some scenarios where the top camera alone cannot meet the inspection requirements, such as requiring the object to have a vertical or inward-inclined side wall surface, and obstacles above the object. In these cases, another tilted camera is needed to shoot from the side to obtain the information of interest.

[0062] Since the imaging surface of a conventional lens (such as a perspective lens or a telecentric lens) is parallel to the camera's photosensitive chip, and considering that the side camera is arranged at an angle, its optical axis is not perpendicular to the surface of the object. This results in the side camera using a conventional lens only being able to focus on a line on the object. The area outside this line will be out of focus, which will greatly affect the detection effect.

[0063] In order to solve this problem, the industry generally adds a Schaum lens (scheimpflug) to the camera based on the conventional lens. The characteristic of the Schaum lens is that it can change the direction of the optical axis of the lens, so that the optical axis of the camera arranged diagonally is perpendicular to the surface of the object, so that the entire picture is in focus. But this brings a new problem. The pictures taken with the camera with a Schaum lens are distorted. More precisely, because it is shot sideways, the camera field of view (FOV) that was originally a rectangle will now be a trapezoid in the picture. Taking into account the error of mechanical processing, the original rectangular camera field of view may be distorted into a quadrilateral of arbitrary shape, so the camera field of view of this arbitrary quadrilateral needs to be corrected back to a rectangular camera field of view.

[0064] In view of this, an embodiment of the present invention provides a camera field of view correction method, which uses a camera to take a calibration plate image, pre-processes the calibration plate image to obtain a new calibration plate image, screens out valid dots of the original quasi-rectangular dot matrix in the new calibration plate image, obtains dot information of the valid dots, establishes a dot matrix coordinate system based on the identified dot information, rotates the original quasi-rectangular dot matrix to positive rotation and finds the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, obtains the maximum inscribed quasi-rectangular dot matrix based on the dot matrix coordinate system, generates an ideal rectangular dot matrix according to the width and height of the maximum inscribed quasi-rectangular dot matrix and the width and height of the calibration plate image, calculates a homography matrix based on the original quasi-rectangular dot matrix and the ideal rectangular dot matrix, and corrects the measured photos according to the homography matrix to correct the camera field of view. In this way, the embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view, so that the surface information of the product structure can be accurately restored, and the accuracy of product structure defect detection can be improved.

[0065] See also Figure 1 , Figure 1 A schematic diagram schematically illustrates an application scenario of a camera field of view correction method provided by some embodiments of the present invention.

[0066] like Figure 1 As shown, the application scenario includes an electronic device 100. The electronic device 100 may be a desktop computer, a laptop computer, a tablet computer, or other suitable type of device with certain computing and control capabilities. Figure 1 In the application scenario shown, the electronic device 100 is a laptop computer, but it does not impose any restrictions on the structure, type, quantity, etc. of the electronic devices in other application scenarios. For example, in some other application scenarios or embodiments, the electronic device may also be a desktop computer, a tablet computer, or other suitable types of devices. The electronic devices in other application scenarios or embodiments may also be larger than 100. Figure 1 The laptop computer shown in the embodiment may include more or fewer components, or may be different from the Figure 1The laptop computers shown in the embodiments have different configurations.

[0067] It is understood that in order to complete the camera field of view correction, the electronic device 100 also needs to work in coordination with other software and hardware, including a camera or image acquisition device, a thermal imaging camera, image processing software, a display device, and a storage device, etc. Those skilled in the art can add, delete, or change the required software and hardware according to actual needs, and work with the electronic device 100 to complete the camera field of view correction. It should be understood that these collaborative software and hardware can be configured in the electronic device 100 according to actual needs, or exist independently of the electronic device 100.

[0068] Specifically, the electronic device 100, as a device providing computing and control capabilities, obtains the calibration plate image taken by the camera, and preprocesses the calibration plate image taken by the camera into a new calibration plate image, wherein the new calibration plate image contains the original rectangular lattice, and the distribution of the original rectangular lattice corresponds to the distribution of the calibration block of the calibration plate. Identify and filter out the valid dots of the original rectangular lattice, and obtain the dot information of the valid dots, wherein the dot information includes the center of mass of the dots. Based on the identified dot information, establish a dot matrix coordinate system, rotate the original rectangular lattice positively, and find the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, and the dot matrix coordinate system includes the X axis and the Y axis. Based on the dot matrix coordinate system, obtain the center of mass of the four vertices of the maximum inscribed rectangular lattice, and find the maximum inscribed rectangular lattice among all valid dots based on the center of mass of the four vertices. Generate an ideal rectangular lattice according to the width and height of the maximum inscribed rectangular lattice and the width and height of the calibration plate image. Calculate the homography matrix based on the original rectangular lattice and the ideal rectangular lattice. The measured photos are corrected according to the homography matrix to correct the camera field of view and obtain measured photos with accurate camera field of view. According to the measured photos corrected by the camera field of view, the surface information of the product structure is accurately restored to improve the accuracy of product structure defect detection.

[0069] To facilitate understanding of the camera field of view correction method provided by the embodiment of the present invention, the electronic device provided by the embodiment of the present invention is first introduced in detail.

[0070] See also Figure 2 , Figure 2 The structural diagram of the electronic device 100 provided in some embodiments of the present invention is schematically shown.

[0071] For example, Figure 2 As shown, the electronic device 100 includes at least one processor 110 and a memory 120 that are communicatively connected. Figure 2In the example, a bus system is connected to a processor. The components in the electronic device 100 are coupled together through the bus system 130, and the bus system 130 is used to realize the connection and communication between the components. It is easy to understand that the bus system 130 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clarity and brevity, Figure 2 In the figure, various buses are labeled as bus system 130. It can be understood that Figure 2 The structure shown in the embodiment is only for illustration and does not impose any limitation on the structure of the electronic device. Figure 2 The structure shown in FIG. 1 may have more or fewer components, or may have the same Figure 2 Different configurations of the structure shown.

[0072] Specifically, the processor 110 is used to provide computing and control capabilities to control the electronic device 100 to perform corresponding tasks, for example, to control the above electronic device 100 to perform any camera field of view correction method provided in the embodiment of the present invention, or to perform any step in any possible implementation of any camera field of view correction method provided in the embodiment of the present invention. Those skilled in the art can understand that the processor 110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0073] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, instructions and modules, such as programs, instructions and modules corresponding to the camera field of view correction method in the embodiment of the present invention. In some embodiments, the memory 120 may include a program storage area and a data storage area, the program storage area may store an operating system, an application required for at least one function, and the data storage area may store data created according to the use of the processor 110, etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the non-transitory software programs, instructions and modules stored in the memory 120 to implement any camera field of view correction method provided in the embodiment of the present invention, or execute any step in any possible implementation of any camera field of view correction method provided in the embodiment of the present invention. In some embodiments, the memory 120 may include a high-speed random access memory and may also include a non-transitory memory. For example, at least one disk storage device, a flash memory device or other non-transitory solid-state storage device. In some embodiments, the memory 120 may also include a memory remotely arranged relative to the processor 110, and these remotely arranged memories may be connected to the processor 110 through a communication network. It can be understood that examples of the above-mentioned communication network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof.

[0074] It can be understood from the above that the implementation subject of any camera field of view correction method provided by the embodiments of the present invention can be any suitable type of electronic device with certain computing and control capabilities, for example, it can be implemented by the above electronic device 100. In some feasible implementations, any camera field of view correction method provided by the embodiments of the present invention can be implemented by a processor calling a computer program instruction stored in a memory.

[0075] The camera field of view correction method provided by the embodiment of the present invention will be described in detail below in conjunction with exemplary applications and implementations of the electronic device provided by the embodiment of the present invention.

[0076] See also Figure 3 , Figure 3 The flowchart of the camera field of view correction method provided by some embodiments of the present invention is schematically shown.

[0077] Those skilled in the art will appreciate that the camera field of view correction method provided in the embodiment of the present invention can be applied to the above electronic device (eg, electronic device 100). Specifically, the camera field of view correction method is executed by one or at least two processors of the electronic device.

[0078] like Figure 3 As shown, the camera field of view correction method includes but is not limited to the following steps S100-S700:

[0079] S100: pre-processing the calibration plate image taken by the camera to obtain a new calibration plate image, wherein the new calibration plate image includes an original rectangular-like dot matrix, and the distribution of the original rectangular-like dot matrix corresponds to the distribution of the calibration blocks of the calibration plate.

[0080] Among them, a plurality of calibration blocks are arranged on the calibration plate, and the shape of the calibration blocks can be circular, rectangular or square, etc., and the embodiment of the present invention sets the shape of the calibration blocks to be circular. The distribution shape of the plurality of calibration blocks can be regular shapes such as rectangular, square or circular, or irregular shapes such as rhombus, ellipse or triangle, and the embodiment of the present invention sets the distribution shape of the plurality of calibration blocks to be rectangular, which is convenient for subsequent calculations.

[0081] In some embodiments, a camera is used to photograph the calibration plate to obtain a calibration plate image, where the calibration plate image includes a circular or approximately circular shadow area formed by a number of calibration blocks.

[0082] Specifically, obtain the calibration plate image taken from the camera, use a common image processing library (such as OpenCV, etc.) to remove the color information of the calibration plate image, that is, convert the number of channels of the calibration plate image from three channels (such as RGB) to a single channel, and obtain an initial grayscale image. Use a threshold segmentation algorithm to perform threshold segmentation on the initial grayscale image, and binarize the initial grayscale image, that is, convert the initial grayscale image into a binary image including only black and white colors, and obtain the final preprocessed new calibration plate image, which is in the form of a binary image. The preprocessed new calibration plate image includes an original rectangular dot matrix, which includes a number of dots, approximate dots or dots of other shapes, and the distribution of the original rectangular dot matrix corresponds to the distribution of the calibration block of the calibration plate. Please refer to Figure 4 , Figure 4 The new calibration plate image obtained by preprocessing the calibration plate image taken by the camera in some embodiments of the present invention is schematically shown. Figure 4 The several dots and other shaped dots in the new calibration plate shown in the figure constitute the original rectangular lattice. Figure 4 The new calibration plate shown is a new calibration plate without camera field of view correction. Figure 4 The camera field of view is only schematically shown as tilted upward along the camera horizontal axis. The camera field of view may also have other types of deflections, for example, see Figure 5 and Figure 6 , Figure 5 In the new calibration board shown in the figure, the camera's field of view is tilted upward along the camera's horizontal axis, turned clockwise along the camera's vertical axis, and swung to the left along the camera's vertical axis. Figure 6 In the new calibration board shown in the figure, the camera's field of view is simultaneously tilted upward along the camera's horizontal axis, flipped counterclockwise along the camera's longitudinal axis, and swung to the right along the camera's vertical axis.

[0083] It can be understood that the threshold segmentation algorithm includes but is not limited to: binary threshold segmentation algorithm, OTSU algorithm and TOZERO algorithm. In some embodiments, Gaussian blur is used to remove noise points in the binary image, smooth the binary image, and obtain a higher quality pre-processed new calibration plate image, so that the features in the binary image are more obvious, so as to reduce interference with feature extraction.

[0084] S200: Identify and filter out valid dots of the original rectangular dot matrix, and obtain dot information of the valid dots, where the dot information includes the centroid of the dots.

[0085] In some embodiments, a circle detection algorithm (such as a Hough transform algorithm) is used to detect the dots of the original rectangular dot matrix in the preprocessed new calibration plate image, and the detected dots are extracted. Among them, the Hough transform algorithm is an image processing algorithm that can be used to identify dots in an image and return the center coordinates and radius of the dots. In some embodiments, any other suitable algorithm can also be used to detect the dots in the original rectangular dot matrix, and the embodiments of the present invention do not limit this in any way.

[0086] In some embodiments, for each extracted dot, according to a preset rule, it is determined whether the dot is valid. If it is valid, the dot is determined to be a valid dot and is screened out as a valid dot. If it is invalid, the dot is determined to be an invalid dot and is removed or processed in other ways. The preset rules include: the radius of the dot is within a preset range, the center position of the dot conforms to the layout of the original rectangular dot matrix, etc. Of course, those skilled in the art can add, modify or delete the preset rules according to actual needs, and the embodiments of the present invention do not limit this in any way.

[0087] In some embodiments, after valid dots of the original rectangular dot matrix are identified and screened out from the extracted dots, dot information of the valid dots is acquired using a circle detection algorithm (e.g., a Hough transform algorithm, etc.), and the dot information includes: the center of mass of the dots and the radius of the dots, wherein the center of mass of the dots is the center coordinate of the dots, and the Hough transform algorithm gives the center coordinates and radius of each dot, so that the dot information of the valid dots can be acquired through the Hough transform algorithm.

[0088] S300: Based on the identified dot information, a dot matrix coordinate system is established, and the original quasi-rectangular dot matrix is ​​rotated to be positive and the coordinates of each valid dot in the dot matrix coordinate system are found out according to the dot information. The dot matrix coordinate system includes an X-axis and a Y-axis.

[0089] Exemplarily, among all the identified valid dots, at least two valid dots located in the same horizontal or vertical direction are found as the reference of the lattice coordinate system, the first vector between the two valid dots is calculated as the X-axis direction, the second vector perpendicular to the first vector is calculated as the Y-axis direction, the center of mass of any one of the at least two valid dots is taken as the origin, the X-axis is constructed in the X-axis direction, and the Y-axis is constructed in the Y-axis direction, thereby establishing the lattice coordinate system.

[0090] In some embodiments, after establishing the dot matrix coordinate system, the original rectangular dot matrix is ​​rotated to the positive direction, including the following steps: determining the current direction of the original rectangular dot matrix by geometric analysis (such as least squares method or principal component analysis (PCA)), calculating the rotation angle of the original rectangular dot matrix according to the current direction, and rotating the original rectangular dot matrix from the current direction to the positive direction according to the rotation angle to obtain the rotated original rectangular dot matrix. For example, the rotation angle of the original rectangular dot matrix is , use the rotation matrix formula to rotate the original rectangular lattice to the positive direction to obtain the original rectangular lattice with positive rotation, where the rotation matrix is:

[0091]

[0092] The rotation matrix formula is:

[0093]

[0094] in,( , ) are the coordinates of valid points in the original rectangular lattice, ( , ) are the coordinates of valid points in the original rectangular lattice with positive rotation.

[0095] In some embodiments, after the original rectangular lattice is rotated, the position coordinates of each valid circle point in the lattice coordinate system are determined according to the centroid of the valid circle points in the original rectangular lattice. Specifically, the first valid circle point in the original rectangular lattice that is rotated is used as the origin, and the coordinates of the remaining valid circle points in the lattice coordinate system are calculated. Assuming that the origin is located at (x0, y0), the coordinates of the remaining valid circle points (x, y) in the lattice coordinate system can be calculated by a simple translation transformation:

[0096]

[0097]

[0098] in,( , ) is the coordinate of the first valid circular point in the original rectangular lattice with positive rotation in the lattice coordinate system, ( , ) are the coordinates of the remaining valid circular points in the original rectangular lattice with positive rotation in the lattice coordinate system.

[0099] S400: Based on the lattice coordinate system, the centroids of four vertices of the largest inscribed rectangular lattice are obtained, and based on the centroids of the four vertices, the largest inscribed rectangular lattice is found among all valid circular points.

[0100] Specifically, according to the coordinates of the valid points in the original rectangular dot matrix in the dot matrix coordinate system, the leftmost and rightmost valid points in each row of the original rectangular dot matrix are screened out, and according to the distance between the leftmost and rightmost valid points in two adjacent rows, the leftmost valid point or the rightmost valid point is used as one of the four vertices of the maximum inscribed rectangular dot matrix, until the four vertices of the maximum inscribed rectangular dot matrix are found, and the central coordinates of the four vertices of the maximum inscribed rectangular dot matrix are used as the centroid of the four vertices of the maximum inscribed rectangular dot matrix.

[0101] In some embodiments, after obtaining the centroid of the four vertices of the maximum inscribed rectangular quasi-dot matrix, starting from the first row where the upper left corner vertex or the upper right corner vertex is located, all valid circular dots located in the same row as the upper left corner vertex or the upper right corner vertex are found according to the centroid of the four vertices of the maximum inscribed rectangular quasi-dot matrix, until the last row where the lower left corner vertex or the lower right corner vertex is located is found, and all valid circular dots located in the same row as the lower left corner vertex or the lower right corner vertex are found, which means that the valid circular dots in each row within the maximum inscribed rectangular quasi-dot matrix are found, and the valid circular dots in each row within the maximum inscribed rectangular quasi-dot matrix form the maximum inscribed rectangular quasi-dot matrix.

[0102] S500: Generate an ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image.

[0103] In an embodiment of the present invention, before using a camera to take a picture of the calibration plate, the size of the picture taken by the camera (including width and height) is set. In this way, after the camera takes the picture of the calibration plate, the width and height of the calibration plate picture can be known.

[0104] Specifically, after obtaining the calibration plate image by photographing with a camera, the size of the calibration plate image is read to obtain the width and height of the calibration plate image. According to the found maximum inscribed quasi-rectangular dot matrix, the width and height of the maximum inscribed quasi-rectangular dot matrix are calculated.

[0105] The ideal rectangular dot matrix is ​​a regular matrix layout, in which dots are distributed along regular rows and columns, and the distances between rows and columns are equal. According to the width and height of the maximum inscribed rectangular dot matrix and the width and height of the calibration plate image, the width and height of the ideal rectangular dot matrix are calculated, and the number of rows and columns of the ideal rectangular dot matrix is ​​determined to be the same as the number of rows and columns of the maximum inscribed rectangular dot matrix, and the coordinate system is converted to calculate the coordinate position of each dot. Based on the coordinate position of each dot, the ideal rectangular dot matrix is ​​generated.

[0106] S600: Calculate a homography matrix based on the original quasi-rectangular lattice and the ideal rectangular lattice.

[0107] Among them, the homography matrix It is a 3x3 matrix that can describe the perspective transformation between two planes. The homography matrix transforms a point on a plane from the original coordinate system to the target coordinate system.

[0108] Exemplarily, at least four groups of corresponding dots are obtained from the original quasi-rectangular dot matrix and the ideal rectangular dot matrix, and each group of dots includes a valid dot in the original quasi-rectangular dot matrix and a dot corresponding to the valid dot in the ideal rectangular dot matrix.

[0109] In some embodiments, the coordinates of at least 4 groups of corresponding dots are homogenized, and the homography matrix is ​​calculated using the cv2.findHomography() function of OpenCV. The coordinates of the at least 4 groups of corresponding dots after homogenization are input into the cv2.findHomography() function. The coordinates of the at least 4 groups of corresponding dots after homogenization are constructed into a matrix multiplication equation through the homography matrix, and the constructed matrix multiplication equation is calculated using the cv2.findHomography() function to obtain a 3x3 homography matrix. .

[0110] For example, suppose the homography matrix for:

[0111]

[0112] Assume that the coordinates of a set of corresponding points in the original rectangular lattice and the ideal rectangular lattice are (homogenized) respectively:

[0113] ,

[0114] The coordinates of the points corresponding to this group (homogenized) are input into the cv2.findHomography() function, and the homography matrix Construct a matrix multiplication equation, which is:

[0115]

[0116] From the above matrix multiplication equation, we can see that a matrix multiplication equation can only solve two unknowns (i.e. , , , , , , and where two of the unknowns are in There are 8 unknowns (i.e. , , , , , , and ), so it is necessary to use the coordinates of at least 4 groups of corresponding dots in the original rectangular lattice and the ideal rectangular lattice, and input the coordinates of each group of corresponding dots (homogenized) into the cv2.findHomography() function respectively, through the homography matrix Construct a matrix multiplication equation, where both sides of the matrix multiplication equation have the coordinates of at least four points (homogenized), and then solve the matrix multiplication equation to calculate the homography matrix All unknowns in , thus obtaining the homography matrix .

[0117] For example, in some embodiments, the matrix multiplication equation constructed using the coordinates (homogenized) of four groups of corresponding dots in the original quasi-rectangular lattice and the ideal rectangular lattice is:

[0118]

[0119] For example, the method in the cv2.findHomography() function is used to solve the above matrix multiplication equation and calculate the homography matrix All unknowns in , thus obtaining the homography matrix .

[0120] It is easy to understand that to calculate the homography matrix, it is necessary to use the coordinates of at least 4 groups of corresponding dots in the original quasi-rectangular dot matrix and the ideal rectangular dot matrix. Therefore, when setting the calibration blocks on the calibration plate, the number of calibration blocks should be at least 4, so that the original quasi-rectangular dot matrix contained in the new calibration plate image obtained by preprocessing includes at least 4 valid dots, and ensures that the generated ideal rectangular dot matrix includes at least 4 dots, so as to ensure that the homography matrix can be calculated based on the original quasi-rectangular dot matrix and the ideal rectangular dot matrix.

[0121] It can be understood that the more coordinates of the corresponding dots in the original rectangular lattice and the ideal rectangular lattice are used to construct the matrix multiplication equation, the more known numbers in the constructed matrix multiplication equation are far greater than the number of unknown numbers. Therefore, the matrix multiplication equation is an overdetermined equation. Algorithms for calculating overdetermined equations include but are not limited to: RANSAC algorithm, LmedS algorithm, and least squares method.

[0122] S700: Correcting the measured photo according to the homography matrix to correct the camera field of view.

[0123] Specifically, obtain and load the measured photos taken by the input camera, perform perspective transformation on the measured photos through the homography matrix, that is, use the cv2.warpPerspective() function to apply the homography matrix to the measured photos for perspective transformation, multiply each pixel in the measured photos by the homography matrix, obtain the coordinates of each pixel in the image after camera field of view correction, and fill the pixel value of each coordinate to the corresponding coordinate position in the image after camera field of view correction to complete the correction of the camera field of view.

[0124] See also Figure 7 , Figure 7 The diagram schematically shows a new calibration plate image obtained by preprocessing the calibration plate image taken by the camera and correcting the camera field of view according to the homography matrix in some embodiments of the present invention. Figure 7 The new calibration plate after preprocessing and camera field of view correction is shown in Figure 4 , Figure 5 or Figure 6 The figure corresponds to the new calibration plate image after preprocessing (without camera field of view correction).

[0125] It should be understood that by correcting the measured photos according to the homography matrix, the measured photos after the camera field of view is corrected can be obtained, that is, the camera field of view is changed from Figure 4 , Figure 5 and Figure 6 The deviated field of vision shown, or other types of deviated fields of vision, are corrected to normal field of vision.

[0126] In some embodiments, the blank areas (i.e., the coordinate positions where the pixel values ​​are not filled) in the image after the camera field of view correction use an interpolation algorithm to calculate the pixel values ​​of the corresponding coordinate positions, and fill the pixel values ​​to the corresponding coordinate positions. Those skilled in the art can understand that the interpolation algorithm includes but is not limited to: Inter Linear algorithm and Inter Neartest algorithm.

[0127] The embodiment of the present invention uses a camera to take a calibration plate image, pre-processes the calibration plate image to obtain a new calibration plate image, screens out valid dots of the original rectangular dot matrix in the new calibration plate image, obtains dot information of the valid dots, establishes a dot matrix coordinate system based on the identified dot information, rotates the original rectangular dot matrix positively and finds the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, obtains the maximum inscribed rectangular dot matrix based on the dot matrix coordinate system, generates an ideal rectangular dot matrix according to the width and height of the maximum inscribed rectangular dot matrix and the width and height of the calibration plate image, calculates a homography matrix based on the original rectangular dot matrix and the ideal rectangular dot matrix, and corrects the measured photos according to the homography matrix to correct the camera field of view. In this way, the embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view, so that the surface information of the product structure can be accurately restored and the accuracy of product structure defect detection can be improved.

[0128] See also Figure 8 , Figure 8 A sub-flow diagram of step S200 in the camera field of view correction method provided by some embodiments of the present invention is schematically shown.

[0129] In some embodiments, identifying and screening out valid dots of the original rectangular dot matrix, and obtaining dot information of the valid dots, specifically includes but is not limited to the following steps S210-S230:

[0130] S210: traverse all connected domains to perform connected domain analysis on the new calibration board map, and obtain connected domains and connected domain information in the new calibration board map. The connected domain information includes the total number of connected domains, centroid coordinates and area, and the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain.

[0131] Specifically, a connected domain analysis algorithm is used to perform a connected domain analysis on the preprocessed new calibration plate image, identify all connected domains in the preprocessed new calibration plate image, and traverse all identified connected domains to obtain connected domain information. The connected domain information includes the total number of connected domains, centroid coordinates and area, and the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain. It is understandable that the connected domain analysis algorithm includes but is not limited to: Spaghetti algorithm, SAUF algorithm and BBDT algorithm.

[0132] Exemplarily, in some embodiments, the area of ​​the connected domain is the total number of pixels in the connected domain, and the total number of pixels in the connected domain is calculated using the cv2.contourArea() function in the OpenCV library or other similar methods to obtain the area of ​​the connected domain.

[0133] Exemplarily, in some embodiments, the centroid coordinates of the connected domain are the geometric center of the connected domain, and the cv2.moments() function in the OpenCV library can be used to calculate the moments of the connected domain to obtain the centroid coordinates of the connected domain.

[0134] Exemplarily, in some embodiments, the bounding rectangle of the connected domain is represented by the coordinates of the upper left corner (x, y) and the width and height (w, h) of the rectangle, and the cv2.boundingRect() function of the OpenCV library is used to obtain the coordinates and size (including width and height) of the bounding rectangle of the connected domain to obtain the bounding rectangle of the connected domain.

[0135] S220: Filter out incomplete connected domains based on the upper left corner coordinates, width, and height of the circumscribed rectangle of each connected domain, wherein the incomplete connected domains are invalid dots.

[0136] Specifically, for each connected domain, the upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle of each connected domain are calculated according to the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain, and the upper left corner coordinates, upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle of each connected domain are respectively compared with the edge of the new calibration plate map. If any one of the upper left corner coordinates, upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle exceeds the edge of the new calibration plate map or is located on the edge of the new calibration plate map, then the connected domain is determined to be an incomplete connected domain, and the incomplete connected domain is an invalid dot, and the incomplete connected domain is filtered out.

[0137] It can be understood that the coordinates of the upper right corner of the circumscribed rectangle are the same as the ordinates of the coordinates of the upper left corner. Therefore, when calculating the coordinates of the upper right corner, the abscissa of the coordinates of the upper left corner is added to the width of the circumscribed rectangle to obtain the abscissa of the coordinates of the upper right corner, thereby obtaining the coordinates of the upper right corner. Similarly, the coordinates of the lower left corner of the circumscribed rectangle are the same as the abscissa of the coordinates of the upper left corner. Therefore, when calculating the coordinates of the lower left corner, the ordinate of the coordinates of the upper left corner is subtracted from the height of the circumscribed rectangle to obtain the ordinate of the coordinates of the lower left corner, thereby obtaining the coordinates of the lower left corner. Similarly, after calculating the coordinates of the upper right corner and / or the coordinates of the lower left corner, the coordinates of the lower right corner of the circumscribed rectangle are the same as the abscissa of the coordinates of the upper right corner. Therefore, when calculating the coordinates of the lower right corner, the ordinate of the coordinates of the upper right corner is subtracted from the height of the circumscribed rectangle to obtain the ordinate of the coordinates of the lower right corner, thereby obtaining the coordinates of the lower right corner. Or the lower right corner coordinate of the circumscribed rectangle is the same as the ordinate of the lower left corner coordinate, so when calculating the lower right corner coordinate, add the abscissa of the lower left corner coordinate to the width of the circumscribed rectangle to get the abscissa of the lower right corner coordinate, and thus get the lower right corner coordinate.

[0138] In some embodiments, the edge of the new calibration plate image is detected by using an edge detection algorithm to determine the edge of the new calibration plate image. It can be understood that the edge detection algorithm includes but is not limited to: Sobel algorithm, Canny algorithm, Prewitt algorithm and Laplacian algorithm.

[0139] S230: Obtaining a median of the area according to the area of ​​each connected domain, and setting an area difference threshold based on the median.

[0140] Specifically, according to the area of ​​each connected domain, the areas of all connected domains are arranged in ascending order to obtain an area arrangement sequence. The area value at the middle position is selected from the area arrangement sequence as the median of the area. If there are two area values ​​at the middle position, the median of the area is taken as the average of the two area values ​​at the middle position.

[0141] After obtaining the median of the area, an area difference threshold is set according to the median of the area. In some embodiments, the area difference threshold can be set according to a preset multiple of the median of the area to distinguish valid and invalid connected domains. For example, the area difference threshold is 1.2 times, 1.5 times, or other suitable multiples of the median of the area. Those skilled in the art can set the area difference threshold according to actual needs, and the embodiments of the present invention do not impose any limitation on this.

[0142] S240: filtering the connected domains based on the area difference threshold to obtain qualified connected domains, where the connected domain information of the qualified connected domains is the dot information of the valid dots.

[0143] Specifically, each connected domain is traversed, and the area of ​​each connected domain is subtracted from the median of the area to obtain an area difference value, and the area difference value is compared with the area difference threshold value. If the area difference value is less than or equal to the area difference threshold value, the connected domain is determined to be a qualified connected domain (i.e., a valid dot). If the area difference value is greater than the area difference threshold value, the connected domain is determined to be an unqualified connected domain (i.e., an invalid dot). The connected domain information of the qualified connected domain obtained by filtering is used as the dot information of the valid dot, thereby obtaining the dot information of the valid dot to filter the connected domain.

[0144] In some embodiments, a dot matrix coordinate system is established based on the identified dot information, including but not limited to the following steps S310-S320:

[0145] S310: Calculate the real resolution of the camera according to the dot information and the actual size of the calibration block on the calibration plate, where the real resolution includes the X-axis resolution and the Y-axis resolution.

[0146] In the embodiment of the present invention, the calibration block on the calibration plate has a known actual size in the actual physical space, and the actual size includes the actual diameter of the circular calibration block.

[0147] Specifically, the width (in pixels) of the circumscribed rectangle of each valid circle point is summed to obtain the total width of the circumscribed rectangle, and the total width of the circumscribed rectangle is divided by the total number of valid circles (i.e., the total number of connected domains) to obtain the diameter of the valid circle point in the X-axis direction (in pixels). The diameter of the valid circle point in the X-axis direction is divided by the actual diameter of the circular calibration block to obtain the X-axis resolution of the camera.

[0148] Sum the heights (in pixels) of the circumscribed rectangles of each valid dot to get the total height of the circumscribed rectangles, divide the total height of the circumscribed rectangles by the total number of valid dots (i.e., the total number of connected domains), and get the diameter (in pixels) of the valid dots in the Y-axis direction. Divide the diameter of the valid dots in the Y-axis direction by the actual diameter of the circular calibration block to get the Y-axis resolution of the camera. Combine the X-axis resolution and the Y-axis resolution of the camera to get the actual resolution of the camera.

[0149] S320: establishing a dot matrix coordinate system according to the X-axis resolution and the Y-axis resolution, wherein the X-axis resolution corresponds to the X-axis of the dot matrix coordinate system, and the Y-axis resolution corresponds to the Y-axis of the dot matrix coordinate system.

[0150] Specifically, an actual dot matrix coordinate system is constructed using the X-axis resolution and the Y-axis resolution, and the position of each valid dot is mapped to the dot matrix coordinate system, that is, the pixel coordinates of each valid dot are converted into actual physical coordinates in the dot matrix coordinate system through the real resolution. The coordinate system of the dot matrix coordinate system is set as follows: the X-axis corresponds to the horizontal direction of the actual physical space, the Y-axis corresponds to the vertical direction of the actual physical space, the X-axis resolution corresponds to the X-axis of the dot matrix coordinate system, the X-axis resolution represents the horizontal distance corresponding to each pixel in the image in the actual physical space (for example, in millimeters / pixels), the Y-axis resolution corresponds to the Y-axis of the dot matrix coordinate system, and the Y-axis resolution represents the vertical distance corresponding to each pixel in the image in the actual physical space (for example, in millimeters / pixels). In the dot matrix coordinate system, the X-axis and the Y-axis represent the horizontal coordinate and the vertical coordinate in the actual physical space, respectively, and each pixel in the image has a corresponding actual coordinate in the dot matrix coordinate system.

[0151] In some embodiments, a dot matrix coordinate system is established with the centroid of the valid dot in the upper left corner as the coordinate origin, the horizontal axis (horizontal direction) as the X-axis, and the vertical axis (vertical direction) as the Y-axis, wherein the dot matrix coordinate system includes the X-axis and the Y-axis, and the unit of the dot matrix coordinate system is the unit distance, that is, the interval between the horizontal coordinate or the vertical coordinate of two adjacent valid dots in the dot matrix coordinate system is the unit distance 1.

[0152] In some embodiments, the original rectangular dot matrix is ​​rotated positively and the coordinates of each valid dot in the dot matrix coordinate system are found according to the dot information, specifically including but not limited to the following steps S330-S340:

[0153] S330: Calculate the rotation angle of the original rectangular lattice, and rotate the original rectangular lattice to a required angle according to the rotation angle to obtain a positive lattice, where the positive lattice includes an X-axis and a Y-axis.

[0154] Specifically, according to the width and height of the calibration plate image, the coordinates (x, y) of the center point of the calibration plate image are calculated, wherein the horizontal coordinate x of the center point is equal to the width W / 2 of the calibration plate image, and the horizontal coordinate y of the center point is equal to the width H / 2 of the calibration plate image. Traverse the centroids of all valid points, and calculate the Euclidean distance between the centroid coordinates of each valid point and the coordinates of the center point of the calibration plate image, and determine the centroid of the valid point closest to the center point of the calibration plate image as the center centroid. Then, take the center centroid as the target centroid, and find the two neighboring centroids closest to the center centroid, wherein the two neighboring centroids are located on the left and right of the target centroid, respectively. According to the target centroid and the two neighboring centroids, use a linear fitting algorithm to fit the target straight line, and calculate the rotation direction and rotation angle of the original rectangular dot matrix based on the target straight line.

[0155] It can be understood that the linear fitting algorithm includes but is not limited to: a linear regression algorithm based on taxis distance, a linear regression algorithm based on Euclidean distance, and a linear regression algorithm based on Welsch distance.

[0156] In some embodiments, the center point of the calibration plate image is set as the rotation center, a rotation matrix is ​​constructed based on the rotation center and the rotation angle, and the rotation matrix is ​​used to rotate the original rectangular dot matrix to a positive position, thereby rotating the original rectangular dot matrix to a required angle to obtain a positive dot matrix, which includes an X-axis and a Y-axis.

[0157] In some embodiments, the original rectangular lattice is rotated positively using a rotation matrix, that is, each valid circular point is rotated using a rotation matrix formula, and the rotation matrix formula is:

[0158]

[0159] in,( , ) are the coordinates of valid points in the original rectangular lattice, ( , ) is the coordinate of the effective point in the positive lattice obtained after rotation. is the rotation angle.

[0160] S340: Based on the actual dot distance and Y-axis resolution of each valid dot on the calibration plate, the centroid of the valid dots is traversed to identify the relative position of the valid dots on the Y-axis, and the relative position of the valid dots in the same row on the X-axis is identified based on the X-axis resolution to find the coordinates of each valid dot in the dot matrix coordinate system.

[0161] In some optional embodiments, the dot information of the valid dots obtained above includes the actual dot distance of the dots, which indicates the spacing of the valid dots in the actual physical space, that is, the distance between the calibration blocks in the calibration plate. The distance between the calibration blocks refers to the distance between the center of mass of the calibration blocks. The distance between the left and right calibration blocks or the upper and lower calibration blocks in the calibration plate is the same (that is, the distance between any two calibration blocks in the calibration plate is equal).

[0162] In some optional embodiments, the Y-axis dot pitch is calculated based on the actual dot pitch of the effective dots and the Y-axis resolution, and the Y-axis dot pitch is equal to the product of the actual dot pitch of the effective dots and the Y-axis resolution. The ordinate of the centroid of each effective dot is multiplied by the Y-axis resolution to obtain the actual physical ordinate of each effective dot in the dot matrix coordinate system. According to the actual physical ordinate of each effective dot in the dot matrix coordinate system, all effective dots are sorted to determine the relative position of each effective dot on the Y-axis, thereby identifying the relative position of the effective dot on the Y-axis.

[0163] In some optional embodiments, a line break threshold is set, and the centroids of valid dots are traversed from the first row. For the centroid of each valid dot, the target difference between the ordinate of the centroid of the next valid dot and the ordinate of the centroid of the current valid dot is calculated, and the target difference is divided by the Y-axis point distance to obtain a first ratio. If the first ratio is less than the line break threshold, the next valid dot is divided into the row where the current valid dot is located. If the first ratio is greater than or equal to the line break threshold, it means that the next valid dot is not located in the same row as the current valid dot, and the next valid dot is divided into the next row of the row where the current valid dot is located.

[0164] After completing the row division, according to the centroid of each valid dot in each row, multiply the horizontal coordinate of the centroid of each valid dot by the X-axis resolution to obtain the actual physical horizontal coordinate of each valid dot in the dot matrix coordinate system. According to the actual physical horizontal coordinate of each valid dot in the dot matrix coordinate system, the valid dots are sorted in ascending order to determine the relative position of the valid dots in the same row on the X-axis (that is, the actual physical horizontal coordinate in the dot matrix coordinate system). Combined with the relative position of the valid dots on the Y-axis (that is, the actual physical vertical coordinate in the dot matrix coordinate system), the coordinates of each valid dot in the dot matrix coordinate system are obtained.

[0165] In some embodiments, based on the lattice coordinate system, the centroids of the four vertices of the largest inscribed rectangular lattice are obtained, including but not limited to the following steps S410-S430:

[0166] S410: Identify the leftmost dot mass center and the rightmost dot mass center of each row among all the valid dot mass centers.

[0167] In some optional embodiments, the dot information of the valid dots obtained above includes the actual dot distance of the dots, which indicates the spacing of the valid dots in the actual physical space, that is, the distance between the calibration blocks in the calibration plate. The distance between the calibration blocks refers to the distance between the center of mass of the calibration blocks. The distance between the left and right calibration blocks or the upper and lower calibration blocks in the calibration plate is the same (that is, the distance between any two calibration blocks in the calibration plate is equal).

[0168] Specifically, among all the valid dots in the dot matrix coordinate system, for the valid dots in each row, the X-axis coordinates of the valid dots in the row are compared to determine whether a certain valid dot is the leftmost valid dot or the rightmost valid dot, thereby identifying the leftmost valid dot and the rightmost valid dot in each row, and determining the centroid of the leftmost valid dot and the rightmost valid dot as the centroid of the leftmost dot and the centroid of the rightmost dot, respectively.

[0169] S420: Calculate the X-axis dot pitch based on the X-axis resolution and the actual dot pitch, where the X-axis resolution and the X-axis dot pitch are both in pixels.

[0170] Specifically, the X-axis dot pitch is calculated according to the actual dot pitch of the effective dots and the X-axis resolution. The X-axis dot pitch is equal to the product of the actual dot pitch of the effective dots and the X-axis resolution. Both the X-axis resolution and the X-axis dot pitch are in pixels.

[0171] S430: Obtain the rotation direction of the original rectangular lattice, and traverse the maximum inscribed rectangular lattice according to different rotation directions to calculate the centroids of the four vertices of the maximum inscribed rectangular lattice, where the centroids of the four vertices include the centroid of the upper left corner, the centroid of the upper right corner, the centroid of the lower left corner, and the centroid of the lower right corner.

[0172] Specifically, before the original rectangular lattice is rotated, the rotation direction and rotation angle of the original rectangular lattice are calculated, and the rotation direction of the original rectangular lattice is obtained from the calculation result. According to the rotation direction and the position of the valid circle point, the centroids of the four vertices of the maximum inscribed rectangle are calculated, and the positions of the centroids of the four vertices are determined. The centroids of the four vertices include the centroid of the upper left corner, the centroid of the upper right corner, the centroid of the lower left corner, and the centroid of the lower right corner.

[0173] In some embodiments, if the rotation direction of the original rectangular dot matrix is ​​clockwise, first calculate and obtain the centroid of the upper right corner of the largest inscribed rectangular dot matrix, start from the leftmost centroid of the first row of the original rectangular dot matrix (i.e., the positive dot matrix) after rotation, search downward row by row along the left side of the positive dot matrix, calculate the difference between the horizontal coordinate of the leftmost centroid of the current row and the horizontal coordinate of the leftmost centroid of the next row, and compare the difference between the horizontal coordinate of the leftmost centroid of the current row and the leftmost centroid of the next row with the X-axis point distance.

[0174] If the difference between the horizontal coordinate of the leftmost centroid of the current row and the horizontal coordinate of the leftmost centroid of the next row is less than or equal to the X-axis point distance, the rightmost centroid of the current row is determined to be the centroid of the upper right corner of the largest inscribed rectangular dot matrix.

[0175] If the difference between the horizontal coordinates of the leftmost centroid of the current row and the leftmost centroid of the next row is greater than the X-axis point distance, the next row is updated to the current row, and the difference between the horizontal coordinates of the leftmost centroid of the current row and the leftmost centroid of the next row and the X-axis point distance are continued to be compared until the centroid of the upper right corner of the largest inscribed rectangular dot matrix is ​​found.

[0176] Next, calculate and obtain the centroid of the lower left corner of the largest inscribed rectangular lattice, starting from the rightmost centroid of the last row of the original rectangular lattice (i.e., the positive lattice) after rotation, and search upward row by row along the right side of the positive lattice, calculate the difference between the horizontal coordinate of the rightmost centroid of the current row and the horizontal coordinate of the rightmost centroid of the previous row, and compare the difference between the horizontal coordinate of the rightmost centroid of the current row and the rightmost centroid of the previous row with the X-axis point distance.

[0177] If the difference between the abscissa of the rightmost centroid of the current row and the abscissa of the rightmost centroid of the previous row is less than or equal to the X-axis point distance, the leftmost centroid of the current row is determined to be the lower left corner centroid of the maximum inscribed rectangular dot matrix.

[0178] If the difference between the horizontal coordinate of the rightmost centroid of the current row and the horizontal coordinate of the rightmost centroid of the previous row is greater than the X-axis point distance, the previous row is updated to the current row, and the difference between the horizontal coordinate of the rightmost centroid of the current row and the horizontal coordinate of the rightmost centroid of the previous row and the X-axis point distance are continued to be compared until the centroid of the lower left corner of the maximum inscribed rectangular dot matrix is ​​found.

[0179] Then, calculate and obtain the lower right corner centroid of the maximum inscribed rectangular lattice, start from the determined upper right corner centroid, search downward row by row, determine the upper right corner centroid as the anchor point, and determine the current row where the anchor point is located. Determine the target valid circle centroid with the closest Euclidean distance to the anchor point among the valid circle centroids in the next row, and update the anchor point to the target valid circle centroid, and determine whether the updated anchor point (i.e., the target valid circle centroid) and the lower left corner centroid are in the same row.

[0180] If the updated anchor point (i.e., the target effective circle point mass center) and the lower left corner mass center are in the same row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the lower right corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) and the lower left corner mass center are not in the same row, the current row where the updated anchor point is located is continued to be determined, and the target effective circle point mass center with the closest Euclidean distance to the updated anchor point is determined among the effective circle point mass centers in the next row, and the anchor point is updated to the target effective circle point mass center, and it is determined whether the updated anchor point (i.e., the target effective circle point mass center) and the lower left corner mass center are in the same row, until the lower right corner mass center of the maximum inscribed rectangular lattice is found.

[0181] Finally, calculate and obtain the centroid of the upper left corner of the maximum inscribed rectangular lattice, start from the determined centroid of the lower left corner, search upward row by row, determine the centroid of the lower left corner as the anchor point, and determine the current row where the anchor point is located. Determine the target valid circle point centroid with the closest Euclidean distance to the anchor point among the valid circle point centroids in the previous row, and update the anchor point to the target valid circle point centroid, and determine whether the updated anchor point (that is, the target valid circle point centroid) and the upper right corner centroid are in the same row.

[0182] If the updated anchor point (i.e., the target effective circle point mass center) and the upper right corner mass center are in the same row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the upper left corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) and the upper right corner mass center are not in the same row, the current row where the updated anchor point is located is continued to be determined, and the target effective circle point mass center with the closest Euclidean distance to the updated anchor point is determined among the effective circle point mass centers of the previous row, and the anchor point is updated to the target effective circle point mass center, and it is determined whether the updated anchor point (i.e., the target effective circle point mass center) and the upper right corner mass center are in the same row, until the upper left corner mass center of the maximum inscribed rectangular lattice is found.

[0183] At this point, the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid of the maximum inscribed rectangle are calculated and obtained, and the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid are used as the centroids of the four vertices of the maximum inscribed rectangular-like lattice.

[0184] In some embodiments, if the rotation direction of the original rectangular dot matrix is ​​counterclockwise, first calculate and obtain the centroid of the upper left corner of the largest inscribed rectangular dot matrix, start from the rightmost centroid of the first row of the original rectangular dot matrix (i.e., the positive dot matrix) after rotation, search downward row by row along the right side of the positive dot matrix, calculate the difference between the horizontal coordinate of the rightmost centroid of the current row and the horizontal coordinate of the rightmost centroid of the next row, and compare the difference between the horizontal coordinate of the rightmost centroid of the current row and the rightmost centroid of the next row with the X-axis point distance.

[0185] If the difference between the abscissa of the rightmost centroid of the current row and the rightmost centroid of the next row is less than or equal to the X-axis point distance, the leftmost centroid of the current row is determined to be the centroid of the upper left corner of the largest inscribed rectangular dot matrix.

[0186] If the difference between the horizontal coordinates of the rightmost centroid of the current row and the rightmost centroid of the next row is greater than the X-axis point distance, the next row is updated to the current row, and the difference between the horizontal coordinates of the rightmost centroid of the current row and the rightmost centroid of the next row and the X-axis point distance are continued to be compared until the centroid of the upper left corner of the largest inscribed rectangular dot matrix is ​​found.

[0187] Next, calculate and obtain the centroid of the lower right corner of the largest inscribed rectangular lattice, starting from the leftmost centroid of the last row of the original rectangular lattice (i.e., the positive lattice) after rotation, and search upward row by row along the left side of the positive lattice, calculate the difference between the horizontal coordinate of the leftmost centroid of the current row and the horizontal coordinate of the leftmost centroid of the previous row, and compare the difference between the horizontal coordinate of the leftmost centroid of the current row and the leftmost centroid of the previous row with the X-axis point distance.

[0188] If the difference between the horizontal coordinate of the leftmost centroid of the current row and the horizontal coordinate of the leftmost centroid of the previous row is less than or equal to the X-axis point distance, the rightmost centroid of the current row is determined to be the centroid of the lower right corner of the maximum inscribed rectangular dot matrix.

[0189] If the difference between the horizontal coordinate of the leftmost centroid of the current row and the horizontal coordinate of the leftmost centroid of the previous row is greater than the X-axis point distance, the previous row is updated to the current row, and the difference between the horizontal coordinate of the leftmost centroid of the current row and the leftmost centroid of the previous row and the X-axis point distance are continued to be compared until the centroid of the lower right corner of the maximum inscribed rectangular dot matrix is ​​found.

[0190] Then, calculate and obtain the lower left corner centroid of the maximum inscribed rectangular lattice, start from the determined upper left corner centroid, search downward row by row, determine the upper left corner centroid as the anchor point, and determine the current row where the anchor point is located, determine the target valid circle point centroid with the closest Euclidean distance to the anchor point among the valid circle point centroids in the next row, and update the anchor point to the target valid circle point centroid, and determine whether the updated anchor point (i.e., the target valid circle point centroid) and the lower right corner centroid are in the same row.

[0191] If the updated anchor point (i.e., the target effective circle point mass center) and the lower right corner mass center are in the same row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the lower left corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) and the lower right corner mass center are not in the same row, the current row where the updated anchor point is located is continued to be determined, and the target effective circle point mass center with the closest Euclidean distance to the updated anchor point is determined among the effective circle point mass centers in the next row, and the anchor point is updated to the target effective circle point mass center, and it is determined whether the updated anchor point (i.e., the target effective circle point mass center) and the lower right corner mass center are in the same row, until the lower left corner mass center of the maximum inscribed rectangular lattice is found.

[0192] Finally, calculate and obtain the centroid of the upper right corner of the maximum inscribed rectangular lattice. Starting from the determined centroid of the lower right corner, search upward row by row, determine the centroid of the lower right corner as the anchor point, and determine the current row where the anchor point is located. Determine the target valid circle centroid with the closest Euclidean distance to the anchor point among the valid circle centroids in the previous row, and update the anchor point to the target valid circle centroid. Determine whether the updated anchor point (i.e., the target valid circle centroid) and the upper left corner centroid are in the same row.

[0193] If the updated anchor point (i.e., the target effective circle point mass center) and the upper left corner mass center are in the same row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the upper right corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) and the upper left corner mass center are not in the same row, the current row where the updated anchor point is located is continued to be determined, and the target effective circle point mass center with the closest Euclidean distance to the updated anchor point is determined among the effective circle point mass centers of the previous row, and the anchor point is updated to the target effective circle point mass center, and it is determined whether the updated anchor point (i.e., the target effective circle point mass center) and the upper left corner mass center are in the same row, until the upper right corner mass center of the maximum inscribed rectangular lattice is found.

[0194] At this point, the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid of the maximum inscribed rectangle are calculated and obtained, and the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid are used as the centroids of the four vertices of the maximum inscribed rectangular-like lattice.

[0195] In some embodiments, if the rotation direction of the original quasi-rectangular dot matrix is ​​no obvious clockwise rotation or counterclockwise rotation, the center of mass of the effective circle point center of mass at the leftmost end of the last row is determined as the center of mass of the lower left corner, and the center of mass of the effective circle point center of mass at the rightmost end of the last row is determined as the center of mass of the lower right corner. It can be understood that no obvious clockwise rotation or counterclockwise rotation means that the rotation angle of the original quasi-rectangular dot matrix is ​​less than or equal to the preset angle threshold. Those skilled in the art can set the preset angle threshold according to actual needs. For example, in some embodiments, the preset angle threshold is set to 3°, 5° or other angle values.

[0196] Next, calculate and obtain the centroid of the upper right corner of the maximum inscribed rectangular lattice, start from the determined centroid of the lower right corner, search upward row by row, determine the centroid of the lower right corner as the anchor point, and determine the current row where the anchor point is located. Determine the target valid circle point centroid with the closest Euclidean distance to the anchor point among the valid circle point centroids in the previous row, and update the anchor point to the target valid circle point centroid, and determine whether the updated anchor point (i.e., the target valid circle point centroid) is located in the first row.

[0197] Among them, if the updated anchor point (i.e., the target effective circle point mass center) is located in the first row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the upper right corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) is not located in the first row, then continue to determine the current row where the updated anchor point is located, determine the target effective circle point mass center with the closest Euclidean distance to the updated anchor point from the effective circle point mass centers of the previous row, and update the anchor point to the target effective circle point mass center, and determine whether the updated anchor point (i.e., the target effective circle point mass center) is located in the first row, until the upper right corner mass center of the maximum inscribed rectangular lattice is found.

[0198] Then, calculate and obtain the centroid of the upper left corner of the maximum inscribed rectangular lattice, start from the determined centroid of the lower left corner, search upward row by row, determine the centroid of the lower left corner as the anchor point, and determine the current row where the anchor point is located. Determine the target valid circle point centroid with the closest Euclidean distance to the anchor point among the valid circle point centroids in the previous row, and update the anchor point to the target valid circle point centroid, and determine whether the updated anchor point (i.e., the target valid circle point centroid) is located in the first row.

[0199] Among them, if the updated anchor point (i.e., the target effective circle point mass center) is located in the first row, the updated anchor point (i.e., the target effective circle point mass center) is determined to be the lower left corner mass center of the maximum inscribed rectangular lattice. If the updated anchor point (i.e., the target effective circle point mass center) is not located in the first row, then continue to determine the current row where the updated anchor point is located, determine the target effective circle point mass center with the closest Euclidean distance to the updated anchor point from the effective circle point mass centers of the previous row, and update the anchor point to the target effective circle point mass center, and determine whether the updated anchor point (i.e., the target effective circle point mass center) is located in the first row, until the upper left corner mass center of the maximum inscribed rectangular lattice is found.

[0200] At this point, the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid of the maximum inscribed rectangle are calculated and obtained, and the upper right corner centroid, lower left corner centroid, lower right corner centroid and upper left corner centroid are used as the centroids of the four vertices of the maximum inscribed rectangular-like lattice.

[0201] In some embodiments, the largest inscribed rectangular-like point array is found among all valid circular points based on the centroid of the four vertices, including but not limited to the following steps S440-S470:

[0202] S440: Find the centroids of all valid dots in the first row of the maximum inscribed rectangular lattice according to the coordinates of the centroid of the upper left corner and the coordinates of the centroid of the upper right corner.

[0203] Specifically, the valid circular points of the maximum inscribed rectangular quasi-dot matrix are arranged in rows and columns, and the valid circular points in the maximum inscribed rectangular quasi-dot matrix that are located in the same row as the centroid of the upper left corner or the centroid of the upper right corner are determined according to the longitudinal coordinate of the centroid of the upper left corner or the longitudinal coordinate of the centroid of the upper right corner. According to the horizontal coordinate of the centroid of the upper left corner and the horizontal coordinate of the centroid of the upper right corner, among the valid circular points that are located in the same row as the centroid of the upper left corner or the centroid of the upper right corner, the valid circular points whose horizontal coordinates are between the horizontal coordinates of the centroid of the upper left corner and the horizontal coordinates of the centroid of the upper right corner are determined, and the valid circular points whose horizontal coordinates are between the horizontal coordinates of the centroid of the upper left corner and the horizontal coordinates of the centroid of the upper right corner are taken as all the valid circular points in the first row of the maximum inscribed rectangular quasi-dot matrix, thereby finding the centroid of all the valid circular points in the first row of the maximum inscribed rectangular quasi-dot matrix.

[0204] S450: Starting from each centroid of the first row, set the jth row as the anchor point, search the centroid closest to the anchor point in the j+1th row, and determine whether the centroid closest to the anchor point is in the same row as the lower left corner centroid or the lower right corner centroid.

[0205] S460: When they are not in the same row, the centroid is set as a new anchor point.

[0206] Exemplarily, starting from each centroid of the first row in the maximum inscribed rectangular lattice, search downward row by row, set the centroid of each valid circle point in the jth row (j=1, the first row at this time) as the anchor point, and for each anchor point in the jth row, search in the next row (j+1th row, the second row at this time) for the target valid circle point centroid with the closest Euclidean distance to the anchor point, and determine whether the target valid circle point centroid is in the same row as the lower left corner centroid or the lower right corner centroid.

[0207] If the target effective circle point centroid is not in the same row as the lower left corner centroid or the lower right corner centroid, set the target effective circle point centroid as the new anchor point, and continue to search for the target effective circle point centroid with the closest Euclidean distance to the new anchor point in the next row, and determine whether the target effective circle point centroid is in the same row as the lower left corner centroid or the lower right corner centroid, until the target effective circle point centroid is in the same row as the lower left corner centroid or the lower right corner centroid, end the search, and record all target effective circle point centroids that are not in the same row as the lower left corner centroid or the lower right corner centroid, so as to obtain all centroids located in the maximum inscribed class rectangle.

[0208] S470: When in the same row, end the search and record the current anchor point. At this time, all the centroids located in the maximum inscribed class rectangle have been found. All the found centroids are recorded as class rectangle centroids, and the dot matrix formed by the valid circular points where the class rectangle centroid is located is recorded as the maximum inscribed class rectangle dot matrix.

[0209] If the target effective circle point centroid is in the same row as the lower left corner centroid or the lower right corner centroid, the search is terminated and all the target effective circle point centroids (i.e., anchor points in each row) that are not in the same row as the lower left corner centroid or the lower right corner centroid are recorded. At this time, all the centroids found in the maximum inscribed class rectangle are recorded as the class rectangle centroid, and the dot matrix composed of the effective circle points where the class rectangle centroid and the four vertex centroids (i.e., the upper left corner centroid, the lower left corner centroid, the upper right corner centroid, and the lower right corner centroid) are located is recorded as the maximum inscribed class rectangle dot matrix.

[0210] In some embodiments, the largest inscribed rectangular-like dot matrix is ​​found among all valid circular points based on the centroid of the four vertices, specifically including but not limited to the following steps S480-S495:

[0211] S480: Find the centroids of all valid dots in the last row of the maximum inscribed rectangular lattice according to the coordinates of the centroid of the lower left corner and the coordinates of the centroid of the lower right corner.

[0212] Specifically, the valid circular points of the maximum inscribed rectangular quasi-dot matrix are arranged in rows and columns, and the valid circular points in the maximum inscribed rectangular quasi-dot matrix that are located in the same row as the centroid of the lower left corner or the centroid of the lower right corner are determined according to the longitudinal coordinate of the centroid of the lower left corner or the centroid of the lower right corner, and the valid circular points whose horizontal coordinates are between the horizontal coordinates of the centroid of the lower left corner and the centroid of the lower right corner are determined among the valid circular points that are located in the same row as the centroid of the lower left corner or the centroid of the lower right corner according to the horizontal coordinates of the centroid of the lower left corner and the centroid of the lower right corner, and the valid circular points whose horizontal coordinates are between the horizontal coordinates of the centroid of the lower left corner and the centroid of the lower right corner are taken as all the valid circular points in the last row of the maximum inscribed rectangular quasi-dot matrix, thereby finding the centroids of all the valid circular points in the last row of the maximum inscribed rectangular quasi-dot matrix.

[0213] S485: Starting from each centroid of the last row, set the jth row as the anchor point, search the centroid closest to the anchor point in the j-1th row in Euclidean distance, and determine whether the centroid closest to the anchor point is in the same row as the upper left corner centroid or the upper right corner centroid.

[0214] S490: When they are not in the same row, the centroid is set as a new anchor point.

[0215] Exemplarily, starting from each centroid of the first row in the maximum inscribed rectangular lattice, search downward row by row, set the centroid of each valid circle point in the jth row (j=1, the last row at this time) as the anchor point, and for each anchor point in the jth row, find the target valid circle point centroid with the closest Euclidean distance to the anchor point in the previous row (j-1th row, the second to last row at this time), and determine whether the target valid circle point centroid is in the same row as the upper left corner centroid or the upper right corner centroid.

[0216] If the target effective circle point centroid is not in the same row as the upper left corner centroid or the upper right corner centroid, set the target effective circle point centroid as the new anchor point, and continue to search for the target effective circle point centroid with the closest Euclidean distance to the new anchor point in the previous row, and determine whether the target effective circle point centroid is in the same row as the upper left corner centroid or the upper right corner centroid, until the target effective circle point centroid is in the same row as the upper left corner centroid or the upper right corner centroid, end the search, and record all target effective circle point centroids that are not in the same row as the upper left corner centroid or the upper right corner centroid, so as to obtain all centroids located in the maximum inscribed class rectangle.

[0217] S495: When in the same row, end the search and record the current anchor point. At this time, all the centroids located in the maximum inscribed class rectangle have been found. All the found centroids are recorded as the class rectangle centroids, and the dot matrix formed by the valid circular points where the class rectangle centroids are located is recorded as the maximum inscribed class rectangle dot matrix.

[0218] If the target effective circle point centroid is in the same row as the upper left corner centroid or the upper right corner centroid, the search is terminated and all the target effective circle point centroids (i.e., anchor points in each row) that are not in the same row as the upper left corner centroid or the upper right corner centroid are recorded. At this time, all the centroids in the maximum inscribed class rectangle have been found. All the centroids found in the maximum inscribed class rectangle are recorded as the class rectangle centroid, and the dot matrix composed of the effective circle points where the class rectangle centroid and the four vertex centroids (i.e., the upper left corner centroid, the lower left corner centroid, the upper right corner centroid, and the lower right corner centroid) are located is recorded as the maximum inscribed class rectangle dot matrix.

[0219] In some embodiments, an ideal rectangular dot matrix is ​​generated according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image, specifically including but not limited to the following steps S510-S540:

[0220] S510: Obtain the width and height of the calibration plate image.

[0221] In an embodiment of the present invention, before using a camera to take a picture of the calibration plate, the size of the picture taken by the camera (including width and height) is set. In this way, after the camera takes the picture of the calibration plate, the width and height of the calibration plate picture can be known.

[0222] Specifically, after obtaining the calibration plate image by photographing with a camera, the size of the calibration plate image is read to obtain the width and height of the calibration plate image.

[0223] S520: Determine the width and height of the ideal rectangular dot matrix according to the width and height of the maximum inscribed quasi-rectangular dot matrix.

[0224] Specifically, the width and height of the largest inscribed quasi-rectangular dot matrix found are calculated. The width and height of the ideal rectangular dot matrix are calculated based on the width and height of the largest inscribed quasi-rectangular dot matrix and the width and height of the calibration plate image.

[0225] According to the width and height of the largest inscribed rectangular lattice and the width and height of the calibration plate image, the width and height of the ideal rectangular lattice are calculated, including:

[0226] The width of the calibration plate image is used as the width of the ideal rectangular dot matrix, the height of the maximum inscribed rectangular lattice and the width of the maximum inscribed rectangular lattice are equal to a first ratio, and the first ratio is multiplied by the width of the calibration plate image to obtain the height of the ideal rectangular dot matrix.

[0227] In some embodiments, the width and height of the ideal rectangular dot matrix are calculated according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image, including:

[0228] The height of the calibration plate image is used as the height of the ideal rectangular dot matrix, the width of the maximum inscribed rectangular lattice and the height of the maximum inscribed rectangular lattice are equal to the second ratio, and the second ratio is multiplied by the height of the calibration plate image to obtain the height of the ideal rectangular dot matrix.

[0229] S530: performing coordinate system conversion according to the width and height of the maximum inscribed quasi-rectangular lattice and the width and height of the ideal rectangular lattice, and obtaining all coordinates of the ideal rectangular lattice.

[0230] Specifically, according to the width and height of the maximum inscribed rectangular lattice and the width and height of the ideal rectangular lattice, the coordinates of the valid points of the maximum inscribed rectangular lattice in the lattice coordinate system are converted into coordinates in the ideal rectangular lattice, the coordinate system conversion is completed, and the coordinates of all valid points in the ideal rectangular lattice are obtained.

[0231] In some embodiments, the specific process of coordinate system conversion is: divide the horizontal coordinate of the effective circle point in the dot matrix coordinate system by the width of the largest inscribed rectangular dot matrix to obtain a first ratio, multiply the first ratio by the width of the ideal rectangular dot matrix to obtain the horizontal coordinate of the effective circle point in the ideal rectangular dot matrix. Divide the vertical coordinate of the effective circle point in the dot matrix coordinate system by the height of the largest inscribed rectangular dot matrix to obtain a second ratio, multiply the second ratio by the height of the ideal rectangular dot matrix to obtain the vertical coordinate of the effective circle point in the ideal rectangular dot matrix.

[0232] S540: Establishing an ideal rectangular dot matrix according to all the obtained coordinates of the ideal rectangular dot matrix.

[0233] Specifically, after obtaining the coordinates of all valid points in the ideal rectangular dot matrix, the coordinates of all valid points are sorted by rows and columns, and the coordinates of the valid points in each row increase in the X-axis direction, and the coordinates in the Y-axis direction remain unchanged, and the coordinates of the valid points in each column increase in the Y-axis direction, and the coordinates in the X-axis direction remain unchanged. The coordinates of the valid points are inserted into a suitable container (such as a two-dimensional array, a list, or a collection of point objects, etc.), and are filled into each row and column of the ideal rectangular dot matrix in turn according to the horizontal and vertical coordinates of each valid point, ensuring that the position of each valid point corresponds correctly, thereby establishing an ideal rectangular dot matrix.

[0234] In some embodiments, based on the original rectangular lattice and the ideal rectangular lattice, a homography matrix is ​​calculated, which specifically includes but is not limited to the following steps S610-S620:

[0235] S610: Obtain the rotation angle and rotation center of the original rectangular lattice, construct a rotation matrix according to the rotation angle, and rotate the rotated lattice based on the rotation matrix to rotate the maximum inscribed rectangular lattice back to the original angle.

[0236] Specifically, the rotation angle of the original rectangular lattice can be obtained by calculating the main directions of the original rectangular lattice (for example, using the least square method to fit a straight line or by principal component analysis (PCA)).

[0237] In some embodiments, obtaining the rotation angle of the original rectangular lattice includes: calculating the coordinates (x, y) of the center point of the calibration plate image according to the width and height of the calibration plate image, the horizontal coordinate x of the center point is equal to the width W / 2 of the calibration plate image, and the horizontal coordinate y of the center point is equal to the width H / 2 of the calibration plate image. Traverse the centroids of all valid dots, calculate the Euclidean distance between the centroid coordinates of each valid dot and the coordinates of the center point of the calibration plate image, and determine the centroid of the valid dot closest to the center point of the calibration plate image as the center centroid. Then, take the center centroid as the target centroid, and find the two neighboring centroids closest to the center centroid, wherein the two neighboring centroids are respectively located on the left and right of the target centroid. According to the target centroid and the two neighboring centroids, use a linear fitting algorithm to fit the target straight line, and calculate the rotation direction and rotation angle of the original rectangular lattice based on the target straight line.

[0238] It can be understood that the linear fitting algorithm includes but is not limited to: a linear regression algorithm based on taxis distance, a linear regression algorithm based on Euclidean distance, and a linear regression algorithm based on Welsch distance.

[0239] In some embodiments, the center point of the calibration plate image is set as the rotation center, a rotation matrix is ​​constructed based on the rotation center and the rotation angle, and the rotation matrix is ​​used to rotate the original rectangular lattice after the rotation back to the original angle (i.e., the angle before the original rectangular lattice), so as to rotate the maximum inscribed rectangular lattice back to the original angle, and obtain the maximum inscribed rectangular lattice rotated back to the original angle. Obviously, the maximum inscribed rectangular lattice is a lattice composed of some valid dots in the original rectangular lattice, so the original rectangular lattice after the rotation is rotated back to the original angle (i.e., the maximum inscribed rectangular lattice follows the rotation back to the original angle), and the maximum inscribed rectangular lattice rotated back to the original angle can be obtained.

[0240] In some embodiments, the original rectangular dot matrix after rotation is rotated back to the original angle using a rotation matrix, that is, each valid circle point is rotated back to the original angle using a rotation matrix formula, and the rotation matrix formula is:

[0241]

[0242] in,( , ) is the coordinate of the valid circle point in the original rectangular lattice after rotation, ( , ) are the coordinates of the valid points in the original rectangular dot matrix rotated back to the original angle, is the rotation angle and is a negative value.

[0243] S620: Calculate a homography matrix based on the ideal rectangular lattice and the rotated maximum inscribed quasi-rectangular lattice.

[0244] Among them, the homography matrix It is a 3x3 matrix that can describe the perspective transformation between two planes. The homography matrix transforms a point on a plane from the original coordinate system to the target coordinate system.

[0245] Specifically, at least 4 groups of corresponding dots are obtained from the maximum inscribed rectangular quasi-matrix rotated back to the original angle and the ideal rectangular quasi-matrix, each group of dots includes a valid dot in the maximum inscribed rectangular quasi-matrix rotated back to the original angle and the dot corresponding to the valid dot in the ideal rectangular dot matrix.

[0246] In some embodiments, the coordinates of at least 4 groups of corresponding dots are homogenized, and the homography matrix is ​​calculated using the cv2.findHomography() function of OpenCV. The coordinates of the at least 4 groups of corresponding dots after homogenization are input into the cv2.findHomography() function. The coordinates of the at least 4 groups of corresponding dots after homogenization are constructed into a matrix multiplication equation through the homography matrix. The constructed matrix multiplication equation is calculated using the cv2.findHomography() function to obtain a 3x3 homography matrix. .

[0247] It can be understood that the more coordinates of the corresponding circles in the maximum inscribed quasi-rectangular lattice and the ideal rectangular lattice rotated back to the original angle are used to construct the matrix multiplication equation, the number of known numbers in the constructed matrix multiplication equation is much larger than the number of unknown numbers, and the matrix multiplication equation is an overdetermined equation. Algorithms for calculating overdetermined equations include but are not limited to the RANSAC algorithm, the LmedS algorithm, and the least squares method.

[0248] The embodiment of the present invention uses a camera to take a calibration plate image, pre-processes the calibration plate image to obtain a new calibration plate image, screens out valid dots of the original rectangular dot matrix in the new calibration plate image, obtains dot information of the valid dots, establishes a dot matrix coordinate system based on the identified dot information, rotates the original rectangular dot matrix positively and finds the coordinates of each valid dot in the dot matrix coordinate system according to the dot information, obtains the maximum inscribed rectangular dot matrix based on the dot matrix coordinate system, generates an ideal rectangular dot matrix according to the width and height of the maximum inscribed rectangular dot matrix and the width and height of the calibration plate image, calculates a homography matrix based on the original rectangular dot matrix and the ideal rectangular dot matrix, and corrects the measured photos according to the homography matrix to correct the camera field of view. In this way, the embodiment of the present invention can correct the camera field of view and obtain a picture with an accurate camera field of view, so that the surface information of the product structure can be accurately restored and the accuracy of product structure defect detection can be improved.

[0249] An embodiment of the present invention provides a computer-readable storage medium, on which computer program instructions executable by a processor are stored. When the computer program instructions are called by the processor, the processor executes any camera field of view correction method provided by the embodiment of the present invention, or executes steps in any possible implementation manner of any camera field of view correction method provided by the embodiment of the present invention.

[0250] Through the description of the above implementation methods, those skilled in the art can understand that all or part of the processes in the above embodiment methods can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a register, a read-only memory (ROM), a flash memory, a hard disk, a magnetic surface storage, a removable disk, a CD-ROM, or a random access memory (RAM), etc.

[0251] The above disclosure is only the preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A camera field of view correction method, characterized in that: The method comprises: The calibration plate image taken by the camera is preprocessed to obtain a new calibration plate image, wherein the new calibration plate image contains an original rectangular dot matrix, and the distribution of the original rectangular dot matrix corresponds to the distribution of the calibration blocks of the calibration plate; Identify and filter out valid dots of the original rectangular dot matrix, and obtain dot information of the valid dots, including the center of mass of the dots; Based on the identified dot information, a dot matrix coordinate system is established, and the original rectangular dot matrix is ​​rotated positively and the coordinates of each valid dot in the dot matrix coordinate system are found according to the dot information. The dot matrix coordinate system includes an X-axis and a Y-axis. Based on the lattice coordinate system, obtain the centroids of the four vertices of the largest inscribed rectangular lattice, and find the largest inscribed rectangular lattice among all valid circular points based on the centroids of the four vertices; Generate an ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image; Based on the original rectangular lattice and the ideal rectangular lattice, the homography matrix is ​​calculated; The measured photos are rectified according to the homography matrix to correct the camera field of view.

2. The method according to claim 1, characterized in that The identifying and screening out the valid dots of the original rectangular dot matrix and obtaining dot information of the valid dots includes: Traverse all connected domains to perform connected domain analysis on the new calibration board map, and obtain the connected domains and connected domain information in the new calibration board map. The connected domain information includes the total number of connected domains, centroid coordinates and area, and the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain; Based on the coordinates of the upper left corner, width and height of the circumscribed rectangle of each connected domain, incomplete connected domains are filtered out, where the incomplete connected domains are invalid dots; Obtain the median area of ​​each connected domain according to its area, and set the area difference threshold based on the median; The connected domains are filtered based on the area difference threshold to obtain qualified connected domains, and the connected domain information of the qualified connected domains is the dot information of the valid dots.

3. The method according to claim 1, characterized in that The step of establishing a dot matrix coordinate system based on the identified dot information includes: Calculate the real resolution of the camera based on the dot information and the actual size of the calibration block on the calibration plate. The real resolution includes the X-axis resolution and the Y-axis resolution. A dot matrix coordinate system is established according to the X-axis resolution and the Y-axis resolution. The X-axis resolution corresponds to the X-axis of the dot matrix coordinate system, and the Y-axis resolution corresponds to the Y-axis of the dot matrix coordinate system.

4. The method according to claim 3, characterized in that The dot information includes the actual dot distance of the dots. The method of rotating the original rectangular dot matrix and finding the coordinates of each valid dot in the dot matrix coordinate system according to the dot information includes: Calculate the rotation angle of the original rectangular dot matrix, and rotate the original rectangular dot matrix to the required angle according to the rotation angle to obtain a positive dot matrix, where the positive dot matrix includes an X-axis and a Y-axis; Based on the actual dot distance and Y-axis resolution of each valid dot on the calibration plate, the centroid of the valid dots is traversed to identify the relative position of the valid dots on the Y-axis, and the relative position of the valid dots in the same row on the X-axis is identified based on the X-axis resolution to find the coordinates of each valid dot in the dot matrix coordinate system.

5. The method according to claim 1, characterized in that The dot information includes the actual dot distance of the dots. The method of obtaining the centroid of the four vertices of the largest inscribed rectangular dot matrix based on the dot matrix coordinate system includes: Identify the leftmost and rightmost centroids of each row of all the centroids of valid dots; Calculate the X-axis dot distance based on the X-axis resolution and the actual dot distance. The X-axis resolution and the X-axis dot distance are both in pixels. The rotation direction of the original rectangular lattice is obtained, and according to different rotation directions, the maximum inscribed rectangular lattice is traversed to calculate the centroids of the four vertices of the maximum inscribed rectangular lattice, where the centroids of the four vertices include the centroid of the upper left corner, the centroid of the upper right corner, the centroid of the lower left corner, and the centroid of the lower right corner.

6. The method according to claim 5, characterized in that The method of finding the largest inscribed rectangular-like dot matrix among all valid circular points based on the centroid of four vertices includes: According to the coordinates of the upper left corner centroid and the upper right corner centroid, find the centroid of all valid circles in the first row of the largest inscribed rectangular lattice; Starting from each centroid in the first row, set the jth row as the anchor point, search for the centroid closest to the anchor point in the j+1th row, and determine whether the centroid closest to the anchor point is in the same row as the lower left corner centroid or the lower right corner centroid; When they are not in the same row, the centroid is set as the new anchor point; When they are in the same row, the search ends and the current anchor point is recorded. At this time, all the centroids located in the maximum inscribed class rectangle have been found. All the found centroids are recorded as class rectangle centroids, and the dot matrix formed by the valid circular points where the class rectangle centroids are located is recorded as the maximum inscribed class rectangle dot matrix.

7. The method according to claim 1, characterized in that The step of generating an ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular dot matrix and the width and height of the calibration plate image includes: Get the width and height of the calibration plate image; Determine the width and height of the ideal rectangular dot matrix according to the width and height of the largest inscribed rectangular-like dot matrix; According to the width and height of the largest inscribed rectangular lattice and the width and height of the ideal rectangular lattice, coordinate system conversion is performed to obtain all coordinates of the ideal rectangular lattice; An ideal rectangular lattice is established according to all the coordinates of the obtained ideal rectangular lattice.

8. The method according to claim 1, characterized in that The step of calculating the homography matrix based on the original rectangular lattice and the ideal rectangular lattice includes: Obtain the rotation angle and rotation center of the original rectangular lattice, and construct a rotation matrix according to the rotation angle, and rotate the rotated lattice based on the rotation matrix to rotate the maximum inscribed rectangular lattice back to the original angle; The homography matrix is ​​calculated based on the ideal rectangular lattice and the rotated maximum inscribed rectangular lattice.

9. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer program instructions executable by the processor, and when the computer program instructions are called by the processor, the processor executes the camera field of view correction method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions executable by a processor, and when the computer program instructions are called by the processor, the processor executes the camera field of view correction method according to any one of claims 1 to 8.

Images