A high-precision image processing device, method and system

Abstract

The application discloses a high-precision image processing device, method and system, and relates to the technical field of image processing.The image processing device comprises a mirror, which is distributed in front of a photosensitive area of an image sensor and is used for reflecting imaging light so as to realize pixel displacement; an auxiliary light source, which is distributed on a light incident side close to the mirror; a monitoring sensor, which is distributed on a light emission side close to the mirror and is used for collecting a laser beam passing through the mirror; and a processor, which is connected with the image sensor and the monitoring sensor respectively and is used for analyzing a deviation between an incident position of a current laser beam on a surface of the monitoring sensor and a reference point, so as to adjust the mirror.The application not only can realize true color detection effect, but also can accurately realize pixel displacement when facing a high-precision detection scene, so that detection deviation of each component is avoided, and high-precision and high-frame-rate detection requirements are met while reliability is met.

Description

Technical Field

[0001] This invention belongs to the field of machine vision, and particularly relates to a high-precision image processing device, method and system. Background Technology

[0002] High-resolution area scan cameras are currently widely used in applications requiring high precision and efficiency, such as mobile phone screen inspection and panel inspection. However, area scan cameras only use sensors in two formats: monochrome and Bayer. Bayer format sensors, regardless of pixel arrangement, cannot cover the entire scene in a single color. Furthermore, the image resolution after Bayer-to-RGB conversion cannot meet certain high-precision inspection requirements. Therefore, Bayer format cameras are rarely used in area scan inspection, and monochrome cameras are primarily used. However, monochrome cameras, due to their single-band limitations, are inadequate for certain inspection scenarios requiring multiple bands.

[0003] In existing technologies, for detection scenarios requiring multiple wavelengths, Bayer format cameras utilize displacement driving devices such as piezoelectric ceramics to move the image sensor, thereby achieving multiple sampling at a fixed physical location. By designing an XY two-dimensional displacement driving device combined with a standard Bayer format sensor, it is possible to acquire the R, G, and B components from the same physical location, thus realizing a high-resolution true-color camera with RGB three wavelengths, enabling true-color detection for screens such as mobile phones, LEDs, and OLEDs.

[0004] However, image sensors are precision electronic components, and long-term displacement control can affect their accuracy and reliability. Moreover, actual displacement control requires simultaneous movement control of the image sensor and its associated PCB board and connectors, which can affect the accuracy and speed of actual displacement control, especially limiting its use in high frame rate scenarios.

[0005] Therefore, in order to meet the requirements of high accuracy and high frame rate detection while satisfying reliability, the present invention provides a high-precision image processing device, method and system. Summary of the Invention

[0006] The purpose of this invention is to overcome the above-mentioned problems existing in the prior art and provide a high-precision image processing device, method and system. An auxiliary light source is introduced to emit invisible light, which passes through a reflector together with the main optical path and is identified by a monitoring sensor. The monitoring sensor identifies the coordinate position of the auxiliary light source beam and obtains the actual offset data according to the pre-completed calibration results. Thus, the control angle of the reflector is precisely adjusted by combining the control angle and the offset.

[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0008] A high-precision image processing device is used to monitor the displacement amount in pixel displacement so that the displacement amount meets accuracy requirements. The image processing device includes:

[0009] Reflectors, located in front of the photosensitive area of ​​the image sensor, are used to reflect the imaging light to change the incident position of the imaging light on the image sensor, thereby achieving pixel displacement.

[0010] An auxiliary light source, located near the light incident side of the reflector, is used to emit a laser beam so that the laser beam and the imaging light pass through the reflector simultaneously.

[0011] The monitoring sensors are distributed on the light-emitting side near the reflector to collect the laser beam passing through the reflector. When the imaging light is deflected to the preset position of the image sensor, the incident position of the laser beam on the surface of the monitoring sensor is obtained as a reference point.

[0012] The processor, connected to the image sensor and the monitoring sensor respectively, is used to analyze the deviation between the incident position of the current laser beam on the surface of the monitoring sensor and the reference point, and then adjust the reflector so that the deviation meets the deviation threshold range, thereby retaining the image data acquired by the corresponding image sensor after the reflector is adjusted.

[0013] Furthermore, it also includes:

[0014] The first beam splitter is located on the light incident side near the reflector, so that the laser beam and the imaging beam pass through the first beam splitter and reach the reflector at the same time.

[0015] The second beam splitter is located on the light-emitting side near the reflector, allowing the laser beam to pass through the second beam splitter to reach the monitoring sensor, and the imaging beam to pass through the second beam splitter to reach the image sensor.

[0016] Furthermore, the processing method for deflecting the imaging rays to a preset position on the image sensor includes:

[0017] Adjust the reflection angle of the mirror to acquire image data formed after the imaging light rays of the calibration object are deflected on the surface of the image sensor;

[0018] Calculate the offset between the centroid coordinates of the image data formed after the imaging rays of the calibration object are offset on the surface of the image sensor and the original centroid coordinates;

[0019] Determine whether the offset is equal to a preset value: if yes, it means that the image information of the calibrator has shifted to the preset position of the image sensor; if no, continue to adjust the reflection angle of the reflector until the centroid offset is equal to the preset value.

[0020] The original centroid coordinates are the centroid coordinates corresponding to the image data of the calibration object before the mirror was adjusted.

[0021] Furthermore, the methods for analyzing reference points include:

[0022] The image of the light spot formed on the surface of the monitoring sensor by the laser beam emitted by the auxiliary light source is acquired;

[0023] The average value of each row of data in the spot image is calculated, and a weighted average is calculated based on the position of each row of data to obtain the centroid position of the spot image, which is used as a reference point.

[0024] The data in each row is distributed vertically along the direction of the offset of the image information of the calibration object.

[0025] The present invention also provides a high-precision image processing method for monitoring the displacement amount in pixel displacement to ensure that the displacement amount meets accuracy requirements. The image processing method includes:

[0026] The pixel displacement is achieved by controlling the rotation of the reflector, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0027] When the imaging ray deviates to the preset position of the image sensor, the incident position of the laser beam on the surface of the monitoring sensor is used as a reference point to obtain the amount of deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point.

[0028] Adjust the reflector so that the deviation meets the deviation threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0029] Furthermore, when the deviation amount does not meet the deviation amount threshold range: the reflection angle of the reflector is adjusted according to the preset angle change amount to reduce the absolute value of the deviation amount, and when the positive or negative value of the deviation amount changes, the reflection angle of the reflector is adjusted by half of the current angle change amount until the deviation amount meets the deviation amount threshold range.

[0030] Furthermore, when the deviation amount does not meet the deviation threshold range, the control angle of the reflector is adjusted for the first time based on the angle adjustment model; wherein, the method for constructing the angle adjustment model includes:

[0031] A mapping model between angle adjustment amount, deviation amount, and temperature was constructed, and angle adjustment amounts under different deviation amounts and temperatures were collected to fit the parameters of the angle adjustment model.

[0032] Furthermore, the imaging light is deflected to the preset position of the image sensor by: adjusting the reflection angle of the reflector four times, so that the position of each pixel of the image information of the calibrator forms a square ring moving path on the image sensor, moving a single pixel size each time;

[0033] The RGB data of each pixel position on the surface of the calibration object are known; the image sensor is based on a Bayer array.

[0034] The present invention also provides a high-precision image processing method for monitoring the displacement amount in pixel displacement to ensure that the displacement amount meets accuracy requirements. The image processing method includes:

[0035] The pixel displacement is achieved by controlling the rotation of the reflector, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0036] The vector difference exists between the displacement of the laser beam at the incident position on the surface of the monitoring sensor before and after the imaging light beam deflects to the preset position of the image sensor, and the actual displacement of the laser beam at the incident position on the surface of the monitoring sensor.

[0037] Adjust the reflector so that the vector difference meets the preset threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0038] The present invention also provides a high-precision image processing system, comprising:

[0039] The displacement control module is used to control the rotation of the reflector to achieve pixel displacement, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0040] The acquisition and analysis module is used to analyze the incident position of the laser beam on the surface of the monitoring sensor when the imaging light beam deviates to the preset position of the image sensor, using it as a reference point, so as to obtain the amount of deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point.

[0041] The adjustment control module is used to adjust the reflector so that the deviation amount meets the deviation threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0042] The beneficial effects of this invention are:

[0043] (1) This invention can not only achieve true color detection effect on the basis of traditional Bayer-format area array image sensor, but also accurately realize pixel displacement when facing high-precision detection scenarios, thereby avoiding detection deviation of each component and meeting the requirements of high precision and high frame rate detection while meeting reliability.

[0044] (2) By introducing a reflector to reflect image information, compared with traditional moving image sensors, controlling the rotation of the reflector not only avoids the influence on the image sensor, but also makes it smaller and lighter. The stability, reliability and accuracy of control are effectively guaranteed. While achieving precise displacement of image information on the image sensor, it has a high response speed and can work with the image sensor to achieve a high frame rate.

[0045] (3) By setting a first beam splitter and a second beam splitter in front of and behind the reflector respectively, the image information passes through the first beam splitter, the reflector and the second beam splitter in sequence to reach the image sensor. The laser beam emitted by the auxiliary light source passes through the first beam splitter, the reflector and the second beam splitter in sequence to reach the monitoring sensor. This ensures that when there is a displacement deviation of the image information on the image sensor, the laser beam emitted by the auxiliary light source has the same deviation on the monitoring sensor. This allows the actual detection result of the image sensor to be monitored in real time by the monitoring sensor, avoiding the influence of the actual environment on the control accuracy of the reflector.

[0046] By introducing a calibration object, if the image information detected by the image sensor is precisely offset according to a preset position, the ideal data result detected by the image sensor can be obtained based on data processing. By comparing and analyzing the actual data result detected by the image sensor with the ideal data result, if they are consistent, it means that the image information of the calibration object has been offset to the preset position of the image sensor. In actual operation, a certain threshold range can also be set for the allowable error range. That is, if the error between the actual data result and the ideal data result meets the threshold range, it meets the preset position requirement; otherwise, the reflection angle of the reflector is adjusted further. By using the incident position of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor as a reference point, the deviation synchronization brought by the first beam splitter and the second beam splitter is reasonably utilized. That is, the actual detection result of the image sensor can be monitored in real time by the monitoring sensor. By setting the reference point, when there is a deviation of the reflector, the incident position of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor will deviate from the reference point, thus accurately reflecting the deviation problem. Furthermore, the reference point obtained by laser beam detection based on auxiliary light source has its own characteristics. When conducting actual detection on the object under test, it is not necessary to analyze the complex image information of the object under test. Only the incident position of the beam needs to be analyzed. This not only reduces the amount of data processing and increases efficiency, but also provides better stability in beam position detection and stronger reliability in deviation calculation compared to complex image processing.

[0047] By statistically analyzing the deviation of the incident position of the laser beam emitted by the current auxiliary light source on the surface of the monitoring sensor relative to the reference point, the analysis results can be obtained accurately, quickly, and stably. By determining whether the deviation meets the deviation threshold range, it is possible to quickly and reasonably decide whether to retain the image information of the current object under test. Even if there is a deviation, it is possible to correct the error in time by adjusting the reflection angle of the reflector. With a fast response, it can not only achieve high-precision detection when dealing with complex environments, but also ensure detection efficiency, making it suitable for detection scenarios of high frame rate image sensors. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 This is a schematic diagram of pixel displacement in a local image of Bayer format in this invention;

[0050] Figure 2 This is a schematic diagram of the image processing device structure in this invention;

[0051] Figure 3 This is a flowchart of the image processing method in this invention;

[0052] Figure 4 This is a schematic diagram of the four-fold displacement of image information in this invention;

[0053] Figure 5 In this invention Figure 4 A schematic diagram after assembly;

[0054] Figure 6 In this invention Figure 5 Diagram showing the removal of invalid logic rows;

[0055] Figure 7 In this invention Figure 6 A schematic diagram showing the result after removing invalid pixels at the beginning of the row;

[0056] Figure 8 It is based on the present invention Figure 7 A schematic diagram of the actual output order;

[0057] Figure 9 This is a block diagram of the image processing system structure in this invention.

[0058] In the diagram: 1-Reflector; 2-Auxiliary light source; 3-First beam splitter; 4-Second beam splitter; 5-Image sensor; 6-Monitoring sensor; 7-Processor. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Since most current area array color image sensors use Bayer format data, a displacement driving device can be used to displace the target object or image sensor in the XY two-dimensional direction, enabling multiple samplings of the same physical location, thereby achieving RGB three-component true color imaging. The basic principle is as follows: Figure 1 As shown, the RGGB Bayer format will be used as an example for explanation. Figure 1 (a) shows a schematic diagram of the original pixel positions. Figure 1 (b) shows the pixel position after shifting one pixel to the right. Figure 1 (c) shows the pixel position after moving down one pixel. Figure 1 (d) is a diagram illustrating the pixel position after shifting one pixel to the left. Figure 1 Based on position (d), move upwards by one pixel to zero, returning to the original position shown in (a), completing one movement cycle. It can be seen that for the physical positions shown in the dashed boxes (containing 4 pixels), each physical position is sampled 4 times. Taking the top left position as an example, the components obtained from the 4 samples are R, Gr, B, and Gb. In fact, the 4 sampling results for each physical position all have four components: R / Gr / Gb / B, only the sampling order and timing are slightly different.

[0061] However, image sensors are precision electronic components, and long-term displacement control can affect their accuracy and reliability. Moreover, actual displacement control requires simultaneous movement control of the image sensor and its associated PCB board and connectors, which can affect the accuracy and speed of actual displacement control, especially limiting its use in high frame rate scenarios.

[0062] To meet the requirements of high accuracy and high frame rate detection while maintaining reliability, such as Figure 2 As shown, the present invention provides a high-precision image processing device for monitoring the displacement amount in pixel displacement to ensure that the displacement amount meets accuracy requirements. The image processing device includes:

[0063] The reflector 1 is located in front of the photosensitive area of ​​the image sensor 5 and is used to reflect the imaging light to change the incident position of the imaging light on the image sensor 5, thereby achieving pixel displacement.

[0064] The auxiliary light source 2 is located on the light incident side near the reflector 1 and is used to emit a laser beam so that the laser beam and the imaging light pass through the reflector 1 simultaneously.

[0065] Monitoring sensor 6 is located on the light-emitting side near the reflector 1. It is used to collect the laser beam passing through the reflector 1 so that when the imaging light is deflected to the preset position of the image sensor 5, the incident position of the laser beam on the surface of monitoring sensor 6 is obtained as a reference point.

[0066] To further ensure the synchronization of the laser beam and the imaging beam, a first beam splitter 3 and a second beam splitter 4 are introduced. The first beam splitter 3 is located near the light incident side of the reflector, allowing the laser beam and the imaging beam to pass through the first beam splitter 3 simultaneously to reach the reflector 1. The second beam splitter 4 is located near the light emitting side of the reflector 1, allowing the laser beam to pass through the second beam splitter 4 to reach the monitoring sensor 6, and the imaging beam to pass through the second beam splitter 4 to reach the image sensor 5. The laser beam emitted by the auxiliary light source 2 passes through the first beam splitter 3 and the reflector 1 together with the imaging beam to reach the second beam splitter 4; wherein, the imaging beam passes through the second beam splitter 4 to reach the image sensor 5, and the laser beam emitted by the auxiliary light source passes through the second beam splitter 4 to reach the monitoring sensor 6.

[0067] The processor 7 is connected to the image sensor 5 and the monitoring sensor 6 respectively. It is used to analyze the deviation between the incident position of the current laser beam on the surface of the monitoring sensor and the reference point, and then adjust the reflector so that the deviation meets the deviation threshold range, thereby retaining the image data collected by the corresponding image sensor after the reflector is adjusted.

[0068] like Figure 3 As shown, to illustrate the above-described image processing apparatus, the present invention also provides a high-precision image processing method, the image processing method comprising:

[0069] The pixel displacement is achieved by controlling the rotation of the reflector, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0070] First, a reflector is introduced to reflect the image information. By controlling the rotation of the reflector, the reflected imaging light is displaced at the sensing position on the image sensor. Compared with traditional moving image sensors, controlling the rotation of the reflector not only avoids affecting the image sensor, but also makes it smaller and lighter. The stability, reliability, and control accuracy are all effectively guaranteed. While achieving precise displacement of image information on the image sensor, it also has a high response speed and can work with the image sensor to achieve a high frame rate.

[0071] Considering that the reflector may exhibit slight displacement deviations at the same control angle under complex environments such as varying temperatures, an auxiliary light source is introduced to ensure control accuracy in real-world conditions. This auxiliary light source uses invisible light, such as infrared light, to prevent a portion of its emitted beam from interfering with the image information as it passes through the second beam splitter to reach the image sensor. This ensures that only a monitoring sensor, such as an infrared sensor, can detect the laser beam emitted by the auxiliary light source. For example, by setting a first beam splitter and a second beam splitter on the light incident and light exit sides of the reflector respectively, the image information sequentially passes through the first beam splitter, the reflector, and the second beam splitter to reach the image sensor. The laser beam emitted by the auxiliary light source sequentially passes through the first beam splitter, the reflector, and the second beam splitter to reach the monitoring sensor. This ensures that when there is a displacement deviation in the image information on the image sensor, the laser beam emitted by the auxiliary light source has the same deviation on the monitoring sensor. This allows the actual detection result of the image sensor to be monitored in real-time by the monitoring sensor, avoiding the influence of the actual environment on the control accuracy of the reflector.

[0072] When the imaging ray deviates to a preset position on the image sensor, the incident position of the laser beam on the surface of the monitoring sensor is used as a reference point to obtain the deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point.

[0073] Adjust the reflector so that the deviation meets the deviation threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0074] Image data such as RGB information for each pixel position on the calibration object surface are known in advance. By introducing the calibration object, if the image information detected by the image sensor is precisely offset according to the preset position, the ideal data result detected by the image sensor can be obtained based on data processing. By comparing and analyzing the actual data result detected by the image sensor with the ideal data result, if they are consistent, it means that the imaging light of the calibration object has shifted to the preset position of the image sensor. In actual operation, a certain threshold range can also be set for the allowable error range. That is, if the error between the actual data result and the ideal data result meets the threshold range, it meets the preset position requirement; otherwise, the reflection angle of the reflector is adjusted further. By using the incident position of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor as the reference point, the deviation synchronization brought by the first beam splitter and the second beam splitter can also be reasonably utilized. That is, the actual detection result of the image sensor can be monitored in real time by the monitoring sensor. By setting the reference point, when there is a deviation of the reflector, the incident position of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor will deviate from the reference point, thus accurately reflecting the deviation problem. Furthermore, the reference point obtained by laser beam detection based on the auxiliary light source, due to its inherent characteristics, eliminates the need for analysis of complex object image information during subsequent actual detection. Only the incident position of the beam needs analysis, resulting in less data processing, higher efficiency, and better stability in beam position detection compared to complex image processing, leading to more reliable deviation calculations. In practical applications, the specific number of adjustments to the reflection angle of the reflector can be determined based on the image sensor's detection results. The preset position refers to the position of the image sensor after displacement relative to the calibration object. For example... Figure 1 The scenario shown, where the movement is cyclical (right, down, left, up), means that for the four different preset positions, four different reference points will be obtained.

[0075] When actually acquiring image information of the object to be measured, Figure 1 Taking a scenario with a periodic shift of right, down, left, and up as an example, at least four frames of image information need to be captured. Whether each frame meets the accuracy requirements needs to be determined. Compared to complex image information analysis and judgment, by statistically analyzing the deviation of the incident position of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor relative to the reference point, accurate, fast, and stable analysis results can be obtained. By judging whether this deviation meets the deviation threshold range, a quick and reasonable decision can be made on whether to retain the image information of the current object under test. Even if there is a deviation, it can be corrected in time by adjusting the reflection angle of the reflector, providing a rapid response. In complex environments, it not only achieves high-precision detection but also ensures detection efficiency, making it suitable for detection scenarios using high-frame-rate image sensors.

[0076] To deflect the imaging rays to a preset position on the image sensor, specific processing methods include:

[0077] The reflection angle of the reflector is adjusted to acquire image data formed after the imaging light rays from the calibration object are deflected on the surface of the image sensor. To accurately analyze the image information deflection results of the calibration object, all feature point data on the calibration object surface can be obtained in advance. The original positions before deflection can be known through the feature point data, and comparative analysis can be performed by combining the feature point data after deflection.

[0078] Calculate the offset between the centroid coordinates of the image data formed after the imaging rays of the calibration object are offset on the image sensor surface and the original centroid coordinates. Based on the feature point data of the calibration object before and after the offset, calculate the original centroid coordinates and the current centroid coordinates, where the original centroid coordinates (C... x0 C y0 The calculation formula is as follows:

[0079]

[0080] Where m_0 represents the total number of pixel rows in the feature point data of the calibrator before the image information is offset, n_0 represents the total number of pixel columns in the feature point data of the calibrator before the image information is offset, i_0 represents the pixel row number in the feature point data of the calibrator before the image information is offset, j_0 represents the pixel column number in the feature point data of the calibrator before the image information is offset, and f0(i_0,j_0) represents the grayscale value of the pixel in the i_0th row and j_0th column in the feature point data of the calibrator before the image information is offset.

[0081] Current centroid coordinates (C x1 C y1 The calculation formula is as follows:

[0082]

[0083] Where m_1 represents the total number of pixel rows in the feature point data after the image information of the calibration object is offset, n_1 represents the total number of pixel columns in the feature point data after the image information of the calibration object is offset, i_1 represents the pixel row number in the feature point data after the image information of the calibration object is offset, j_1 represents the pixel column number in the feature point data after the image information of the calibration object is offset, and f1(i_1,j_1) represents the gray value of the pixel in the i_1 row and j_1 column in the feature point data after the image information of the calibration object is offset.

[0084] Determine whether the offset is equal to a preset value: if yes, it means that the image information of the calibrator has shifted to the preset position of the image sensor; if no, continue to adjust the reflection angle of the reflector until the centroid offset is equal to the preset value.

[0085] The original centroid coordinates are the centroid coordinates of the feature points of the calibration object before the mirror was adjusted.

[0086] The formula for calculating the centroid offset d is as follows:

[0087]

[0088] The centroid offset can usually be known in advance. Figure 1 Taking the scenario shown as an example where the movement is periodic (rightward, downward, leftward, and upward), each movement is one pixel in size, meaning the preset value corresponding to each centroid offset is 1.

[0089] To enable fast and accurate calculation of reference points, reference point analysis methods include:

[0090] The laser beam emitted by the auxiliary light source forms a spot image on the surface of the monitoring sensor. The laser beam emitted by the auxiliary light source passes through the first beam splitter, the reflector, and the second beam splitter in sequence before reaching the monitoring sensor and forming a spot image. The position of the spot image at this time corresponds to the image information of the calibration object shifting to the preset position of the image sensor. This means that at any time, as long as the image information of the calibration object shifts to the preset position of the image sensor, the corresponding spot image information can be detected at the position of the spot image at this time.

[0091] The average value of each row of data in the spot image is calculated, and a weighted average is calculated based on the position of each row of data to obtain the centroid position of the spot image, which is used as a reference point.

[0092] The data in each row is distributed vertically along the direction of the offset of the image information of the calibration object.

[0093] Since the data in each row is distributed perpendicular to the offset direction of the image information of the calibration object, we only need to focus on the coordinate position along the offset direction of the image information of the calibration object, i.e., the reference point position. The formula for calculating the reference point position Q0 is as follows:

[0094]

[0095] Where M_0 represents the total number of pixel rows in the spot image corresponding to the reference point, and u_0 represents the ordinal number of the pixel row in the spot image corresponding to the reference point. This represents the average gray value of the u_0 row in the spot image corresponding to the reference point.

[0096] Similarly, when acquiring image information of the object under test, the incident position Q1 of the laser beam emitted by the current auxiliary light source on the surface of the monitoring sensor can be calculated, and the deviation ΔQ relative to the reference point is ΔQ = Q1 - Q0.

[0097] The deviation threshold range is [-ρ, ρ], where ρ is a preset parameter used in the calculation. It can be adjusted according to the actual accuracy requirements. The smaller the value, the higher the accuracy, thus determining whether ΔQ meets the deviation threshold range.

[0098] As can be seen from the above, when the deviation amount does not meet the deviation amount threshold range, the reflection angle of the reflector needs to be adjusted until the deviation amount meets the deviation amount threshold range. In order to achieve the adjustment accurately and efficiently, the specific operation method is as follows: adjust the reflection angle of the reflector according to the preset angle change amount to reduce the absolute value of the deviation amount, and when the positive or negative value of the deviation amount changes, adjust the reflection angle of the reflector by half of the current angle change amount until the deviation amount meets the deviation amount threshold range.

[0099] To improve the efficiency of angle adjustment, the preset angle change is usually too large. This allows the deviation to quickly approach the deviation threshold range after adjustment. For example, if the deviation is positive at the beginning of the adjustment, it will change to negative after several adjustments. At this point, to further improve the adjustment efficiency, the angle change for the next adjustment can be reduced to half of the current angle change. This process is repeated to quickly bring the offset to the deviation threshold range.

[0100] To ensure the adjustment efficiency of the reflector, the influence of the actual ambient temperature can usually be ignored during the initial adjustment. However, when the influence of the actual ambient temperature is significant, adjustments can be made based on a pre-calibrated model. That is, when the deviation does not meet the deviation threshold range, the control angle of the reflector is adjusted for the first time based on the angle adjustment model. The method for constructing the angle adjustment model includes:

[0101] A mapping model between angle adjustment amount, deviation amount, and temperature was constructed, and angle adjustment amounts under different deviation amounts and temperatures were collected to fit the parameters of the angle adjustment model.

[0102] Specifically, the mapping model between the angle adjustment amount, deviation amount, and temperature is constructed as follows:

[0103]

[0104] Where Δθ is the angle adjustment amount, T is the ambient temperature, ΔQ is the deviation amount, and α1, α2, α3, α4, α5, and α6 are all fitting parameters involved in the calculation. By collecting the angle adjustment amount Δθ of the reflector at different temperatures T and deviation amounts ΔQ, several sets of statistical data are obtained. Substituting these data into the mapping model allows for the analysis and acquisition of all fitting parameters. To ensure the reliability of the fitting parameters, the range of temperature T and the range of deviation amounts ΔQ are designed to cover the temperature fluctuation range that the reflector may encounter during actual operation as much as possible.

[0105] To ensure that the image sensor can still stably obtain true color effects based on the Bayer format, the imaging light is shifted to a preset position on the image sensor by: adjusting the reflection angle of the mirror four times, so that the pixel positions of the image information of the calibration object form a square ring moving path on the image sensor, moving a single pixel size each time; wherein, the RGB data of each pixel position on the surface of the calibration object are known; the image sensor is based on the Bayer array.

[0106] By moving a single pixel size each time to form a square ring movement path, any pixel position in the image information of the calibration object logically corresponds to the four pixels RGGB on the image sensor. Since the RGB data of each pixel position on the surface of the calibration object is known, the two data are compared one by one. If they are consistent, it means that each adjustment of the control angle of the reflector has accurately shifted the image information of the calibration object to the preset position of the image sensor. Otherwise, further correction is required.

[0107] As can be seen from the above, the present invention can precisely control and adjust the reflector as needed, so that the image information of the object under test can be precisely shifted on the image sensor according to a preset position. This not only enables true color effects on Bayer-format image sensors, but also allows for sub-pixel displacement in the XY direction to achieve super-resolution effects. The following will describe in detail how the Bayer-format image sensor achieves true color effects. The image processing method further includes: when acquiring the image information of the object under test, extracting the image data after each movement and aligning the data to obtain different RGB data at the same physical location in the image information of the object under test.

[0108] like Figure 4 As shown, the reflector is adjusted four times, and the image sensor's image information relative to the object being measured is sequentially shifted four times. The shift directions are: origin → right → down → left → up (back to origin). The image sensor output is arranged as follows: the first active line is row RG, the second line is row GB, and so on. (Regarding...) Figure 4 In certain situations, the relationship between logical rows and physical rows needs to be handled carefully. The first row and first column of the image require special processing because there may be missing sampling points in the first row and first column. Figure 4 Pixels that are not filled with shadow are considered non-existent and belong to logical rows, while the image sensor only outputs physical rows. Figure 4 The pixels in the image have already been shaded. Without modifying the current stitching module, four rows of data from four different positions are read from DDR each time, then stitched together into two rows (doubling the row resolution) for output. The result is as follows. Figure 5 As shown. Figure 6The image shown is a diagram illustrating the removal of invalid logical rows (i.e., removing all pixels in the image that have not been shaded). Figure 7 The image shown is a schematic diagram of the result after removing invalid pixels at the beginning of a row. The output order of the concatenated rows is shown by the arrows in the right-hand image. Figure 7 Based on this, the actual output order is marked to form Figure 8 The diagram shows the actual output order, with the pixel data corresponding to the same physical location within the dashed box.

[0109] like Figure 8 As shown, the first row of data needs to be processed separately (it only has RG components, no B components, while the first row and first column only have R components, no GB components). The second and third rows of data are grouped together (corresponding to rows with the same physical location), the fourth and fifth rows are grouped together, and so on. That is, the 4N+2 row of data is grouped with the 4N+3 row, the 4N+4 row with the 4N+5 row, and so on, with the relationships within each group being reversed. For example, in the group containing the second and third rows of data, the second row is an RG row and the third row is a GB row, while in the group containing the fourth and fifth rows of data, the fourth row is a GB row and the fifth row is an RG row. This needs to be noted. Additionally, since the pixels in the first column only have RG components and no B components, padding is required. The last row also needs to be processed separately, as it only has GB components and no R components, while the first column of the last row only has G components and no RB components. If the first row is output, the last row does not need to be output; conversely, if the first row is discarded, the last row must be padded; otherwise, the row count will be one row less. Following the above operation method, by using DDR to cache images, the RGB true color image output of a two-dimensional scene can be completed within the FPGA. Depending on the needs of the terminal scene, the RGB three-part separation can also be performed within the FPGA to achieve independent output of the three corresponding R / G / B images.

[0110] like Figure 4-8 As shown, a multispectral (RGB) camera implementation method was designed based on hardware (FPGA) and using pixel displacement technology. By adjusting the reflector, pixel displacement in the XY two-dimensional direction was achieved, enabling four samplings of the same physical location and obtaining RGB data components. The FPGA / DDR was used to process and stitch the multiple acquired images in real time according to the pixel displacement order and direction, realizing the output of RGB true-color image (single true-color image) or R / G / B three-spectral component image (three images). This greatly reduced the processing complexity of the PC software and provided a multispectral implementation method for various screen defect detection and high-precision detection scenarios in the present and future.

[0111] As another embodiment of the image processing method of the present invention, it can also be implemented through the following steps:

[0112] The pixel displacement is achieved by controlling the rotation of the reflector, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0113] The vector difference exists between the displacement of the laser beam at the incident position on the surface of the monitoring sensor before and after the imaging light beam deflects to the preset position of the image sensor, and the actual displacement of the laser beam at the incident position on the surface of the monitoring sensor.

[0114] Adjust the reflector so that the vector difference meets the preset threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0115] Compared with the embodiments in the above image processing method, this embodiment differs in that it introduces the concept of displacement vector. Before the imaging light is deflected to the preset position of the image sensor, the incident position of the laser beam on the surface of the monitoring sensor is the initial point. After the imaging light is deflected to the preset position of the image sensor, it corresponds to the aforementioned reference point. The displacement formed between the initial point and the reference point is the ideal displacement. In the actual detection process, after the reflector reflects the imaging light, the current incident position of the laser beam on the surface of the monitoring sensor will form an actual displacement with the initial point. By comparing the magnitude of the vector difference between the actual displacement and the ideal displacement, the reflector is adjusted accordingly so that the magnitude of the vector difference meets the preset threshold range. The preset threshold range is the same as the aforementioned deviation threshold range. Therefore, although this embodiment utilizes the displacement vector comparison method compared with the embodiments of the above image processing method, it essentially focuses on the deviation between the actual incident position and the ideal incident position of the laser beam on the surface of the monitoring sensor before and after pixel displacement.

[0116] like Figure 9 As shown, the present invention also provides a high-precision image processing system, comprising:

[0117] The displacement control module is used to control the rotation of the reflector to achieve pixel displacement, so that the imaging light and the laser beam pass through the reflector simultaneously and reach the image sensor and the monitoring sensor respectively.

[0118] The acquisition and analysis module is used to analyze the incident position of the laser beam on the surface of the monitoring sensor when the imaging light beam deviates to the preset position of the image sensor, using it as a reference point, so as to obtain the deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point.

[0119] The adjustment control module is used to adjust the reflector so that the deviation amount meets the deviation threshold range, thereby preserving the image information acquired by the corresponding image sensor after the reflector is adjusted.

[0120] For detailed operation methods and principles of each module in the image processing system, please refer to the image processing methods described above, and they will not be repeated here.

[0121] A fourth aspect of the present invention also provides a computer-readable storage medium including a computer program that, when executed by a processor, implements the above-described image processing method.

[0122] In practical applications, a computer-readable storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0123] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0124] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0125] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A high-precision image processing device for monitoring the displacement amount in pixel displacement to ensure that the displacement amount meets accuracy requirements, characterized in that, The image processing device comprises: a mirror distributed in front of a light-sensitive area of an image sensor, used for reflecting imaging light to change the incident position of the imaging light on the image sensor, thereby realizing pixel shift; an auxiliary light source distributed on the light incident side close to the mirror, used for emitting a laser beam so that the laser beam and the imaging light pass through the mirror at the same time; a monitoring sensor distributed on the light exit side close to the mirror, used for collecting the laser beam passing through the mirror, so that when the imaging light is offset to a preset position of the image sensor, the incident position of the laser beam on the surface of the monitoring sensor is obtained as a reference point; a processor connected to the image sensor and the monitoring sensor respectively, used for analyzing the deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point, thereby adjusting the mirror so that the deviation meets the deviation threshold interval, thereby retaining the image data collected by the image sensor after adjusting the mirror.

2. The high-precision image processing apparatus according to claim 1, wherein Further comprising: a first light splitting prism distributed on the light incident side close to the mirror, so that the laser beam and the imaging light pass through the first light splitting prism to reach the mirror at the same time; a second light splitting prism distributed on the light exit side close to the mirror, so that the laser beam passes through the second light splitting prism to reach the monitoring sensor, and the imaging light passes through the second light splitting prism to reach the image sensor.

3. The high-precision image processing apparatus according to claim 2, wherein The processing method for offsetting the imaging light to the preset position of the image sensor comprises: adjusting the reflection angle of the mirror to collect the image data formed by the imaging light of the calibration object after being offset on the surface of the image sensor; calculating the deviation between the centroid coordinates corresponding to the image data formed by the imaging light of the calibration object after being offset on the surface of the image sensor and the original centroid coordinates; judging whether the deviation is equal to a preset value: if yes, it indicates that the image information of the calibration object is offset to the preset position of the image sensor, and if not, the reflection angle of the mirror is continuously adjusted until the centroid deviation is equal to the preset value; wherein the original centroid coordinates are the centroid coordinates corresponding to the image data of the calibration object before adjusting the mirror.

4. The high-precision image processing apparatus according to claim 2, wherein The analysis method of the reference point comprises: collecting the light spot image of the laser beam emitted by the auxiliary light source on the surface of the monitoring sensor; calculating the average value of each row of data of the light spot image, and performing weighted average calculation with the position of each row of data, thereby obtaining the centroid position of the light spot image as the reference point; wherein each row of data is distributed along the vertical direction of the offset direction of the image information of the calibration object.

5. A high-precision image processing method for monitoring a displacement amount in pixel displacement so that the displacement amount meets a precision requirement, characterized by, The image processing method comprises: controlling the mirror to rotate to realize pixel shift, so that after the imaging light and the laser beam pass through the mirror at the same time, they reach the image sensor and the monitoring sensor respectively; analyzing the incident position of the laser beam on the surface of the monitoring sensor when the imaging light is offset to the preset position of the image sensor as the reference point, thereby obtaining the deviation between the current incident position of the laser beam on the surface of the monitoring sensor and the reference point; adjusting the mirror so that the deviation meets the deviation threshold interval, thereby retaining the image information collected by the image sensor after adjusting the mirror.

6. The high-precision image processing method according to claim 5, characterized in that, When the deviation does not satisfy the deviation threshold interval, the reflection angle of the mirror is adjusted according to a preset angle change amount, so that the absolute value of the deviation is reduced, and when the positive and negative values of the deviation change, the reflection angle of the mirror is adjusted at half of the current angle change amount until the deviation satisfies the deviation threshold interval.

7. The high-precision image processing method of claim 5, wherein, When the deviation does not satisfy the deviation threshold interval, the control angle of the mirror is first adjusted based on an angle adjustment model; wherein the angle adjustment model construction method comprises: A mapping model between the angle adjustment amount and the deviation and the temperature is constructed, and the angle adjustment amount under different deviations and temperatures is collected to fit the parameters of the angle adjustment model.

8. The high-precision image processing method according to claim 6 or 7, characterized in that, The imaging light is offset to the preset position of the image sensor by adjusting the reflection angle of the mirror four times, so that the image information of the calibration object forms a square ring movement path on the image sensor, and each time a single pixel size is moved; Wherein, the RGB data of each pixel position on the surface of the calibration object is known; the image sensor is based on a Bayer array.

9. A high-precision image processing method for monitoring a displacement in pixel displacement so that the displacement meets a precision requirement, characterized in that, The image processing method comprises: Controlling the mirror to rotate to realize pixel displacement, so that the imaging light and the laser beam pass through the mirror at the same time and respectively reach the image sensor and the monitoring sensor; Analyzing the vector difference between the displacement formed by the incident position of the laser beam on the surface of the monitoring sensor before and after the imaging light is offset to the preset position of the image sensor, and the actual displacement formed by the incident position of the current laser beam on the surface of the monitoring sensor; Adjusting the mirror so that the size of the vector difference satisfies a preset threshold interval, thereby retaining the image information collected by the image sensor after adjusting the mirror.

10. A high-precision image processing system, characterized by comprising: Comprise: A displacement control module for controlling the mirror to rotate to realize pixel displacement, so that the imaging light and the laser beam pass through the mirror at the same time and respectively reach the image sensor and the monitoring sensor; A collection and analysis module for analyzing the incident position of the laser beam on the surface of the monitoring sensor when the imaging light is offset to the preset position of the image sensor, as a reference point, so as to obtain the deviation between the incident position of the current laser beam on the surface of the monitoring sensor and the reference point; An adjustment control module for adjusting the mirror so that the deviation satisfies a deviation threshold interval, thereby retaining the image information collected by the image sensor after adjusting the mirror.

Images