A method for real-time detection of crystal length and width during continuous crystallization using binocular backlight imaging

Abstract

This invention discloses a method for real-time detection of crystal length and width during continuous crystallization using binocular backlight imaging, belonging to the field of industrial process control and detection technology. First, a binocular backlight imaging system is designed, including two telecentric cameras, two telecentric backlight illuminators, and a customized four-degree-of-freedom optical platform, to acquire high-quality stereo images of flowing crystals in different regions of a continuous oscillating baffle crystallizer. Second, based on the in-situ captured binocular images of the crystal solution and their deep learning segmentation results, a crystal interest point matching algorithm is established. This algorithm uses telecentric epipolar lines and pixel intensity constraints to find the correspondence between interest points related to crystal length and width in each crystal image pair. Finally, an optimal three-dimensional point reconstruction method is proposed to reconstruct all matching crystal length-width interest point pairs, thereby quantitatively measuring the three-dimensional length and width of each crystal in the continuous crystallizer. This invention features novel and practical equipment with strong operability. Combined with stereo image analysis algorithms, it can achieve real-time automatic measurement of crystal three-dimensional dimensions.

Description

Technical Field

[0001] This invention belongs to the field of industrial process control and detection technology, and relates to in-situ image processing technology for industrial continuous crystallization processes. In particular, it relates to a non-invasive online monitoring system based on binocular backlight imaging, and a method for measuring the length and width of crystals based on deep learning stereo image analysis. Specifically, it refers to a method for measuring the three-dimensional length and width of crystal populations in a continuous oscillating baffle crystallizer in real time in situ using two high-definition industrial telecentric cameras, a telecentric light source, and an adjustable optical platform. Background Technology

[0002] The technology and control of continuous crystallization processes play a crucial role in the high-quality development of my country's advanced manufacturing sector. In continuous crystallization using a tubular continuous oscillating baffle crystallizer, online detection and control optimization of crystals at different time stages and locations help generate crystals with the desired consistent shape and size, thus ensuring the quality of the crystal particles. However, in the in-situ monitoring of continuous crystallization using computer vision, there is a lack of effective and convenient online process analysis equipment to accurately and intuitively characterize the true shape and size information of crystal populations in the solution. Furthermore, since the projected size of a crystal on the image plane depends on its three-dimensional position in the tubular crystallizer solution, extracting the length and width characteristic directions and interest point set of each crystal, and realizing the three-dimensional reconstruction of the crystal's length and width dimensions, is of great significance. Few domestic and international publications and patents describe widely applicable binocular real-time monitoring equipment for continuous crystallization processes, or standardized methods for online measurement of the length, width, and three-dimensional dimensions of crystals during growth.

[0003] Although a small number of devices for in-situ monitoring of continuous crystallizers have been developed, such as the application of inline imaging for monitoring crystallization process in a continuous oscillatory baffled crystallizer by Professor H. Kramer et al. of Delft University of Technology in the Netherlands, published in AIChE Journal, 2018, 64(7), 2450–2461, to invasively photograph melamine crystals in the continuous crystallization process and extract one-dimensional size information of the crystal population. For example, in their recent paper, "Continuous spherical crystallization of lysozyme in an oscillatory baffled crystallizer using emulsion solvent diffusion in droplets" (published in Crystal Growth & Design, 2020, 20, 934-947), Professor ZK Nagy et al. of Purdue University used a Blaze 900 camera probe to capture high-resolution images to characterize the spherical crystals of lysozyme in different types of continuous crystallizers. However, the above analytical methods all involve inserting the probe used for imaging into the crystallizer solution. Therefore, during the crystallization process, crystals may adhere to the probe surface and affect subsequent image acquisition. In addition, this type of invasive probe may affect the hydrodynamics during continuous crystallization, thereby altering the final product characteristics. Therefore, how to acquire monocular or even binocular images of crystals in situ during continuous crystallization, and extract the crystal length and width feature point set in real time and reconstruct its three-dimensional dimensions, remains a research and application challenge.In addition, as Professor ZGGao et al. of Tianjin University pointed out in the paper "Averified open-access AI-based chemical microparticle image database for in-situ particle visualization and quantification in multi-phase flow" (published in Chemical Engineering Journal, 2023, 451, 138940, a major international journal in the field of chemical engineering), deep learning-based image analysis strategies play a key role in the detection and quantification of crystallization processes, and can provide reliable and intuitive information on crystallization processes. The image segmentation network proposed by BWCheng et al. from Facebook AI Research in the paper "Masked-attention mask transformer for universal image segmentation" (abbreviated as Mask2Former; published in the top international computer vision conference IEEE Conference on ComputerVision and Pattern Recognition, 2022, 1280-1289) can accurately segment target objects. Therefore, it can be transferred to the segmentation and extraction of crystal targets in solution images, but further development of accurate and efficient in-situ real-time stereo image analysis methods based on deep learning is needed. Summary of the Invention

[0004] The technical problem this invention aims to solve is how to acquire images of crystals at different angles during the crystallization process in a continuous crystallizer in situ, and how to process these three-dimensional images in real time and measure the three-dimensional length and width dimensions of each crystal. To address these problems, a binocular backlight imaging system was designed to simultaneously acquire images of crystals in solution, along with a technical method for real-time processing and three-dimensional reconstruction of the corresponding crystallization images, in order to monitor the three-dimensional length and width information of crystals during continuous crystallization.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for real-time detection of crystal length and width during continuous crystallization based on binocular backlight imaging is proposed. This method is implemented using a designed non-contact high-definition binocular telecentric backlight image acquisition device, which includes two object-side telecentric cameras, two telecentric backlight illuminators, and a four-axis optical platform. First, two high-resolution object-side telecentric lenses are connected to a matching high-definition industrial CMOS camera to form a microscope camera lens system. Second, two LED light sources are connected to the other two telecentric lenses to form a telecentric light source that emits parallel beams for backlight illumination of the translucent crystal. Third, the camera lens system and the telecentric light source system are mounted on a long-stroke four-axis optical platform to simultaneously acquire binocular images (left and right views) from two angles at different locations outside the crystallizer. For the acquired binocular images, a deep learning-based stereo image analysis method is proposed, including crystal image segmentation and contour extraction, extraction and matching of crystal length and width feature directions and interest point sets. Finally, for each crystal length and width interest point matching set, a three-dimensional reconstruction of the crystal length and width interest point set is performed using an in-situ stereo calibration and correction model, thereby quantitatively evaluating its three-dimensional dimensions. This method includes the following steps:

[0007] The first step is the design of a binocular backlight imaging system.

[0008] (1) Utilizing the orthographic projection imaging characteristics of telecentric cameras, precise measurement of crystal targets at the micrometer scale is achieved. Specifically, firstly, based on the pixel size of the high-definition industrial camera and the sensor size, a matching object-side telecentric lens is selected, and attention must also be paid to selecting the interface method for both. After installing the two cameras and lenses, they are mounted on a four-degree-of-freedom micro-displacement stage, providing high-precision XYZ-axis translation and vertical rotation of the camera-lens system, thereby enabling the two camera-lens systems to focus on a common imaging area. In addition, the baselines of the two telecentric cameras should be parallel to the horizontal axis of the straight tube of the tubular continuous crystallizer, that is, the two cameras are placed horizontally, in order to minimize the optical refraction effect caused by the bending of the crystallizer glass tube wall during imaging.

[0009] (2) Two telecentric backlight illuminators are designed and placed on either side of the imaging target along with the binocular telecentric camera. This facilitates obtaining images with more obvious contrast, making it easier to further measure the size of the object being measured, especially for translucent and low-reflection crystal particles, such as glutamic acid crystals. Specifically, an LED light source is connected to the telecentric lens, allowing only uniform parallel light rays to pass through the tubular crystallizer and then into the telecentric camera for imaging. Both telecentric backlight illuminators are mounted in custom-designed light source brackets for easy movement and rotation, aligning the parallel beams with the optical axes of the two camera lenses respectively.

[0010] (3) Design a long-stroke four-axis optical platform for mounting the above-mentioned binocular telecentric camera, telecentric backlight illuminator and its fixing components. The optical platform moves the measurement area of ​​the binocular camera along the straight tube direction of the tubular continuous crystallizer by displacement along the X-axis; it moves the binocular camera approximately within its working distance by displacement along the first Y-axis; it then focuses the binocular camera by precise displacement along the second Y-axis so that its common focusing area can capture a clear crystal image; and it moves the binocular camera vertically by displacement along the Z-axis to monitor the straight tubes of different layers of the crystallizer.

[0011] In addition, before actual measurement, the simultaneously acquired micron-level checkerboard calibration image needs to be used, combined with the calibration and correction algorithm of the telecentric camera, to establish a stereo imaging model of this binocular backlight imaging system for three-dimensional reconstruction of crystal size.

[0012] The second step involves crystal image segmentation, extraction and matching of length and width interest point sets.

[0013] (1) Deep Learning-Based In-situ Crystal Image Segmentation and Contour Extraction: First, a large number of in-situ crystal images under different continuous crystallization experimental conditions, different detection areas, and different measurement times were collected. A continuous crystal image segmentation database was constructed using manual annotation. Second, using this database, a Masked-attention masktransformer (Mask2Former) instance segmentation network was trained to segment the crystal target of interest from the new in-situ images and extract its corresponding crystal contour. Third, some statistically insignificant crystal images, such as incomplete crystals intersecting the entire image boundary, were removed.

[0014] (2) Extraction of Crystal Length and Width Interest Points: First, the minimum bounding rectangle method is used to enclose the contour of each crystal image. Simultaneously, on the stereo-corrected binocular images, crystal image pairs are identified based on the coordinates of the horizontal epipolar line and the four corner points of the bounding rectangle. The length and width of a crystal image can be calculated by the angles between the long and short sides of its enclosing rectangle and the horizontal axis. Therefore, the two-dimensional feature length of the crystal image can be determined by any two points on its contour that are parallel to the length feature direction of the crystal and have the maximum distance. Similarly, its two-dimensional feature width can be quantized along the width feature direction of the crystal. Therefore, for the left image in a crystal image pair, its contour points are uniformly divided along its length and width feature directions to obtain multiple two-dimensional feature lines of length and width in the left image of the crystal. Then, the intersections of these lines with the left crystal contour can be classified into its length and width interest point set. and

[0015] To calculate the characteristic length and width of a crystal in three-dimensional space, it is necessary to find the set of left interest points. and Matching point set in the right image of the crystal and The following proposes a crystal interest point matching algorithm under two matching constraints.

[0016] ① Telecentric polar constraint:

[0017] by Taking a point of interest as an example, calculate the vertical distance d(i) between its epipolar line and the contour point of the right image of the crystal:

[0018] d(i)=|v Ri -v epipolar |,i=1,2,...,n / 2 (1)

[0019] Among them, v epipolar It is the vertical coordinate of the polar line; v Ri is the vertical coordinate of the right contour point; n represents the number of right contour points. When d(i) = 0, the right contour point is a possible matching point. Considering the possibility of image missegmentation, the right contour point and the other four points extending horizontally together form the initial matching set.

[0020] ②Pixel intensity constraint:

[0021] True matching point pairs in the initial matching set are found by analyzing the intensity differences between subsets in the left (reference) and right (target) images. The zero-mean normalized minimum squared distance (ZNSSD) matching criterion is employed to reduce the impact of brightness differences in the binocular images.

[0022]

[0023] in, r j and t j These are the intensity values ​​of the j-th pixel in the reference subset and the target subset, respectively; and These are the average intensity values ​​of all pixels in the reference subset and the target subset, respectively. Therefore, the χ² value between subsets in the initial matching set can be found. ZNSSD The minimum value is used to determine the correct correspondence between matching points.

[0024] Similarly, all matching points of interest related to length and width in a crystal can be obtained, i.e. and and This serves as the basis for the three-dimensional measurement of the crystal's length and width.

[0025] Step 3: Three-dimensional measurement of crystal size

[0026] Based on the crystal length and width interest point set obtained above, let's take one matched interest point pair as an example. The three-dimensional coordinates of this interest point pair in space can be obtained by the intersection position of its forward rays in the two cameras, where each ray passes through the optical center of the telecentric camera (located at infinity) and its respective pixel coordinates. Due to image noise and camera calibration errors, the two rays may not intersect in space. Therefore, a method based on analytical solutions is proposed to calculate the optimal position of the three-dimensional reconstructed coordinates. The positions of the two rays in space are represented as follows:

[0027]

[0028] Where, q′ L and q′ R These are matching point pairs in the world coordinate system after telecentric polar correction; o′ L and o′ R They are light rays q′ L Q L and q′ R Q R The unit direction vector.

[0029] Let P L =P R Calculate the shortest distance |Q| between the two light rays at this moment. L Q R |,q′ L and q′ R The optimal point for 3D reconstruction is located on line segment |Q L Q R Above:

[0030] Q(x,y,z)=(1-w)P L +wP R (4)

[0031] Here, w represents the scale factor, and the reconstructed point Q(x,y,z) is optimal and has the minimum reprojection error.

[0032] Based on the aforementioned interest point reconstruction method, the 3D coordinates of all matching interest points related to the crystal's length and width can be effectively calculated. Specifically, the 3D characteristic length of a crystal is simply determined by finding the longest spatial distance between all length feature lines. It is worth noting that the actual crystal profile may have protrusions or indentations, which can affect the edge regularity of its width measurement. To ensure the accuracy of the width measurement, a histogram-based measurement method based on the width feature lines is used to measure the width from a statistical perspective. Therefore, the 3D characteristic width of the crystal is determined by the median value of the rectangle with the highest frequency in the width histogram.

[0033] The beneficial effects of this invention are as follows: This invention can achieve non-invasive, high-quality binocular imaging monitoring at different locations in a tubular continuous crystallizer, thereby accurately calculating and analyzing the three-dimensional length and width of the crystal in the crystallizer, effectively avoiding measurement errors in the two-dimensional imaging plane, and more accurately analyzing the crystal growth state during the continuous crystallization process. This method is highly operable, requires less experience and technical expertise, and can automatically measure the three-dimensional dimensions of the crystal during the continuous crystallization process, facilitating practical industrial applications. Attached Figure Description

[0034] Figure 1 is a schematic diagram of the device of the present invention, wherein Figure 1(a) is a schematic diagram of the binocular backlight imaging system monitoring the straight tube of the continuous crystallizer, and Figure 1(b) is a schematic diagram of the in-situ calibration process;

[0035] Figure 2 This is a diagram illustrating the binocular image acquisition and crystal three-dimensional size measurement process of the present invention.

[0036] Figure 3 This is a diagram showing the overall structure of the Mask2Former instance segmentation model of the present invention.

[0037] Figure 4 This is a schematic diagram illustrating the extraction of crystal feature orientations and length and width interest point sets according to the present invention: Figure 4 (a) shows the crystal image pair and the aspect ratio of the left image. Figure 4 (b) is a map of the length and width of the two-dimensional feature lines and the extracted interest point set in the left image;

[0038] Figure 5 This is a schematic diagram illustrating the matching of crystal length and width interest point sets in crystal image pairing according to the present invention: Figure 5 (a) represents the initial matching sets of two types. Figure 5 (b) is the set of interest points of the matching length. Figure 5 (c) represents the set of interest points for matching width;

[0039] Figure 6 This is a schematic diagram of the optimal three-dimensional reconstruction method of the present invention;

[0040] Figure 7 The following are the 3D reconstruction coordinates of the crystal interest points and the corresponding crystal length and width measurement results of this invention: Figure 7 (a) consists of multiple three-dimensional length feature lines. Figure 7 (b) consists of multiple three-dimensional width feature lines. Figure 7 (c) is a histogram of the dimensions of the three-dimensional width feature line.

[0041] In the diagram: 1. Binocular telecentric camera; 2. Two telecentric backlight illuminators; 3. Four-axis optical platform; 4. Straight tube; 5. Microscale checkerboard calibration plate; A. Chamber; B. Four-axis micro-stage; C. Light source support; D. Optical platform X-axis.

[0042] E is the first Y-axis of the optical platform; F is the second Y-axis of the optical platform; G is the Z-axis of the optical platform; H is the process access port. Detailed Implementation

[0043] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] This method first designs a binocular backlit imaging system, including a binocular telecentric camera 1, two telecentric backlight illuminators 2, and a long-stroke four-axis optical platform 3, to simultaneously acquire images of the crystal solution at two different angles outside the straight tube 4 of the continuous crystallizer, as shown in Figure 1(a). Second, for the in-situ acquired binocular images, a deep learning-based stereo image segmentation method is proposed. A continuous crystallization image segmentation database is constructed, and an instance segmentation network is trained using transfer learning to accurately extract the crystal image mask and contour. Then, for the identified segmented crystal image pairs, their length and width feature directions, as well as the corresponding interest point sets, are extracted. Based on this, a crystal interest point matching algorithm is proposed to match the interest points in the crystal image pairs, constructing a matching interest point set related to the crystal's length and width. Finally, an optimal point 3D reconstruction method is used to reconstruct the above matching point set, and the length and width 3D feature lines are calculated, thereby quantitatively evaluating the crystal's 3D dimensions.

[0045] The designed binocular backlit imaging system is a non-contact industrial in-situ monitoring device for monitoring a continuous oscillating baffle crystallizer (COBC). This crystallizer comprises: 15 jacketed glass straight tubes 4 with an inner diameter of 15 mm (each tube contains periodically spaced baffles, forming 25 internal chambers), 15 glass bends, 4 circulating temperature control devices (controlling the temperature of four sections of the COBC), and 2 jacketed stirring tanks (using peristaltic pumps to deliver the crystallizing solute and seed crystals into the COBC). To achieve in-situ measurement of crystals suspended and flowing within the COBC, a binocular backlit imaging system was designed, including two industrial-grade high-resolution color CMOS cameras connected to a computer, and two telecentric backlight illuminators 2 to provide uniform backlight illumination, located on either side of the monitored COBC straight tube 4. During monitoring, the binocular telecentric camera 1 is positioned approximately 95 mm from the outer glass wall of the straight tube 4, ensuring that the focusing area is located within a single chamber A of the straight tube 4. In addition, a synchronous acquisition signal is sent from the computer's serial port to the binocular telecentric camera 1 via a synchronous trigger to acquire binocular images (left and right views) during the continuous crystallization process, which are then rapidly acquired by a data acquisition card. Finally, the proposed deep learning-based stereo image analysis method is used to obtain the three-dimensional dimensions of the crystal, measuring the entire process as follows: Figure 2 As shown.

[0046] The method specifically includes the following steps:

[0047] The first step is the design of a binocular backlight imaging system.

[0048] First, a binocular telecentric camera system was constructed. The camera used was a Daheng 2 / 3" global exposure color CMOS sensor, which is compact, lightweight, robust, and durable. The telecentric lens has a fixed magnification and low imaging distortion, and its orthogonal projection imaging characteristics are particularly suitable for accurately measuring crystal targets at the micrometer scale. Therefore, two Dehong 2x object-space telecentric lenses were selected and connected to the two cameras via a C-mount to form the microscopic binocular telecentric camera system. During imaging, each pixel in this system corresponds to an actual length of 1.75 μm, and the 2448×2048 image pixels correspond to 4.2× A 3.5mm field of view. Each telecentric camera is mounted on a four-axis miniature stage B, enabling high-precision XYZ-axis translation and vertical rotation, allowing both cameras to accurately focus on the same imaging area from different angles. It should be noted that the binocular telecentric cameras 1 are horizontally arranged (with an optical axis angle of approximately 15.61°), ensuring the camera baselines are parallel to the horizontal axis of the COBC straight tube 4. This facilitates in-situ measurement and minimizes distortion and optical refraction effects caused by the bending of the crystallizer glass tube wall during imaging. Both cameras are connected to an industrial control computer via a high-speed USB 3.0 port, and a custom camera trigger allows both cameras to simultaneously acquire crystal solution images at a maximum frame rate of 36Hz.

[0049] Next, two telecentric backlight illuminators 2 were constructed. Since backlighting improves the contrast between the object's outline and the background, it is more suitable for measuring micron-sized targets such as crystals. The designed two telecentric backlight illuminators 2 consist of two LED light sources connected to two telecentric lenses, allowing only a uniform parallel beam of light to pass through the COBC straight tube 4 and enter the binocular telecentric camera 1, thereby producing a high-quality image of the crystal particles. At this point, the two telecentric backlight illuminators 2 and the binocular telecentric camera 1 are placed on either side of the imaging crystal. Each designed illuminator outputs monochromatic green light with a peak wavelength of 521 nm to mitigate chromatic aberration and diffraction limits. The beam diameter of both illuminators is 30 mm. Furthermore, both illuminators are mounted in a light source support C for easy movement and rotation, ensuring the parallel beam is aligned with the optical axes of the two camera lenses.

[0050] Finally, a long-stroke four-axis optical platform 3 is constructed. This optical platform is used to mount the aforementioned binocular telecentric camera 1, two telecentric backlight illuminators 2, and their fixing components. Specifically, its functions are as follows: The measurement area of ​​the binocular telecentric camera 1 is moved along a straight tube 4 of the COBC via the X-axis displacement (400mm stroke); the binocular telecentric camera 1 is approximately positioned within its working distance (105mm) via the first Y-axis displacement (400mm stroke); then, the binocular camera is focused via the second Y-axis precise displacement (F) so that its common focusing area can capture a clear crystal image; and the binocular telecentric camera 1 is moved vertically via the Z-axis displacement (G) to monitor the straight tubes of different layers of the COBC.

[0051] It should be noted that when using the binocular backlit imaging system designed above for in-situ imaging, both cameras must simultaneously capture crystal particles in the flowing suspension. The camera exposure time is set to 10 μs to effectively avoid motion blur. In subsequent crystal image processing, the G component is extracted from the original RGB image for analysis.

[0052] The second step is in-situ stereoscopic imaging calibration.

[0053] Before acquiring binocular crystal images for analysis, the parameters of the aforementioned binocular telecentric camera 1 need to be accurately calibrated in advance. Because the telecentric camera has orthogonal projection characteristics, a three-dimensional world point Q(x,y,z) and its corresponding two-dimensional image point q on both cameras are... L (u L ,v L ) and q R (u R ,v R The mapping relationship between them is expressed as:

[0054]

[0055] Among them, I i and (R) i ,t i C represents the camera's internal and external parameter matrices, respectively; i Let C′ represent the affine form of the camera projection matrix (the last row is (0,0,0,1)); i∈{L,R} identifies the left and right camera sources. To simplify stereo image matching, telecentric epipolar correction is introduced to transform the stereo images so that the epipolar lines in each image are parallel and horizontal. Therefore, the corrected camera projection matrix C′ L and C′ R Represented as:

[0056]

[0057] Where, q′ L and q′ R This represents the matching point pair in the corrected binocular image.

[0058] To determine the model parameters of the two cameras in equations (1) and (2), an in-situ calibration device as shown in Figure 1(b) is used for stereo imaging calibration. The microscale checkerboard calibration plate 5 is inserted into the process access port H of the COBC straight tube 4, placing it in the chamber A between the baffles. The calibration plate is rotated and moved within the common field of view of the binocular telecentric camera 1, and several sets of checkerboard pose image pairs are acquired for in-situ calibration. Specifically, during calibration, initial values ​​of the camera's intrinsic and extrinsic parameters are first obtained through a closed-loop solution, followed by further refinement and adjustment of these parameters and the distortion coefficients (imaging distortion caused by the lens and crystallizer glass wall) through nonlinear optimization. Afterward, the initial and corrected binocular camera parameter matrices can be used for three-dimensional reconstruction of the crystal.

[0059] Step 3: Crystal image segmentation

[0060] Accurate image segmentation of the acquired in-situ stereo images is an indispensable step in extracting the crystal's interest point set. Deep learning-based image segmentation techniques do not rely on manually extracted features and can obtain more comprehensive low-level image features through training, which is beneficial for improving the accuracy and robustness of crystal image segmentation. Therefore, this invention uses the recently published high-performance Mask2Former instance segmentation network to extract pixel-by-pixel mask information of the crystal image. The overall structure is as follows: Figure 3 As shown, it includes three modules with different functions: a backbone feature extractor, a pixel decoder, and a Transformer decoder.

[0061] Specifically, the backbone module extracts low-resolution features from the entire input image. The resulting feature maps are typically small, with each pixel having a large receptive field and containing rich semantic information related to the crystal object. Subsequently, the pixel decoder module progressively upsamples these feature maps to generate multi-scale high-resolution features, ultimately obtaining feature pyramids with resolutions of 1 / 32, 1 / 16, and 1 / 8 of the original image. This improves the detection performance for crystals of various sizes, particularly small crystal particles in the first few straight sections of the COBC. Finally, the multi-scale image features are sequentially input into the Transformer decoder layer, where the mask attention mechanism of this module confines attention to the prediction mask region for each query, rather than searching the entire feature map.

[0062] To train the Mask2Former network, a custom database of 600 images of continuous glutamate crystals, taken under different experimental conditions and in different straight tube regions, was constructed. 480 images were randomly selected for training, and the remaining 120 were used for validation. The target crystals in the database were annotated with polygons using the open-source software Labelme to depict their outlines. Furthermore, supervised data augmentation strategies were employed to improve the robustness and generalization ability of the Mask2Former model, including flipping, blurring, randomized cropping, affine transformation, and brightness and contrast adjustments. Ultimately, the total number of annotated crystal images in the training set was expanded to over 100,000, enabling effective training and validation of the Mask2Former model.

[0063] Based on the constructed image database, when training the Mask2Former model, minimizing the deviation between the predicted results and the true labels, i.e., the training loss function, can be defined as:

[0064]

[0065] in, It is the cross-entropy loss in classification tasks; while and These are the binary cross-entropy loss and the Dice loss in the binary image, which together constitute the segmentation mask loss.

[0066] During training, precision, recall, and mean precision (MSP), commonly used in object detection and instance segmentation, are used to quantitatively evaluate the performance of the trained model. By setting an appropriate number of epochs, the model with the highest MSP is recorded as the crystal image segmentation model used in this invention.

[0067] The fourth step is the extraction and matching of crystal length and width interest point sets.

[0068] After using the optimal Mask2Former model to detect the in-situ acquired binocular images and obtain pixel-by-pixel masks of crystal instances, the minimum bounding rectangle method is then used to enclose the contour of each crystal image, and crystal pairs appearing simultaneously in two images are identified, i.e., crystal image matching pairs. Furthermore, taking the left image of a crystal image pair as an example, the length and width of the left crystal image can be calculated using the angles between the long and short sides of its bounding rectangle and the horizontal axis. The extracted length and width feature directions are as follows: Figure 4 The dashed and dotted lines in (a) illustrate this. Therefore, the two-dimensional feature length of the left image of the crystal can be determined by the maximum distance between two pixels on the crystal contour parallel to the length feature direction. Similarly, the two-dimensional feature width of the crystal can be quantized along the width feature direction of the crystal. Therefore, for the left image in a pair of crystal images, by uniformly dividing its contour points along its length and width feature directions respectively, multiple feature lines representing the length and width of the left image of the crystal can be obtained, such as... Figure 4 As shown in (b). Simultaneously, the intersections of these feature lines with the contour of the left crystal image record the set of interest points for their length and width. and like Figure 4 (b) shows the points marked with triangles of the same color.

[0069] To obtain the characteristic length and width of this example crystal in three-dimensional space, it is necessary to find the set of left interest points. and Matching point set in the right image of the crystal and Therefore, this invention proposes a two-step crystal interest point matching algorithm, which applies two matching constraints in sequence: 1) adopts a telecentric epipolar constraint to reduce the search area for establishing the initial matching set; 2) adopts an intensity constraint to further select point correspondences that satisfy the minimum intensity difference criterion.

[0070] (1) Telecentric epipolar constraint:

[0071] by Taking a point of interest as an example, calculate the vertical distance d(i) between its epipolar line and the contour point of the right image of the crystal:

[0072] d(i)=|v Ri -v epipolar |,i=1,2,...,n / 2 (4)

[0073] Among them, v epipolar It is the vertical coordinate of the polar line; v Riis the vertical coordinate of the right mask contour point; n represents the number of right mask contour points. Right contour points satisfying d(i) = 0 are potential matching points. Considering potential image segmentation errors, this right contour point, along with four other pixels extending horizontally on both sides, simultaneously constitutes the initial matching set. It should be noted that when multiple points satisfy d(i) = 0, it indicates that this segment of the crystal contour is parallel to the epipolar line, and the initial matching set will be further expanded accordingly. Figure 5 (a) illustrates the two types of initial matching sets mentioned above.

[0074] (2) Pixel intensity constraint:

[0075] Next, for the initial matching set obtained in the first step, the true matching point pairs are determined by the intensity difference between subsets in the left (reference) image and the right (target) image. Considering the influence of brightness variations in the binocular images caused by differences in the binocular camera's field of view, the zero-mean normalized minimum squared distance (ZNSSD) matching criterion is used to calculate the pixel value differences between subsets:

[0076]

[0077] in, r j and t j These represent the intensity values ​​of the j-th pixel in the reference subset and the target subset, respectively. and These represent the average intensity values ​​of all pixels in the reference subset and the target subset, respectively. Therefore, by finding the χ² values ​​between subsets in the initial matching set... ZNSSD The minimum value is used to determine the correct point correspondence.

[0078] By analogy, all matching pairs of interest points related to length and width in a crystal image pair can be obtained, i.e. and and like Figure 5 As shown in (b)-(c), this forms the basis for the three-dimensional measurement of the length and width of the crystal.

[0079] Step 5: Three-dimensional measurement of crystal size

[0080] Based on the crystal length and width interest point set obtained above, let's take one matched interest point pair as an example. The three-dimensional position of this interest point can usually be obtained by the intersection of two corresponding rays from two cameras, where each ray passes through the optical center of the telecentric camera (located at infinity) and the corresponding pixel position in that camera. However, due to image noise and camera calibration errors, these two rays may not intersect in three-dimensional space. Therefore, this invention proposes a three-dimensional reconstruction method based on the optimal point of the analytical solution.

[0081] Figure 6 A schematic diagram of the method for 3D reconstruction in the world coordinate system is given, where O′ LC -X L ′ C Y L ′ C Z L ′ C and O′ RC -X′ RC Y R ′ C Z′ RC q′ represents the coordinate systems of the left and right cameras after telecentric polar correction; L and q′ R This represents the pair of points of interest in the world coordinate system after correction. Without loss of generality, this invention defines the corrected left camera coordinate system as the world coordinate system. The optimal 3D point Q to be reconstructed lies on the shortest line segment |Q| between two light rays. L Q R |above, and satisfying the minimum reprojection error, the specific derivation process is as follows.

[0082] q′ L Q L and q′ R Q R P represents two light rays in the world coordinate system. L and P R Representing three-dimensional point Q respectively L and Q R In q′ L Q L and q′ R Q R The position above:

[0083]

[0084] Among them, o′ L and o′ R They are q′ L Q L and q′ R Q R The unit direction vector. Due to the optical center O′ of the telecentric camera. LC and O′ RC Located at infinity, the light rays in the left and right views are parallel to the optical axes (i.e., the Z-axis) of the left and right cameras, respectively. Therefore, o′ L and o′ R We can get from R′ respectively L and R′ R Determined in the third line.

[0085] Let P L =P RAt this moment, the shortest distance between the two light rays is |Q L Q R It can be calculated using the least squares method:

[0086] [|q′ L Q L | |q′ R Q R |] T =[o′ L -o′ R ] + (q′ R -q′ L (7)

[0087] in,[] + P represents the left pseudoinverse of a matrix; L and P R According to |q′ L Q L | and |q′ R Q R The result has also been confirmed.

[0088] Using the obtained |Q L Q R | Assume that the optimal point Q has the minimum reprojection error and satisfies:

[0089] |Q L Q|=w|Q L Q R | (8)

[0090] Where w represents the scale factor, the reprojection error of Q can be defined as:

[0091]

[0092] Where, q′ L,repro and q′ R,repro These are the reprojection points of Q in the two cameras, respectively.

[0093] Then, the two Euclidean distances were calculated. and in It is Q R The reprojection point in the left camera; It is Q L The reprojection point in the right camera. Since all rays from each camera are parallel, there exists |q′. L,repro -q′ L |=wd1 and |q′ R,repro -q′ R |=(1-w)d2, equation (9) can be rewritten as:

[0094] e 2 =w 2 d1 2 +(1-w) 2 d2 2 (10)

[0095] The optimal value of w can be obtained by minimizing equation (10):

[0096]

[0097] Once w is known, the optimal three-dimensional coordinate point Q(x,y,z) can be calculated as follows:

[0098] Q(x,y,z)=(1-w)P L +wP R (12)

[0099] Based on the above interest point reconstruction method, the optimal three-dimensional coordinates of all interest point pairs related to the crystal length and width can be effectively calculated. Therefore, Figure 4 (b) The three-dimensional dimensions of the multiple feature lines representing the length and width of a crystal, as shown, can be measured individually by Euclidean distances in three-dimensional space. Specifically, the three-dimensional feature length of a crystal is determined by simply finding the longest spatial distance between all the length feature lines, such as... Figure 7 (a) shows the thick solid line and the two triangles marking the points of interest. It should be noted that the actual crystal profile may have protrusions or depressions, which can affect the edge regularity of its width measurement. To ensure the accuracy of the width measurement, this invention employs a measurement method based on a histogram of all width feature lines, performing width measurement from a statistical perspective. Therefore, the three-dimensional feature width of the crystal is determined by the median value of the rectangle with the highest frequency in the width histogram, such as... Figure 7 (b) and (c), where the three-dimensional width values ​​of the example crystals are as follows Figure 7 As shown by the dashed line in (c). By statistically analyzing the three-dimensional length and width of multiple pairs of crystal images, the size distribution of the crystal length and width over a certain period of time can be formed, thereby obtaining the most critical process monitoring information in the continuous crystallization process.

[0100] The above-described embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A method for measuring the length and width of crystals in a continuous crystallization process in real time by binocular backlit imaging, characterized in that Includes the following steps: The first step is to establish a binocular backlight imaging system. (1) Utilize the orthographic projection imaging characteristics of a telecentric camera to accurately measure crystal targets at the micrometer scale; select a matching object-side telecentric lens based on the pixel size and sensor size of the high-definition industrial camera used. After installing the two cameras and lenses, they were mounted on a four-degree-of-freedom miniature displacement stage to provide high precision for the camera and lens system. X - Y - Z - Axial translation and vertical rotation are used to focus the two camera lens systems on a common imaging area; the baselines of the two telecentric cameras are parallel to the horizontal axis of the straight tube of the tubular continuous crystallizer, that is, the two cameras are placed horizontally to minimize the optical refraction effect caused by the bending of the crystallizer glass tube wall during imaging. (2) Two telecentric backlight illuminators are placed on both sides of the imaging target along with the binocular telecentric camera; the LED light source is connected to the telecentric lens, allowing only uniform parallel light rays to pass through the tubular crystallizer and then into the telecentric camera for imaging; both telecentric backlight illuminators are mounted in custom light source brackets for easy movement and rotation, so that the parallel beams are aligned with the optical axes of the two camera lenses respectively. (3) A long-stroke four-axis optical platform for mounting the above-mentioned binocular telecentric camera, telecentric backlight illuminator, and their fixing components; this optical platform is achieved through... X The displacement of the axis causes the measurement area of ​​the binocular camera to move along the straight tube direction of the tubular continuous crystallizer; through the first Y The displacement of the axis moves the binocular camera into its working range; then through the second... Y Precise displacement of the axis is used to focus the binocular camera, ensuring that a clear crystal image is captured in its common focal area; through Z The displacement of the axis is used to move the binocular camera vertically to monitor the straight tubes of different layers of the crystallizer; In addition, before actual measurement, the simultaneously acquired micron-level checkerboard calibration image needs to be used, combined with the calibration and correction algorithm of the telecentric camera, to establish a stereo imaging model of this binocular backlight imaging system for three-dimensional reconstruction of crystal size. The second step involves crystal image segmentation, extraction and matching of length and width interest point sets. (1) Deep learning-based in-situ crystal image segmentation and contour extraction: A large number of in-situ crystal images under different continuous crystallization experimental conditions, different detection areas and different measurement times were collected. A continuous crystallization image segmentation database was constructed by manually annotating the images. Using this database, the Mask2Former instance segmentation network was trained to segment the crystal target of interest from the new in-situ images and extract its corresponding crystal contour. Some crystal images without statistical significance were deleted. (2) Crystal length and width interest point extraction: The minimum bounding rectangle method is used to enclose the contour of each crystal image. At the same time, on the stereo-corrected binocular image, the crystal image pairs in the binocular image are identified based on the coordinate information of the four corner points of the horizontal epipolar line and the bounding rectangle. The length and width of a crystal image can be calculated by the angle between the long and short sides of its enclosing rectangle and the horizontal axis. Therefore, the two-dimensional feature length of the crystal image can be determined by any two points on its contour that are parallel to the length feature direction of the crystal and have the maximum distance. The two-dimensional feature width can be quantized along the width feature direction of the crystal. Therefore, for the left image in a crystal image pair, the contour points are evenly divided along its length and width feature directions to obtain multiple two-dimensional feature lines of length and width in the left image of the crystal. The intersections of these lines with the left crystal contour can be classified into the interest point set of its length and width. and ; To calculate the characteristic length and width of a crystal in three-dimensional space, it is necessary to find the set of left interest points. and Matching point set in the right image of the crystal and The following proposes a crystal interest point matching algorithm under two matching constraints; ① Telecentric polar constraint: by Taking a point of interest as an example, calculate the vertical coordinate distance between its epipolar line and the contour point of the right image of the crystal. : (1) in, These are the vertical coordinates of the polar lines; These are the vertical coordinates of the right contour point; n Represents the number of points on the right contour; when the following conditions are met... At that time, the right contour point is a possible matching point; considering the possible image missegmentation, the right contour point and the other four points extending horizontally together form the initial matching set. ②Pixel intensity constraint: The true matching point pairs in the initial matching set are found by using the intensity difference between subsets of the left image (reference image) and the right image (target image); the zero-mean normalized minimum squared distance (ZNSSD) matching criterion is adopted to reduce the impact of brightness differences in the binocular images. (2) in, ; ; and These are the reference subset and the target subset, respectively. j The intensity value of each pixel; and These are the average intensity values ​​of all pixels in the reference subset and the target subset, respectively; therefore, by finding the subsets in the initial matching set... The minimum value is used to determine the correct correspondence between matching points; Similarly, all matching points of interest related to length and width in a crystal can be obtained, i.e. and , and This serves as the basis for the three-dimensional measurement of the crystal's length and width; Step 3: Three-dimensional measurement of crystal size Based on the crystal length and width interest point set obtained above, taking one matched interest point pair as an example: the three-dimensional coordinates of this interest point pair in space can be obtained by the intersection position of its forward rays in the two cameras, where each ray passes through the optical center of the telecentric camera and its respective pixel coordinates; due to image noise and camera calibration errors, the two rays may not intersect in space, therefore, a method based on analytical solutions is proposed to calculate the optimal position of the three-dimensional reconstructed coordinates; the positions of the two rays in space are represented as follows: (3) in, and These are matching point pairs in the world coordinate system after telecentric polar correction; and They are light rays and The unit direction vector; make Calculate the shortest distance between the two light rays at this moment. , and The optimal point for 3D reconstruction is located on the line segment. superior: (4) in, Represents the scale factor, at which point the reconstruction point is located. It is optimal and has the smallest reprojection error; Based on the above interest point reconstruction method, all matching points in the crystal length and width interest point sets are reconstructed, and the three-dimensional feature line dimensions representing the crystal length and width are obtained by calculating the Euclidean distance. Therefore, the three-dimensional feature length of the crystal can be determined by simply finding the longest spatial distance between all length feature lines. To ensure the accuracy of the width measurement of needle-shaped or rod-shaped crystals, a measurement method based on the histogram of the width feature lines is adopted to measure the width from a statistical perspective. Therefore, the three-dimensional feature width of the crystal is determined by the median value of the rectangle with the highest frequency in the width histogram.

2. The method for real-time detection of crystal length and width during continuous crystallization process using binocular backlight imaging according to claim 1, characterized in that, The custom continuous crystallization database in step (1) of the second step is constructed as follows: First, 600 images of continuous glutamic acid crystallization taken under different experimental conditions and in different straight tube regions are collected and randomly selected as a custom continuous crystallization image database. Among them, 480 images are randomly selected for training, and the remaining 120 images are used for model validation. All crystal instances in the database are annotated with polygons using the open-source software Labelme to depict the crystal outline. Then, some supervised data augmentation strategies are used to improve the robustness and generalization ability of the deep segmentation network, including flipping, blurring, random size cropping, affine transformation, brightness and contrast adjustment operations. Finally, the total number of annotated crystal images in the training set of the constructed database is expanded to more than 100,000 for effective training and validation of the Mask2Former segmentation model.

Images