Method for detecting length and width of crystal in continuous crystallization process in real time through binocular backlight imaging
Through binocular backlight imaging system and deep learning analysis methods, the problems of in-situ acquisition and three-dimensional measurement during continuous crystallization are solved, and high-quality three-dimensional dimension monitoring of the crystal is achieved, ensuring the accuracy and non-invasiveness of the measurement.
Patent Information
- Application Number
- CN202411012429.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The prior art is difficult to acquire binocular images of crystals in situ and process them in real time during continuous crystallization, measure their three-dimensional length and width dimensions, and intrusive probes may affect the crystallization process and product characteristics.
A binocular backlight imaging system is designed, using high-definition industrial cameras and telecentric backlight illuminators for non-contact image acquisition, combined with deep learning stereoscopic image analysis method to achieve the extraction of crystal length and width feature directions and interest sets and Three-dimensional reconstruction.
High-quality binocular imaging monitoring of crystals in continuous crystals is realized, and the three-dimensional length and width are accurately calculated, which avoids the measurement error of the two-dimensional imaging plane and improves the accuracy of the growth state analysis of the continuous crystallization process.
Smart Images

Figure CN118967782A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of industrial process control and detection technology, and relates to in-situ image processing technology for industrial continuous crystallization processes, and in particular to a non-invasive online monitoring system based on binocular backlight imaging, and a method for measuring the length and width of crystals based on deep learning stereo image analysis. Specifically, it refers to a method for real-time in-situ measurement of the three-dimensional length and width of a crystal population in a continuous oscillation baffle crystallizer using two high-definition industrial telecentric cameras, a telecentric light source and an adjustable optical platform. Background Art
[0002] The process and control technology of continuous crystallization process plays a very important role and significance to the high-quality development of my country's advanced manufacturing field. In the process of continuous crystallization using tubular continuous oscillating baffle crystallizer, online detection and control optimization of crystals at different time stages and different positions can help generate crystals of the desired consistent shape and size, thereby ensuring the quality of crystal particle products. In the process of in-situ monitoring of continuous crystallization using computer vision, there is a lack of effective and convenient online process analysis equipment to accurately and intuitively characterize the true shape and size information of the crystal population in the solution. In addition, since the projected size of the crystal in the image plane depends on its three-dimensional position in the tubular crystallizer solution, it is of great significance to extract the length and width characteristic directions of each crystal and its set of points of interest, and realize the three-dimensional reconstruction of the length and width of the crystal. There are few literatures and patents at home and abroad that introduce binocular-based real-time monitoring equipment for continuous crystallization process that can be promoted and used, as well as standardized methods for online measurement of the length and width three-dimensional dimensions of crystals during growth.
[0003] Although a small number of devices for in-situ monitoring of continuous crystallizers have been developed, for example, H. Kramer, a professor at Delft University of Technology in the Netherlands, and others used the SOPAT-VI industrial camera probe of the German company SOPAT GmbH in the paper "Application of inline imaging for monitoring crystallization process in a continuous oscillatory baffled crystallizer" (published in AIChE Journal, 2018, 64(7), 2450–2461), to intrusively photograph melamine crystals during the continuous crystallization process and extract the one-dimensional size information of the crystal population. For example, Professor ZK Nagy of Purdue University in the United States and others used the Blaze 900 camera probe to take high-resolution images in the recent paper “Continuous spherical crystallization of lysozyme in an oscillatory baffled crystallizer using emulsion solvent diffusion in droplets” (abbreviated translation: Continuous spherical crystallization of lysozyme in an oscillatory baffled crystallizer using emulsion solvent droplet diffusion method, published in Crystal Growth & Design, 2020, 20, 934-947, an important international journal in the field of chemical industry) to characterize the spherical crystals of lysozyme in different types of continuous crystallizers. However, the above analysis methods all insert the probe used for shooting into the crystallizer solution, which may cause the crystals to adhere to the probe surface during the crystallization process and affect the subsequent image acquisition. In addition, this type of invasive probe may affect the fluid dynamics during continuous crystallization, thereby changing the final product characteristics of the crystal. Therefore, how to collect monocular or even binocular images of the crystal in the crystallizer in situ during the continuous crystallization process, and extract the length and width feature point set of the crystal in real time and reconstruct its three-dimensional size based on this, is still a difficult problem in current research and application.In addition, as Professor ZGGao of Tianjin University and others pointed out in the paper "Averified open-access AI-based chemical microparticle image database for in-situ particle visualization and quantification in multi-phase flow" (abbreviated translation: a verified open-source chemical microparticle image database based on artificial intelligence, used for in-situ particle visualization and quantitative characterization in multiphase flow, published in Chemical Engineering Journal, 2023, 451, 138940, an important international journal in the chemical industry), the image analysis strategy based on deep learning plays a key role in the detection and quantification of the crystallization process, which can provide reliable and intuitive crystallization process information. For example, the image segmentation network proposed by BWCheng et al. from Facebook AI Research in the paper "Masked-attention mask transformer for universal image segmentation" (abbreviated as: Masked-attention mask transformer under mask attention mechanism for universal image segmentation; Masked-attention mask transformer, referred to as Mask2Former; published in the top international computer vision conference IEEE Conference on Computer Vision and Pattern Recognition, 2022, 1280-1289) can accurately segment the target objects, and therefore can be transferred to the segmentation and extraction of crystal targets in solution images, but it is necessary to further develop accurate and efficient in-situ real-time stereo image analysis methods based on deep learning. Summary of the invention
[0004] The technical problem to be solved by the present invention is how to collect images of crystals at different angles in the continuous crystallizer in situ, and how to process the above stereo images in real time and measure the three-dimensional length and width of each crystal. To solve these problems, a binocular backlight imaging system is designed to synchronously collect crystal images in the solution, as well as a technical method for real-time processing and three-dimensional reconstruction of the corresponding crystal images, so as to realize the monitoring of the three-dimensional length and width information of the crystals in the continuous crystallization process.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] A method for real-time detection of crystal length and width in a continuous crystallization process based on binocular backlight imaging is implemented based on a designed non-contact high-definition binocular telecentric backlight image acquisition device, which includes two object-side telecentric cameras, two telecentric backlight illuminators, and a four-axis optical platform. First, two high-resolution object-side telecentric lenses are connected to a high-definition industrial CMOS camera with matching specifications to form a microscope camera lens system. Secondly, two LED light sources are connected to the other two telecentric lenses to form a telecentric light source that can emit parallel light beams for backlighting of translucent crystals. Thirdly, the camera lens system and the telecentric light source system are installed on a long-stroke four-axis optical platform to achieve synchronous acquisition of binocular images (left view and right view) from two angles at different positions outside the crystallizer. For the acquired binocular images, a stereo image analysis method based on deep learning is proposed, including crystal image segmentation and contour extraction, crystal length and width feature direction and interest point set extraction and matching. Finally, for the matching set of crystal length and width interest points, the in-situ stereo calibration and correction model is used to perform three-dimensional reconstruction of each crystal length and width interest point set, thereby quantitatively evaluating its three-dimensional size. The method includes the following steps:
[0007] Step 1: Design of binocular backlight imaging system
[0008] (1) Use the orthographic imaging characteristics of the telecentric camera to accurately measure crystal targets at the micron scale. Specifically, first select the matching object-side telecentric lens based on the pixel size and sensor size of the high-definition industrial camera used, and also pay attention to the selection of the matching interface method for the two. After installing the two cameras and lenses, they are respectively installed on a four-degree-of-freedom micro-displacement stage to provide high-precision XYZ-axis translation and vertical rotation of the camera lens system, so that the two camera lens systems focus on a common imaging area. In addition, the baselines of the two telecentric cameras should be parallel to the horizontal axis of the straight tube of the tubular continuous crystallizer, that is, the two cameras are placed horizontally to minimize the optical refraction effect caused by the bending of the crystallizer glass tube wall during imaging.
[0009] (2) Design two telecentric backlight illuminators and place them and the binocular telecentric camera on both sides of the imaging target, which is conducive to obtaining images with more obvious contrast and facilitating further measurement of the size of the object being measured, especially for translucent and low-reflection crystal particles, such as glutamate crystals. Specifically, the LED light source is connected to the telecentric lens, allowing only uniform parallel light to pass through the tubular crystallizer and then enter the telecentric camera and form an image. Both telecentric backlight illuminators are mounted in a customized light source bracket, which is easy to move and rotate so that the parallel light beams are aligned with the optical axes of the two camera lenses respectively.
[0010] (3) Design a long-stroke four-axis optical platform for installing the above binocular telecentric camera, telecentric backlight illuminator and its fixing parts. The optical platform moves the measurement area of the binocular camera along the straight tube direction of the tubular continuous crystallizer through the displacement of the X axis; the binocular camera is roughly sent into its working distance through the displacement of the first Y axis; then the binocular camera is focused through the precise displacement of the second Y axis so that its common focus area can capture a clear crystal image; the displacement of the Z axis is used to move the binocular camera in the vertical direction to monitor the straight tubes at different layers of the crystallizer.
[0011] In addition, before the actual measurement, the micron-level checkerboard calibration image collected simultaneously is used, combined with the calibration and correction algorithm of the telecentric camera, to establish a stereo imaging model of this binocular backlight imaging system for three-dimensional reconstruction of the crystal size.
[0012] The second step is crystal image segmentation, length and width interest point set extraction and matching
[0013] (1) In-situ crystal image segmentation and contour extraction based on deep learning: First, a large number of in-situ crystal images under different continuous crystallization experimental conditions, different detection areas, and different measurement times are collected, and a continuous crystallization crystal image segmentation database is constructed by manual annotation. Secondly, this database is used to train a Masked-attention masktransformer (Mask2Former) instance segmentation network to segment the crystal targets of interest from the new in-situ images and extract their corresponding crystal contours. Thirdly, some crystal images that have no statistical significance are deleted, such as incomplete crystals that intersect with the entire image border.
[0014] (2) Extraction of crystal length and width interest points: First, use the minimum circumscribed rectangle method to enclose the contour of each of the above crystal images. At the same time, on the stereo-corrected binocular image, identify the crystal image pair in the binocular image based on the horizontal polar lines and the coordinate information of the four corner points of the rectangular enclosing box. The length and width directions of a crystal image can be calculated by the angles between the long and short sides of its enclosing rectangle and the horizontal axis. Therefore, the two-dimensional characteristic length of the crystal image can be determined by any two points on its contour that are parallel to the length characteristic direction of the crystal and have the maximum distance. Similarly, the two-dimensional characteristic width of the crystal can be quantified along the width characteristic direction of the crystal. Therefore, for the left image in a crystal image pair, its contour points are evenly divided along its length and width characteristic directions, respectively, and multiple two-dimensional characteristic lines of length and width in the left image of the crystal can be obtained. Then, the intersection of these lines with the left crystal contour can be classified as a set of interest points for its length and width. and
[0015] In order to calculate the characteristic length and width of a crystal in three-dimensional space, it is necessary to find the left interest point set and The set of matching points in the right image of the crystal and A crystal interest point matching algorithm under two matching constraints is proposed below.
[0016] ①Telecentric polar line constraint:
[0017] by Take an interest point in the image as an example, and calculate the vertical coordinate distance d(i) between its epipolar line and the contour point of the right image of the crystal:
[0018] d(i)=|v Ri -v epipolar |,i=1,2,...,n / 2 (1)
[0019] Among them, v epipolar is the vertical coordinate of the polar line; v Ri is the vertical coordinate of the right contour point; n represents the number of right contour points. When d(i) = 0, the right contour point is a possible matching point. Considering the possible image mis-segmentation, the right contour point and the other four points extending horizontally from it constitute the initial matching set.
[0020] ②Pixel intensity constraint:
[0021] The true matching point pairs in the initial matching set are found by the intensity difference between the subsets in the left (reference) image and the right (target) image. The zero-mean normalized minimum square distance (ZNSSD) matching criterion is used to reduce the influence of different brightness in the binocular images:
[0022]
[0023] in, r j and t j are the intensity values of the jth pixel in the reference subset and the target subset, respectively; and are the average intensity values of all pixels in the reference subset and the target subset, respectively. Therefore, we can find the χ between the subsets in the initial matching set. ZNSSD The minimum value is used to determine the correct matching point correspondence.
[0024] Similarly, all matching points of interest related to length and width in a crystal can be obtained, that is, and and Serves as the basis for three-dimensional measurement of the length and width of the crystal.
[0025] Step 3: 3D measurement of crystal size
[0026] Based on the above-obtained crystal length and width interest point set, we still take one of the matching interest point pairs as an example. The three-dimensional coordinates of the interest point pair in space can be obtained by the intersection position of its forward rays in the two cameras, where each ray passes through the optical center of the telecentric camera (located at infinity) and the respective pixel coordinates. Due to image noise and camera calibration errors, the two rays may not intersect in space, so a method based on analytical solution is proposed to calculate the optimal position of the three-dimensional reconstructed coordinates. The positions of the two rays in space are expressed as follows:
[0027]
[0028] Among them, q′ L and q′ R is the matching point pair in the world coordinate system after the telecentric polar correction; o′ L and o′ R The light q′ L Q L and q′ R Q R The unit direction vector of .
[0029] Let P L =P R , calculate the shortest distance between the two rays at this time |Q L Q R |,q′ L and q′ R The optimal point of 3D reconstruction is located on the line segment |Q L Q R |Top:
[0030] Q(x,y,z)=(1-w)P L +wP R (4)
[0031] Among them, w represents the scale factor, at this time the reconstructed point Q(x, y, z) is optimal and has the smallest reprojection error.
[0032] Based on the above interest point reconstruction method, the three-dimensional coordinates of all matching interest points related to the length and width of the crystal can be effectively calculated. Specifically, the three-dimensional characteristic length of a crystal is simply determined by finding the longest spatial distance between all length characteristic lines. It is worth noting that the actual crystal contour may have protrusions or depressions, which will affect the edge regularity of its width measurement. In order to ensure the accuracy of the width measurement, a measurement method based on the histogram of the width characteristic line is adopted to measure the width from a statistical perspective. Therefore, the three-dimensional characteristic width of the crystal is determined by the median value of the rectangle with the largest frequency in the width histogram.
[0033] The beneficial effects of the present invention are as follows: the present invention can realize non-invasive high-quality binocular imaging monitoring of different positions of the tubular continuous crystallizer, so that the three-dimensional length and width of the crystal in the crystallizer can be accurately calculated and analyzed, and the measurement error in the two-dimensional imaging plane can be effectively avoided, and the growth state of the crystal in the continuous crystallization process can be analyzed more accurately. The method is highly operable, has low requirements for experience and technology, can achieve the effect of automatically measuring the three-dimensional size of the crystal in the continuous crystallization process, and is convenient for practical industrial application. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] FIG. 1 is a schematic diagram of the equipment of the present invention, wherein FIG. 1( a ) is a schematic diagram of a binocular backlight imaging system monitoring a straight tube of a continuous crystallizer, and FIG. 1( b ) is a schematic diagram of an in-situ calibration process;
[0035] Figure 2 This is a diagram showing the binocular image acquisition and crystal three-dimensional size measurement process implementation of the present invention;
[0036] Figure 3 The overall structure diagram of the Mask2Former instance segmentation model of the present invention;
[0037] Figure 4 This is a schematic diagram of extracting the crystal characteristic direction, length and width interest point set of the present invention: Figure 4 (a) is the crystal image pair and the length and width characteristic direction map of the left image. Figure 4 (b) shows the length and width two-dimensional feature lines and the extracted interest point set in the left image;
[0038] Figure 5 This is a schematic diagram of matching the crystal length and width interest point sets in the crystal image of the present invention: Figure 5 (a) are two types of initial matching sets, Figure 5 (b) is the length of the matching interest point set, Figure 5 (c) is the matched width interest point set;
[0039] Figure 6 It is a schematic diagram of the optimal three-dimensional reconstruction method of the present invention;
[0040] Figure 7 The three-dimensional reconstructed coordinates of the crystal interest points of the present invention and the corresponding crystal length and width measurement results: Figure 7 (a) is a number of three-dimensional length characteristic lines, Figure 7 (b) is a number of three-dimensional width feature lines, Figure 7 (c) is the histogram of the three-dimensional width feature line size.
[0041] In the figure: 1 binocular telecentric camera; 2 two telecentric backlight illuminators; 3 four-axis optical platform; 4 straight tube; 5 micro-scale checkerboard calibration plate; A chamber; B four-axis micro-displacement stage; C light source bracket; D optical platform X axis;
[0042] E optical platform first Y axis; F optical platform second Y axis; G optical platform Z axis; H process access port. DETAILED DESCRIPTION
[0043] In order to better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0044] The method first designs a binocular backlight imaging system, including a binocular telecentric camera 1, two telecentric backlight illuminators 2 and a long-stroke four-axis optical platform 3, to synchronously collect crystal solution images at two different angles outside the continuous crystallizer straight tube 4, as shown in Figure 1(a). Secondly, a deep learning-based stereo image segmentation method is proposed for the binocular images collected in situ. By building a continuous crystallization image segmentation database, an instance segmentation network is obtained by transfer learning training to accurately extract the crystal image mask and contour. Then, for the identified segmented crystal image pairs, their length and width feature directions and the corresponding interest point sets are extracted. Based on this, a crystal interest point matching algorithm is proposed to match the interest points in the crystal image pairs and construct a set of matching interest points related to the length and width of the crystal. Finally, the above matching point set is reconstructed through an optimal point three-dimensional reconstruction method, and the length and width three-dimensional feature lines are calculated, thereby quantitatively evaluating the three-dimensional size of the crystal.
[0045] The designed binocular backlight imaging system is a non-contact industrial in-situ monitoring device for monitoring the continuous oscillating baffle crystallizer (COBC). This crystallizer includes: 15 jacketed glass straight tubes 4 with an inner diameter of 15 mm (each straight tube has periodically spaced baffles and forms 25 internal chambers), 15 glass bends, 4 circulating temperature control devices (control the temperature of the four tube sections of COBC) and 2 jacketed stirring tanks (the crystallization solute and seed are sent into COBC through peristaltic pumps). In order to achieve in-situ measurement of crystals suspended and flowing in COBC, a binocular backlight imaging system is designed, which includes two industrial-grade high-resolution color CMOS cameras connected to a computer, and also includes two telecentric backlight illuminators 2 to provide uniform backlight illumination, which are located on both sides of the monitored COBC straight tube 4. During the monitoring process, the binocular telecentric camera 1 is placed at a distance of about 95 mm from the outer glass wall of the straight tube 4, and ensures that the focus area is located in a single chamber A of the straight tube 4. In addition, through the synchronization trigger, the computer serial port sends a synchronization acquisition signal to the binocular telecentric camera 1 to obtain the binocular images (left view and right view) during the continuous crystallization process, and the data acquisition card performs rapid acquisition. Finally, the proposed deep learning-based stereo image analysis method is used to obtain the three-dimensional dimensions of the crystal length and width. The whole measurement process is as follows: Figure 2 shown.
[0046] The method specifically comprises the following steps:
[0047] Step 1: Design of binocular backlight imaging system
[0048] The first step is to build a binocular telecentric camera 1 system. The camera uses a Daheng 2 / 3" global exposure color CMOS sensor, which is compact, light, sturdy and durable; the telecentric lens has a fixed magnification and low imaging distortion. Its orthogonal projection imaging characteristics are particularly suitable for accurately measuring crystal targets at the micrometer scale. Therefore, two Dehong double object space telecentric lenses are selected and connected to the two cameras through the C port to form a microscopic binocular telecentric camera 1 system. When shooting, each pixel of this system corresponds to an actual length of 1.75μm, and the image pixel of 2448×2048 corresponds to 4.2× 3.5mm field of view. Each telecentric camera is mounted on a four-axis micro-stage B to achieve high-precision XYZ-axis translation and vertical rotation, so that the two cameras can accurately focus on the same imaging area from different angles. It should be noted that the binocular telecentric camera 1 is arranged horizontally (the camera optical axis angle is about 15.61°), so that the baseline of the binocular camera is parallel to the horizontal axis of the COBC straight tube 4, which is convenient for in-situ measurement and minimizes the distortion and optical refraction effects caused by the bending of the crystallizer glass tube wall during imaging. Both cameras are connected to the industrial control computer through a high-speed USB 3.0 port, and through a customized camera trigger, the two cameras can simultaneously capture crystal solution images at a maximum frame rate of 36Hz.
[0049] Then, two telecentric backlight illuminators 2 are constructed. Since the backlight light source is conducive to improving the contrast between the outline of the object under test and the background, it is more suitable for the measurement of micron-level targets such as crystals. The two designed telecentric backlight illuminators 2 are connected by two LED light sources and two telecentric lenses, allowing only uniform parallel light beams to pass through the COBC straight tube 4 and enter the binocular telecentric camera 1, thereby producing high-quality images of crystal particles. At this time, the two telecentric backlight illuminators 2 and the binocular telecentric camera 1 are placed on both sides of the imaging crystal respectively. Each designed illuminator outputs monochromatic green light with a peak wavelength of 521nm to alleviate chromatic aberration and diffraction limit. The beam diameter of both illuminators is 30mm. In addition, both illuminators are installed in the light source bracket C, which is easy to move and rotate so that the parallel light beams are aligned with the optical axes of the two camera lenses.
[0050] Finally, a long-stroke four-axis optical platform 3 is built. The optical platform is used to install the above-mentioned binocular telecentric camera 1, two telecentric backlight illuminators 2 and their fixing parts. The specific functions are: through the displacement of the X-axis D (the stroke is 400mm), the measurement area of the binocular telecentric camera 1 is moved along a straight tube 4 of COBC; through the displacement of the first Y-axis E (the stroke is 400mm), the binocular telecentric camera 1 is roughly sent into its working distance (105mm); then through the precise displacement of the second Y-axis F, the binocular camera is focused so that its common focus area can capture a clear crystal image; through the displacement of the Z-axis G, the binocular telecentric camera 1 is used to move in the vertical direction to monitor the straight tubes of different layers of COBC.
[0051] It should be noted that when using the binocular backlight imaging system designed above for in-situ imaging, two cameras need to capture crystal particles in the flowing suspension at the same time. The camera exposure time is set to 10 μs, which effectively avoids motion blur. When performing subsequent crystal image processing, the G component in the original RGB image is extracted for analysis.
[0052] Step 2: In-situ stereo imaging calibration
[0053] Before collecting binocular crystal images for analysis, the above binocular telecentric camera 1 parameters need to be accurately calibrated in advance. Since the telecentric camera has the orthogonal projection characteristic, a three-dimensional world point Q (x, y, z) and its two-dimensional image point q in the two cameras L (u L ,v L ) and q R (u R ,v R ) is represented as follows:
[0054]
[0055] Among them, I i and (R i ,t i ) are the camera’s internal and external parameter matrices respectively; C i represents the camera projection matrix in affine form (the last line is (0,0,0,1)); i∈{L,R} denotes the left and right camera sources. In order to simplify the binocular image matching, the telecentric epipolar correction is introduced to transform the binocular images so that the epipolar lines in each image are parallel and horizontal. Therefore, the corrected camera projection matrix C′ L and C′ R It is expressed as:
[0056]
[0057] Among them, q′ L and q′ R Represents the matching point pairs in the rectified stereo image.
[0058] In order to determine the model parameters of the two cameras in equations (1) and (2), an in-situ calibration device as shown in Figure 1(b) is used for stereo imaging calibration. The microscale checkerboard calibration plate 5 is inserted into the process access port H of the COBC straight tube 4 so that it is placed in the chamber A between the baffles. The calibration plate is rotated and moved in the common field of view of the binocular telecentric camera 1, and several sets of checkerboard pose image pairs are collected for in-situ calibration. During the specific calibration, the initial values of the internal and external parameters of the camera are first obtained through a closed-loop solution, and then the above parameters and the distortion coefficient (imaging distortion caused by the lens and the glass wall of the crystallizer) are further refined and adjusted through nonlinear optimization. Thereafter, the initial and corrected binocular camera parameter matrix can be used for three-dimensional reconstruction of the crystal.
[0059] The third step is crystal image segmentation
[0060] Accurate image segmentation of the acquired in-situ binocular images is an indispensable step in extracting the crystal interest point set. The image segmentation technology based on deep learning does not rely on manually extracted features, and can obtain more comprehensive underlying image features through training, which is conducive to improving the accuracy and robustness of crystal image segmentation. Therefore, the present invention adopts the recently published Mask2Former instance segmentation network with excellent performance to extract the pixel-by-pixel mask information of the crystal image. The overall structure is as follows: Figure 3 As shown, it includes three modules with different functions, namely backbone feature extractor, pixel decoder and Transformer decoder.
[0061] Specifically, the backbone module is used to extract low-resolution features of the entire input image. The feature map generated at this time is usually small in size, each pixel has a large receptive field, and contains rich semantic information related to the crystal object. The pixel decoder module is then used to gradually upsample the above feature map to generate multi-scale high-resolution features, and finally obtain a feature pyramid with a resolution of 1 / 32, 1 / 16, and 1 / 8 of the original image. Therefore, the detection effect of crystals of various sizes can be improved, especially for small crystal particles in the first few straight tubes of COBC. Finally, the multi-scale image features are sequentially input into the Transformer decoder layer, and the mask attention mechanism of this module is used to constrain the attention to the predicted mask area of each query instead of searching on the entire feature map.
[0062] In order to train the Mask2Former network, a custom database of 600 continuous glutamate crystallization images taken under different experimental conditions and in different straight tube regions was constructed, of which 480 were randomly selected for training and the remaining 120 were used for validation. The target crystals in the database were annotated with polygons using the open source software Labelme to depict the crystal outlines. In addition, some supervised data augmentation strategies were used to improve the robustness and generalization ability of the Mask2Former model, including flipping, blurring, random size cropping, affine transformation, brightness and contrast adjustment. Ultimately, the total number of annotated crystal images in the training set was expanded to more than 100,000, which was used to effectively train and validate the Mask2Former model.
[0063] Based on the constructed image database, when training the Mask2Former model, the deviation between the predicted results and the true labels is minimized, that is, the training loss function, which can be defined as:
[0064]
[0065] in, is the cross entropy loss in the classification task; and and They are the binary cross entropy loss and dice loss in binary images, respectively, which together constitute the segmentation mask loss.
[0066] During training, the precision, recall and average precision commonly used in target detection and instance segmentation are used to quantitatively evaluate the performance of the training model. By setting appropriate rounds, the model with the highest average precision is the crystal image segmentation model used in the present invention.
[0067] Step 4: Extract and match the crystal length and width interest point set
[0068] After using the optimal Mask2Former model to detect the binocular images collected in situ and obtain the pixel-by-pixel mask of the crystal instance, the minimum bounding rectangle method is then used to enclose the outline of each crystal image and identify the crystal pairs that appear simultaneously in the two images, i.e., the crystal image matching pairs. At the same time, taking the left image of a pair of crystal images as an example, the length and width of the left image of the crystal can be calculated by the angle between the long and short sides of its bounding rectangle and the horizontal axis. The feature directions of the length and width extracted are as follows: Figure 4 (a) is shown by the dashed and dotted lines. Therefore, the two-dimensional characteristic length of the left image of the crystal can be determined by the maximum distance between two pixel points on the crystal contour parallel to the length characteristic direction. Similarly, the two-dimensional characteristic width of the crystal can be quantified along the width characteristic direction of the crystal. Therefore, for the left image in a crystal image pair, its contour points are evenly divided along its length and width characteristic directions, and multiple characteristic lines representing the length and width of the left image of the crystal can be obtained, such as Figure 4 (b) At the same time, the intersection of these feature lines with the left crystal image contour is recorded as the interest point set of its length and width and like Figure 4 (b) is indicated by the triangle marks of the same color.
[0069] In order to obtain the characteristic length and width of the sample crystal in three-dimensional space, it is necessary to find the left interest point set and The set of matching points in the right image of the crystal and Therefore, the present invention proposes a two-step crystal interest point matching algorithm, which imposes two matching constraints in sequence: 1) using a telecentric epipolar constraint to reduce the search area for establishing an initial matching set; 2) using an intensity constraint to further select point correspondences that meet the minimum intensity difference criterion.
[0070] (1) Telecentric polar line constraint:
[0071] by Take an interest point in the image as an example, and calculate the vertical coordinate distance d(i) between its epipolar line and the contour point of the right image of the crystal:
[0072] d(i)=|v Ri -v epipolar |,i=1,2,...,n / 2 (4)
[0073] Among them, v epipolar is the vertical coordinate of the polar line; v Riis the vertical coordinate of the right mask contour point; n represents the number of right mask contour points. The right contour point that satisfies d(i) = 0 is a possible matching point. Considering the possible image segmentation error, the right contour point and the other four pixels extending horizontally on the left and right sides constitute the initial matching set. It should be noted that when multiple points satisfy d(i) = 0, it indicates that this section of the crystal contour is parallel to the polar line, and the initial matching set will be further expanded accordingly. Figure 5 (a) illustrates the two types of initial matching sets mentioned above.
[0074] (2) Pixel intensity constraint:
[0075] Next, for the initial matching set obtained in the first step, the true matching point pairs are determined by the intensity difference between the subsets in the left (reference) image and the right (target) image. Considering the influence of the brightness change of the binocular image caused by the difference in the binocular camera's viewing angle, the zero-mean normalized minimum square distance (ZNSSD) matching criterion is used to calculate the pixel value difference between the subsets:
[0076]
[0077] in, r j and t j denote the intensity value of the jth pixel in the reference subset and the target subset respectively; and denote the average intensity values of all pixels in the reference subset and the target subset, respectively. Therefore, by finding the χ between the subsets in the initial matching set ZNSSD The minimum value is used to determine the correct point correspondence.
[0078] By analogy, all matching interest point pairs related to length and width in a crystal image pair can be obtained, that is, and and like Figure 5 (b)-(c) serve as the basis for the three-dimensional measurement of the length and width of the crystal.
[0079] Step 5: Three-dimensional measurement of crystal size
[0080] Based on the above-obtained crystal length and width interest point set, one of the matching interest point pairs is still taken as an example. The three-dimensional position of the interest point can usually be obtained by the intersection of two corresponding rays from two cameras, where each ray passes through the optical center of the telecentric camera (located at infinity) and the corresponding pixel position in the camera. However, due to the existing image noise and camera calibration errors, the two rays may not intersect in three-dimensional space. Therefore, the present invention proposes a three-dimensional reconstruction method of the optimal point based on an analytical solution.
[0081] Figure 6 A schematic diagram of the method for 3D reconstruction in the world coordinate system is given, where O′ LC -X L ' C Y L ' C Z L ' C and O′ RC -X′ RC Y R ' C Z′ RC Respectively represent the left and right camera coordinate systems after telecentric epipolar correction; q′ L and q′ R represents the point of interest pair in the world coordinate system after correction. Without loss of generality, the present invention defines the left camera coordinate system after correction as the world coordinate system. The optimal three-dimensional point Q to be reconstructed is located at the shortest line segment |Q between the two rays L Q R |, and the reprojection error is minimized. The specific derivation process is as follows.
[0082] q′ L Q L and q′ R Q R Represents two rays in the world coordinate system, P L and P R Represent the three-dimensional point Q L and Q R In q′ L Q L and q′ R Q R Position on:
[0083]
[0084] Among them, o′ L and o′ R They are q′ L Q L and q′ R Q R The unit direction vector of the telecentric camera. LC and O′ RC Located at infinity, the light rays in the left and right views are parallel to the optical axes (i.e., Z-axis) of the left and right cameras, respectively. Therefore, o′ L and o′ R We can distinguish from R′ L and R′ R is determined in the third row of .
[0085] Let P L =P R, at this time the shortest distance between the two rays |Q L Q R | can be calculated by the least squares method:
[0086] [|q′ L Q L | |q′ R Q R |] T =[o′ L -o′ R ] + (q′ R -q′ L ) (7)
[0087] in,[] + represents the left pseudo-inverse of a matrix; P L and P R According to |q′ L Q L | and |q′ R Q R |Get is also determined.
[0088] Using the obtained |Q L Q R |, assuming that the optimal point Q has the minimum reprojection error and satisfies:
[0089] |Q L Q|=w|Q L Q R | (8)
[0090] Where w represents the scale factor, and the reprojection error of Q can be defined as:
[0091]
[0092] Among them, q′ L,repro and q′ R,repro are the reprojection points of Q in the two cameras respectively.
[0093] Then, two Euclidean distances are calculated and in It's Q R Reprojected point in the left camera; It's Q L The reprojected point in the right camera. Since all rays from each camera are parallel, there exists |q′ L,repro -q′ L |=wd 1 and |q′ R,repro -q′ R |=(1-w)d 2, formula (9) can be rewritten as:
[0094] e 2 =w 2 d 1 2 +(1-w) 2 d 2 2 (10)
[0095] The optimal value of w can be obtained by minimizing formula (10):
[0096]
[0097] Once w is known, the optimal three-dimensional coordinate point Q(x,y,z) can be calculated as:
[0098] Q(x,y,z)=(1-w)P L +wP R (12)
[0099] Based on the above interest point reconstruction method, the optimal three-dimensional coordinates of all interest point pairs related to the length and width of the crystal can be effectively calculated. Figure 4 The three-dimensional dimensions of the multiple characteristic lines representing the length and width of the crystal shown in (b) can be measured by the Euclidean distance in three-dimensional space, respectively. Specifically, the three-dimensional characteristic length of a crystal is determined by simply finding the longest spatial distance between all length characteristic lines, such as Figure 7 (a) shows the thick solid line and the two triangles marking the points of interest. It should be noted that the actual crystal contour may have protrusions or depressions, which will affect the edge regularity of its width measurement. In order to ensure the accuracy of the width measurement, the present invention adopts a measurement method based on the histogram of all width characteristic lines to measure the width from a statistical perspective. Therefore, the three-dimensional characteristic width of the crystal is determined by the middle value of the rectangle with the largest frequency in the width histogram, such as Figure 7 (b) and (c), where the three-dimensional width values of the exemplified crystals are as follows Figure 7 By counting the three-dimensional length and width of multiple pairs of crystal images, the size distribution of crystal length and width within a certain period of time can be formed, thereby obtaining the most critical process monitoring information in the continuous crystallization process.
[0100] The above-described embodiments merely express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A method for real-time detection of crystal length and width in a continuous crystallization process by binocular backlight imaging, characterized in that The following steps are involved: The first step is to establish a binocular backlight imaging system (1) Use the orthographic projection imaging characteristics of the telecentric camera to accurately measure crystal targets at the micrometer scale; select the matching object-side telecentric lens according to the pixel size and sensor size of the high-definition industrial camera used; After installing the two cameras and lenses, they are respectively mounted on a four-degree-of-freedom micro-displacement stage to provide high-precision XYZ-axis translation and vertical rotation of the camera lens system, so that the two camera lens systems can focus on a common imaging area; the baselines of the two telecentric cameras should be parallel to the horizontal axis of the straight tube of the tubular continuous crystallizer, that is, the two cameras are placed horizontally to minimize the optical refraction effect caused by the bending of the crystallizer glass tube wall during imaging; (2) Two telecentric backlight illuminators are placed on both sides of the imaging target together with the binocular telecentric camera. The LED light source is connected to the telecentric lens to allow only uniform parallel light to pass through the tubular crystallizer and then enter the telecentric camera and form an image. Both telecentric backlight illuminators are installed in a customized light source bracket to facilitate movement and rotation so that the parallel light beams are aligned with the optical axes of the two camera lenses respectively. (3) A long-stroke four-axis optical platform for mounting the above binocular telecentric camera, telecentric backlight illuminator and their fixing parts; the optical platform moves the measurement area of the binocular camera along the straight tube direction of the tubular continuous crystallizer through the displacement of the X-axis; the binocular camera is roughly brought into its working distance through the displacement of the first Y-axis; then the binocular camera is focused through the precise displacement of the second Y-axis so that a clear crystal image can be captured in its common focus area; and the binocular camera is moved vertically through the displacement of the Z-axis to monitor the straight tubes at different layers of the crystallizer; In addition, before the actual measurement, the micron-scale checkerboard calibration image collected at the same time is used, combined with the calibration and correction algorithm of the telecentric camera, to establish a stereo imaging model of this binocular backlight imaging system for three-dimensional reconstruction of crystal size; The second step is crystal image segmentation, length and width interest point set extraction and matching (1) In-situ crystal image segmentation and contour extraction based on deep learning: A large number of in-situ crystal images under different continuous crystallization experimental conditions, different detection areas, and different measurement times are collected, and a continuous crystallization crystal image segmentation database is constructed by manual annotation. Using this database, the Mask2Former instance segmentation network is trained to segment the crystal targets of interest from the new in-situ images and extract their corresponding crystal contours; some crystal images that are not statistically significant are deleted; (2) Extraction of crystal length and width interest points: The minimum circumscribed rectangle method is used to enclose the outline of each of the above crystal images. At the same time, on the stereo-corrected binocular image, the crystal image pair in the binocular image is identified based on the horizontal polar line and the coordinate information of the four corner points of the rectangular enclosing frame. The length and width directions of a crystal image can be calculated by the angles between the long and short sides of its enclosing rectangle and the horizontal axis. Therefore, the two-dimensional characteristic length of the crystal image can be determined by any two points on its contour that are parallel to the length characteristic direction of the crystal and have the maximum distance. Similarly, the two-dimensional characteristic width of the crystal can be quantified along the width characteristic direction of the crystal. Therefore, for the left image in a crystal image pair, its contour points are evenly divided along its length and width characteristic directions, respectively, to obtain multiple two-dimensional characteristic lines of length and width in the left image of the crystal. The intersections of these lines with the left crystal contour can be classified as a set of interest points for its length and width. and In order to calculate the characteristic length and width of a crystal in three-dimensional space, it is necessary to find the left interest point set and The set of matching points in the right image of the crystal and A crystal interest point matching algorithm under two matching constraints is proposed below; ①Telecentric polar line constraint: by Take an interest point in the image as an example, and calculate the vertical coordinate distance d(i) between its epipolar line and the contour point of the right image of the crystal: d(i)=|v Ri -v epipolar |, i=1,2,...,n / 2 (1) Among them, v epipolar is the vertical coordinate of the polar line; v Ri is the vertical coordinate of the right contour point; n represents the number of right contour points; when d(i)=0, the right contour point is a possible matching point; considering the possible image mis-segmentation, the right contour point and the other four points extending horizontally from it constitute the initial matching set; ②Pixel intensity constraint: The true matching point pairs in the initial matching set are found by the intensity difference between the subsets in the left (reference) image and the right (target) image; the zero-mean normalized minimum square distance (ZNSSD) matching criterion is used to reduce the influence of different brightness in the binocular images: in, r j and t j are the intensity values of the jth pixel in the reference subset and the target subset, respectively; and are the average intensity values of all pixels in the reference subset and the target subset, respectively; therefore, the χ between the subsets in the initial matching set can be found ZNSSD Minimum value is used to determine the correct matching point correspondence; Similarly, all matching points of interest related to length and width in a crystal can be obtained, that is, and and As the basis for three-dimensional measurement of the length and width of the crystal; Step 3: 3D measurement of crystal size Based on the above-obtained crystal length and width interest point set, one of the matching interest point pairs is still taken as an example: the three-dimensional coordinates of the interest point pair in space can be obtained by the intersection position of its forward rays in the two cameras, where each ray passes through the optical center of the telecentric camera (located at infinity) and its respective pixel coordinates; due to image noise and camera calibration errors, the two rays may not intersect in space, so a method based on analytical solution is proposed to calculate the optimal position of the three-dimensional reconstructed coordinates; the positions of the two rays in space are expressed as follows: Among them, q′ L and q′ R is the matching point pair in the world coordinate system after the telecentric polar correction; o′ L and o′ R The light q′ L Q L and q′ R Q R The unit direction vector of ; Let P L =P R , calculate the shortest distance between the two rays at this time |Q L Q R |,q′ L and q′ R The optimal point of 3D reconstruction is located on the line segment |Q L Q R |Top: Q(x,y,z)=(1-w)P L +wP R (4) Where w represents the scale factor, at which point the reconstructed point Q(x, y, z) is optimal and has the smallest reprojection error; Based on the above-mentioned interest point reconstruction method, all matching points in the crystal length and width interest point set are reconstructed, and the three-dimensional characteristic line size representing the length and width of the crystal is obtained by Euclidean distance calculation; therefore, the three-dimensional characteristic length of the crystal can be determined by simply finding the longest spatial distance between all length characteristic lines; in order to ensure the accuracy of the width measurement of needle-shaped or rod-shaped crystals, a measurement method based on the histogram of width characteristic lines is adopted to measure the width from a statistical perspective; therefore, the three-dimensional characteristic width of the crystal is determined by the middle value of the rectangle with the largest frequency in the width histogram.
2. The method for real-time detection of crystal length and width in a continuous crystallization process by binocular backlight imaging according to claim 1, characterized in that: The custom continuous crystallization database in step (1) described in the second step is constructed as follows: first, 600 images of glutamate continuous crystallization taken under different experimental conditions and in different straight tube regions are collected and randomly selected as the custom continuous crystallization image database, of which 480 are randomly selected for training and the remaining 120 are used for model verification; all crystal instances in the database are annotated with polygons using the open source software Labelme to depict the crystal outline; then, some supervised data enhancement strategies are used to improve the robustness and generalization ability of the deep segmentation network, including flipping, blurring, random size cropping, affine transformation, brightness and contrast adjustment, etc.; Ultimately, the total number of annotated crystal images in the training set of the constructed database was expanded to more than 100,000, which was used to effectively train and validate the Mask2Former segmentation model.
Citation Information
Patent Citations
Calibration modeling and crystal length and width angular point three-dimensional reconstruction method for binocular telecentric camera in crystallization detection process
CN116977441A
Stereophotogrammetric method based on binocular vision
US11959744B1
Object detection method, related device and computer storage medium
WO2020103427A1
Cited By
Continuous crystallization quality detection device and method
CN121453768A
