Device for monitoring bacterial colony growth condition of culture dish in real time based on multi-angle high-definition camera

Abstract

The invention belongs to the field of biomedical engineering, and particularly relates to a device for monitoring the growth condition of bacterial colonies in a culture dish in real time based on a multi-angle high-definition camera. A detection module of the device is integrated with three 48,000,000-pixel high-definition cameras which are symmetrically arranged in a one-vertical and two-horizontal manner, and images of a culture dish are acquired from multiple angles. And the image processing system carries an RAFT-Stereo algorithm, adopts a three-camera parallax array and closed-loop calibration strategy, and calculates internal and external parameters of the cameras through OpenCV stereoCalibrate to realize accurate analysis of the bacterial colony thickness. The device can be used for monitoring the size, spatial density, hypha network distribution and three-dimensional morphological parameters of bacterial colonies in the culture dish in real time and with high precision, as well as multi-strain competition dynamics, blue-white spot screening and metabolic activity based on a chromogenic or fluorescent substrate culture medium. The method breaks through the limitation of traditional two-dimensional monitoring, significantly improves the accuracy and repeatability of bacterial colony monitoring, and provides comprehensive and accurate data support for microbiological research.

Description

Technical Field

[0001] This invention belongs to the field of biomedical engineering, specifically relating to a device for real-time monitoring of bacterial colony growth in culture dishes based on a multi-angle high-definition camera. Background Technology

[0002] Microbiological research plays a crucial role in fields such as biomedicine, food safety, and environmental science. Among these, observing the dynamic growth of microbial colonies is a key step in understanding microbial physiological characteristics, screening superior strains, and evaluating drug efficacy. Traditional colony counting and morphological observation often rely on manual microscopy or simple image analysis systems. These methods are not only time-consuming and labor-intensive but also susceptible to subjective factors, resulting in insufficient data accuracy and repeatability, making them unsuitable for the demands of modern high-throughput, high-precision microbiological research.

[0003] To address the efficiency and accuracy issues of traditional colony monitoring methods, automated colony analysis systems have been widely applied in recent years, driven by advancements in image processing and computer vision technologies. For example, patent CN 109342378 B discloses a colony growth status detection device and method based on multimodal imaging technology. This method utilizes multimodal imaging to acquire various information about the colony, aiming for a more comprehensive detection of its growth status. Patent CN 114170598 B proposes a colony height scanning imaging device, an automatic colony counting device and method capable of identifying atypical colonies, which acquires three-dimensional information of the colony through height scanning and combines image processing technology to automatically count and identify atypical colonies. Furthermore, patent CN 119206716 A relates to a method, system, device, and storage medium for identifying single colonies of brewing microorganisms. This technology focuses on identifying single colonies of brewing microorganisms through image analysis. Patent CN 115029209 A provides a colony image acquisition and processing device and method, aiming to improve the acquisition efficiency and processing accuracy of colony images to achieve effective analysis of colonies. These automated systems improve the efficiency and objectivity of colony detection to a certain extent.

[0004] However, existing colony monitoring technologies still have some limitations, particularly in comprehensively acquiring three-dimensional information about colonies. For example, while patent CN 109342378 B mentions multimodal imaging, its specific technical features and implementation details are not clearly stated in the background information provided regarding its ability and limitations in accurately acquiring colony thickness or complex three-dimensional morphological structures. Existing colony monitoring technologies mainly rely on two-dimensional image analysis, which is insufficient to comprehensively and accurately reflect the true growth status of colonies. Especially in scenarios involving complex colony morphology, bacterial interactions, or the need for fine screening, relying solely on two-dimensional data often fails to provide sufficient information. More importantly, images from a single perspective cannot effectively capture changes in colony thickness. This results in significant limitations in screening colonies with specific three-dimensional characteristics, such as screening deep hyphae that produce antibiotics or assessing biofilm thickness, restricting the depth and breadth of research. Therefore, existing technologies still face significant bottlenecks in accurately acquiring three-dimensional information about colonies, especially thickness data, and comprehensively reflecting the true growth status of colonies. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a device for real-time monitoring of bacterial colony growth in petri dishes using multi-angle high-definition cameras. It aims to overcome the limitations of existing technologies where a single camera cannot accurately acquire three-dimensional information and thickness data of the colony, and two-dimensional image analysis cannot comprehensively reflect the complex growth of the colony. This invention introduces three high-definition cameras to acquire image data from petri dishes from multiple angles, and utilizes the resulting parallax to accurately analyze the colony thickness. This enables more precise and real-time monitoring and evaluation of colony size, thickness, hyphal distribution, and morphology in petri dishes, as well as precise monitoring and evaluation of competitive dynamics in multi-species co-culture systems, blue-white screening results after gene editing, and metabolic activity based on chromogenic or fluorescent substrate media.

[0006] This invention is achieved through the following technical solution: A device for real-time monitoring of microbial colony growth in petri dishes using a multi-angle high-definition camera. The device body connects to a PC via a serial port. The device body encapsulates the collected microbial colony growth data from the petri dishes into a communication protocol and transmits it to the PC's host computer COM port via wired or wireless means. The COM port then processes the data and plots the analyzed data into a counting result graph. The device body includes: A petri dish platform is used to place petri dishes and initiate detection programs to create a stable environment for microbial growth. The detection module integrates three high-definition cameras arranged in a symmetrical "one vertical and two horizontal" layout to capture images of the culture dishes from above and the sides, respectively, and provides uniform illumination through a backlight panel; the detection module can detect at least 16 culture dishes simultaneously; the image data collected by the detection module is transmitted to the MCU microcontroller for preliminary processing before being transmitted to the computer; MCU (Microcontroller Unit) is used for control, communication, complex algorithm calculations, and high-speed data transmission, and features low power consumption. A voltage conversion module is used to convert a wide range of externally input voltages into various stable voltage levels. The motor drive module is used to drive the high-definition camera to move horizontally and adjust its position. The automatic temperature control module based on the PID algorithm includes a temperature sensor and a heating plate located inside a U-shaped copper tube. It is used to collect temperature data in real time and make precise adjustments to avoid condensation and ensure that the temperature of the culture dish platform is constant. The wireless Bluetooth pass-through module adopts Bluetooth serial communication technology and is compatible with RS232 protocol to realize bidirectional communication between the host computer and the slave computer. The image processing system employs the RAFT-Stereo deep learning stereo matching method. Through a "three-camera parallax array and closed-loop calibration strategy," it processes multi-angle images to identify diameter changes in individual colonies, monitor colony height, and calculate volume, thereby achieving accurate colony identification and counting. This invention improves upon the existing RAFT-Stereo deep stereo matching framework by employing a three-camera, multi-baseline input structure. Two sets of disparity maps are constructed using the central and side cameras respectively. Confidence weights are built based on cost volume consistency and gradient magnitude, and the two sets of disparities are weighted and fused. Simultaneously, reflection region suppression, illumination equalization, and feature pyramid enhancement modules are introduced to improve the ability to identify colony edges and resolve small disparities. To further enhance 3D reconstruction accuracy, this invention introduces a closed-loop calibration strategy based on reprojection error. This involves projecting the 3D points obtained from multi-view reconstruction and calculating the reprojection deviation. This deviation information is used to optimize disparity bias and extrinsic parameter perturbation, ensuring the stereo matching process continuously converges to a smaller error range. Through these improvements, this invention can obtain sub-pixel-level colony thickness maps, which can then be used for volume estimation and growth curve analysis.

[0007] The data processing system uses the YOLOv8 algorithm to extract features and predict class probabilities from images, and combines nonmaximum suppression technology to optimize the prediction results, outputting accurate reports on colony quantity, location, and class analysis. Simultaneously, it plots microbial counting results curves based on colony appearance time and quantity changes. (Specifically, the image processing system performs depth matching and spatial reconstruction on multi-angle petri dish images acquired by three cameras. By constructing a three-camera parallax array consisting of a vertical camera and two side cameras, it performs depth feature matching and parallax fusion on multi-view images, and iteratively compensates for stereo reconstruction errors based on a closed-loop calibration strategy, thereby obtaining the thickness, volume, and morphological parameters of colonies, achieving accurate colony identification and counting.)

[0008] Specifically, this invention constructs a three-camera parallax array using a vertical camera and two side cameras. Epipolar correction is applied to the acquired multi-angle images, and a dual-baseline parallax map is built based on an improved RAFT-Stereo stereo matching network. Pixel confidence is obtained through cost volume consistency, gradient magnitude, and matching residuals. The depth results of the two baselines are weighted and fused to form a consistent 3D depth map. Based on the fused depth, a reference plane is established by fitting the bottom surface of the culture dish and the colony thickness is calculated. The volume and morphological parameters of the colony region are then obtained by integration. To improve the accuracy of stereo reconstruction, this invention further constructs a closed-loop calibration mechanism based on reprojection error. The 3D points reconstructed from the depth are reprojected onto the multi-camera plane, and the projection error is calculated. The camera extrinsic perturbation and parallax bias are optimized using least squares. The above steps are repeated in each frame to form an iterative compensation process, causing the stereo matching results to converge to a low error range. Finally, accurate colony thickness and volume parameters are output. Preferably, the detection module includes a high-definition camera I, a high-definition camera II, and a high-definition camera III. The high-definition camera I is mounted on the X-axis movable bracket and is mounted on the sliding rail via the X-axis moving frame. It is driven by a stepper motor of the motor drive module. A photoelectric detector is set above the camera movable bracket.

[0009] Preferably, the base is embedded inside the device body, and a culture dish placement platform is provided on the base. The culture dish placement platform is fixed under the moving slide rail. A cell culture plate is also provided on the culture dish placement platform. The cell culture plate is also provided with an X-axis movable frame, a power switch, a first U-shaped photoelectric switch, a second U-shaped photoelectric switch, and a culture plate fixing clip. The power switch and the first U-shaped photoelectric switch are installed on one side of the stepper motor. The first U-shaped photoelectric switch and the second U-shaped photoelectric switch are symmetrically installed on both sides of the base. The culture plate fixing clip is installed on one side of the bottom of the cell culture plate.

[0010] Preferably, both the first U-shaped photoelectric switch and the second U-shaped photoelectric switch are wired to the control main board. The first U-shaped photoelectric switch is combined with the outlet baffle. When the outlet baffle and the photoelectric switch block each other, a successful outlet flag is sent to the culture dish placement platform. The second U-shaped photoelectric switch, combined with the baffle, provides the system with the original position information and the detection start flag.

[0011] Preferably, the bottom of the device body is equipped with a petri dish placement platform moving device, a U-shaped copper tube, a temperature sensor for the petri dish placement platform, a control motherboard, a power interface, an integrated circuit chip, and a data interface. The entire system receives power through the power interface to power the high-definition cameras I, II, and III, the control motherboard, and the integrated circuit chip. The integrated circuit chip in the control motherboard is responsible for processing image data, executing image recognition algorithms, identifying and counting microbial colonies in the petri dishes. The number and information of the identified colonies are transmitted to an external computer through the data interface for data analysis and recording. The cell culture plate is moved on the petri dish placement platform by the petri dish moving device so that the high-definition cameras II and III can capture images of the petri dishes at different heights. The U-shaped copper tube is connected to the temperature sensor and heating element to provide real-time temperature feedback and maintain a set temperature cycle within the petri dish platform to achieve constant temperature culture of the petri dishes.

[0012] Preferably, the MCU microcontroller is an STM32F103VCT6 chip, which has a 32-bit width and a 72MHz main frequency, and has a variety of built-in peripheral interfaces.

[0013] Preferably, the image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate, and the distortion correction error is controlled within ≤0.15 px; the image processing system performs Zhang calibration and epipolar correction using a ceramic chessboard-dot dual-mode calibration board, with a reprojection error ≤0.08 px; the image processing system triggers lateral white light secondary imaging for colonies with a thickness greater than 0.6 mm and verifies it through shadow length, and verifies it using a blue light bottom transmittance OD attenuation model for colonies with a thickness less than 0.2 mm.

[0014] Preferably, the image processing system first performs dual-camera calibration using three cameras to obtain initial extrinsic parameters; then, during the stereo reconstruction process, high-confidence 3D points are selected in real time and projected onto the planes of each camera. A reprojection error function is constructed based on the projection deviation, and nonlinear least squares optimization is used to perturb and correct the camera extrinsic parameters and disparity bias; the extrinsic parameter optimization and deviation compensation are performed iteratively in a closed-loop manner, so that the reprojection error gradually converges, and finally the joint calibration parameters of the three cameras are obtained. The image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate. The specific process is as follows: Before performing dual-camera calibration, Zhang's calibration was performed on the three cameras C0, C1, and C2 using a ceramic checkerboard-dot dual-mode calibration board to obtain their respective intrinsic parameter matrices and distortion parameters. SCSS C0 → K0, D0 C1 → K1, D1 C2 → K2, D2 Where: scss represents the code block label, K0, K1, and K2 are the intrinsic parameter matrices of the three cameras; D0, D1, and D2 are the corresponding distortion coefficients. When stereoCalibrate performs extrinsic parameter estimation for each pair of cameras, the flag CALIB_FIX_INTRINSIC is enabled to ensure that stereoCalibrate no longer optimizes any camera intrinsic parameters or distortion coefficients, thereby ensuring that the three cameras maintain a consistent intrinsic parameter model in all dual-camera calibrations and avoiding calibration drift.

[0015] Step 1: Initial Static Calibration Phase First, using a ceramic checkerboard / dot dual-mode calibration board, dual-calibration was performed on the three cameras (C0, C1, C2) respectively: SCSS (C0, C1) → yields K0, K1, R 01 , T 01 ; (C0, C2) → yields K0, K2, R 02 , T 02 ; (C1, C2) → yields K1, K2, R 12 ,T 12 ; Where scss represents a code block tag, K0 is the intrinsic parameter of camera C0; K1 is the intrinsic parameter of camera C1; K2 is the intrinsic parameter of camera C2; R 01 T 01 =External parameters of C0→C1; R 02 T 02 =External parameters of C0→C2; R 12， T 12 =C1→C2 is an external parameter that is necessary for the three-camera chain geometric constraint.

[0016] Obtain initial calibration results: R 01 (0), T 01 (0); R 02 (0), T 02 (0); R 12 (0), T 12 (0); Among them, R 01 (0) represents the initial value of the rotation matrix from C0 to C1, T 01 (0) represents the initial value of the translation vector from C0 to C1; R 02 (0) represents the initial value of the rotation matrix from C0 to C2, T 02 (0) represents the initial value of the translation vector from C0 to C2; R 12 (0) represents the initial value of the rotation matrix from C1 to C2, T 12 (0) represents the initial value of the translation vector from C1 to C2.

[0017] Step 2: Consistency Constraints of Extrinsic Parameters for Three Cameras Based on the geometric chain relationship of the three-camera system, the following extrinsic parameter consistency condition is established: ; Among them, R 01 T 01 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C1; R 12 T 12 This indicates that the extrinsic parameter of camera C1 points to that of camera C2; R 02 T 02 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C2; Through joint optimization, the following can be achieved: (C0, C1); (C0, C2); (C1, C2); The extrinsic parameters of the three pairs of cameras remain strictly consistent in the same coordinate system; Step 3: Online reprojection error calculation After the image processing system runs, once each depth reconstruction is complete, N high-confidence stable 3D points are selected from the 3D point cloud or depth map: ; in, This represents the i-th 3D point, which is represented by 3D coordinates. Standard Xi, Yi, Zi composition.

[0018] Projected onto the planes of the three cameras respectively: vbnet

[0019] Where vbnet represents the language tag for the code block, It is the projection of camera C0, representing the pixel coordinates of the i-th 3D point projected onto camera C0; It is a projection onto camera C1, representing the pixel coordinates of the i-th 3D point projected onto camera C1. It is the projection of camera C2, representing the pixel coordinates of the i-th 3D point projected onto camera C2; Let "the i-th point" be the reference point. The intrinsic parameter matrices of camera C0 at center K0, camera C1 at K1, camera C2 at K2, and the pose of camera C0 at center R0 and T0 are typically set as the reference of the world coordinate system. Calculate the errors between the three projections and the actual observation point coordinates: ; Where pi is the actual pixel seen, and pi′ is the pixel obtained by projection based on the current extrinsic parameters; And calculate the average error: ; Among them, N is the number of reprojection errors in the tender documents. If E > 0.15 px, the online compensation mechanism will be triggered. Step 4: External parameter perturbation optimization The reprojection error is constructed as a nonlinear least-squares objective function. By fine-tuning the camera's rotation ΔR and translation ΔT, the reprojection error is minimized, enabling online dynamic calibration of the three-camera extrinsic parameters. ; Where ΔR represents the rotation perturbation, which represents a small adjustment to the current camera rotation matrix R; ΔT represents the translation perturbation, which represents a small correction to the current camera displacement vector T; π represents the camera projection function, which means "projecting a 3D point onto a 2D pixel plane"; K represents the intrinsic parameter matrix; R represents the current rotation matrix; T represents the current translation vector; and (P_i) represents the i-th 3D point. Solve for the optimal perturbation using Levenberg–Marquardt or Ceres Solver: nginx R 01 ← R 01 + ΔR 01 ; T 01 ← T 01 + ΔT 01 ; R 02 ← R 02 + ΔR 02 ; T 02 ← T 02 + ΔT 02 ; Where nginx represents the code block language tag, extrinsic parameter perturbation update, ΔR 01 It is the camera's rotation correction for (C0→C1), ΔT 01 It is the camera's translation correction for (C0→C1), ΔR 02 It is the camera's rotation correction for (C0→C2), ΔT 02 It is the camera's translation correction for (C0→C2); Step 5: Dynamic parallax offset compensation The depth error of stereo matching is decomposed as follows: ; Wherein, the original disparity d raw There is a systematic error, and this step obtains the bias through geometric error feedback. Δd After compensation, a more accurate true parallax is obtained. d true ; The system estimates the parallax bias based on the reprojection error: ; in, , This represents the sensitivity of depth Z to parallax d. E is the average reprojection error, representing the pixel deviation of 3D points projected onto the image; and the depth is updated accordingly. ; The original depth Zraw is dynamically corrected by parallax bias Δd to obtain a more accurate depth Zcorrected, which is used for 3D reconstruction and extrinsic parameter closed-loop optimization. Here, ΔZ represents the depth correction amount. Step Six: Dynamic Closed-Loop Iteration The entire system continuously executes the following during operation: Deep Reconstruction → Reprojection error estimation → Extrinsic parameter perturbation optimization → Parallax offset compensation → Parameter Update → A new round of reconstruction Until satisfied: Or Δ parameter < threshold (0.0001) Ultimately, this forms an adaptive, self-correcting closed-loop calibration system.

[0020] Preferably, the data processing system employs the RAFT-Stereo deep learning stereo matching method. This method utilizes a binocular combination consisting of a central camera and left and right cameras forming two different baselines. Through multi-baseline disparity estimation, disparity confidence modeling and fusion, and reflection suppression and feature enhancement mechanisms, high-precision 3D reconstruction is achieved. This method is particularly suitable for 3D measurement and depth recovery in scenarios with low texture, reflection, blurred edges of biological colonies, and minute-scale depth gradient changes.

[0021] Specifically as follows: (1) Introduction of multi-baseline input structure The following three cameras are deployed: -Center camera C0; - The left-side cameras C1 and C0 form a short baseline B01; - The right-side cameras C2 and C0 form a long baseline B02; Using the same central camera as a reference can effectively reduce the cumulative error caused by three independent viewpoints and improve the geometric consistency of multiple views. (2) Independent calculation of the two sets of disparity While maintaining the original distributed gradient iteration and cost volume construction mechanism of RAFT-Stereo, disparity is solved for the two baselines respectively: ; in: -D 01 Short baseline parallax results in high precision but is susceptible to interference in reflective or low-texture areas; -D 02 Long baseline parallax offers higher depth resolution but requires greater robustness in matching. The two complement each other and can form multiple baseline redundancy information, which significantly improves the stability of depth estimation.

[0022] (3) Modeling based on the confidence level of cost volume consistency and gradient magnitude For the disparity obtained for each baseline, confidence weights are constructed based on the following metrics: Cost body consistency Measure the convergence and internal consistency of cost volume updates during the bidirectional iteration of RAFT; gradient magnitude It reflects the local texture complexity and feature strength, and the disparity reliability is higher in high gradient regions.

[0023] Calculate the weights based on the above indicators. : ; Where f can be a normalized exponential function, the output of a confidence network, or a piecewise linear function; CV 01 G is the cost volume consistency index for disparity D01. 01 It is the gradient intensity of the pixel corresponding to C0→C1; CV 02 G is the cost-volume consistency index for disparity D02. 02 C0→C2 represents the gradient intensity of the corresponding pixel; CV, the cost volume, represents the cost distribution of each pixel matching different disparity values ​​in RAFT or other stereo matching methods; the sharper the cost volume and the more obvious the peak, the more reliable the matching; conversely, the matching uncertainty is high and the confidence is low; G, the gradient magnitude, measures the local texture intensity of the image or feature map; in low-texture regions, the gradient is small, the matching is prone to errors, and the confidence should be reduced; high gradient regions are more reliable. (4) Parallax fusion model Two sets of disparities are fused using a confidence-weighted method: ; (5) Use local RANSAC or least squares fitting for noisy regions; In the fused parallax map, for areas prone to local anomalies such as reflections, liquid surface refraction, and colony edges, the following is introduced: - Local RANSAC plane fitting; where local least-squares smooth fitting.

[0024] Preferably, the data processing system employs U-Net++ as the core segmentation model, optimizing it to address the issues of blurred colony boundaries and severe adhesion in petri dishes, thus improving edge segmentation accuracy. To further distinguish adhered colony individuals, Mask R-CNN is introduced for instance-level segmentation, combined with SAM (Segment Anything Model) for interactive fine-tuning. The system extracts color, texture, and shape features from colony images using convolutional neural networks. During instance segmentation, Non-Maximum Suppression (NMS) is used to remove redundant detection boxes, ultimately outputting colony quantity, area, location, and morphological parameters. A classification model is integrated, allowing for expanded support for colony category prediction. The system also supports visualization of time-series-based colony counting results, dynamically plotting microbial growth curves to achieve accurate analysis of colony quantity, spatial distribution, and morphological characteristics. The "three-camera parallax array and closed-loop calibration strategy" involves using a multi-view parallax array composed of three cameras (vertical (C0) and two side (C1, C2)) to obtain the parallax map for each pair of views through stereo matching, and then performing multi-parallax fusion + Closed-loop calibration with minimized reprojection error iteratively compensates for extrinsic / intrinsic errors and parallax bias, thereby obtaining accurate estimates of colony depth (thickness) and volume, and improves stability by using confidence weighting and time smoothing.

[0025] Preferably, the voltage conversion module is located on the main control circuit board and is used for voltage conversion. The main control board voltage control circuit is composed of a URA2409 chip. The overall input 24V power supply voltage is converted to ±12V voltage by related circuits. Then, through the second and third stage voltage conversion circuits, the 12V voltage is further converted to 5V and 3.3V to ensure that the main control board and other peripheral sensors and circuit boards connected to the main control board can operate normally.

[0026] Compared with the prior art, the advantages of the present invention are as follows: 1. This device is equipped with three high-definition cameras arranged in a symmetrical "one vertical and two horizontal" configuration. Combined with advanced image processing technology, it ensures accurate identification and counting of microbial colonies. More importantly, through multi-angle parallax data, this invention can analyze three-dimensional information such as colony thickness, greatly improving the accuracy and repeatability of monitoring results and breaking through the limitations of traditional two-dimensional monitoring.

[0027] 2. This device can capture the growth status of microbial colonies in real time and quickly generate detailed data, including counting curves and statistical reports. This provides researchers and quality control departments with immediate and comprehensive data support, helping to adjust experimental protocols or production processes in a timely manner.

[0028] 3. The integrated PID-based automatic temperature control module ensures a constant temperature in the culture environment. This advantage provides ideal and stable growth conditions for microorganisms, thus guaranteeing the reliability and consistency of experimental results. Attached Figure Description

[0029] The invention will now be further described with reference to the accompanying drawings.

[0030] Figure 1 This is a system block diagram of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 2 This is a schematic diagram of a real-time analysis system for a device based on a multi-angle high-definition camera to monitor the growth of bacterial colonies in a petri dish in real time, as proposed in this invention. Figure 3 This is an internal structural diagram of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 4 This is a schematic diagram of the culture platform structure proposed in this invention; Figure 5 A schematic diagram of the bottom structure of a detection and analysis system for a device based on a multi-angle high-definition camera to monitor the growth of bacterial colonies in a petri dish in real time, as proposed in this invention; Figure 6This is a schematic diagram of the detection module structure proposed in this invention; Figure 7 This is a software control flowchart of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 8 This is a pin diagram of the STMF103VCT6 main control chip of the device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 9 This invention relates to a main control board voltage conversion circuit for a device that uses a multi-angle high-definition camera to monitor the growth of bacterial colonies in a petri dish in real time. Figure 10 This is a PCB circuit diagram of the main control board of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 11 This is a schematic diagram of the data interaction circuit of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 12 This is a data acquisition PCB circuit diagram of a device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera, as proposed in this invention. Figure 13 This is a circuit diagram of a constant temperature reading device for a device based on a multi-angle high-definition camera to monitor the growth of bacterial colonies in a petri dish in real time, as proposed in this invention. Figure 14 This invention relates to a device for real-time monitoring of bacterial colony growth in petri dishes using a multi-angle high-definition camera, which includes a temperature adjustment button. Figure 15 This is a data output image of a device proposed in this invention that uses a multi-angle high-definition camera to monitor the growth of bacterial colonies in a petri dish in real time. Figure 16 This is a schematic diagram of the structure of a device proposed in this invention for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera.

[0031] The components are as follows: 1. Main body; 2. Serial port; 3. PC terminal; 4. Base; 5. Culture dish placement platform; 6. High-definition camera I; 7. High-definition camera II; 8. Stepper motor; 9. Sliding rail; 10. High-definition camera III; 11. Heating element; 12. Cell culture plate; 13. X-axis movable frame; 14. Power switch; 15. First U-shaped photoelectric switch; 16. Second U-shaped photoelectric switch; 17. Culture table fixing clip; 18. Camera bracket; 19. Data interface; 20. Integrated circuit chip; 21. Power interface; 22. Control motherboard; 23. Culture dish placement platform moving device; 24. U-shaped copper tube; 25. Temperature sensor; 26. Base; 27. Photodetector. Detailed Implementation

[0032] To more clearly illustrate the technical solutions and embodiments of the present invention, the accompanying drawings will be briefly described below. These drawings are intended to provide a further understanding of the present invention and serve as part of the specification. They are used in conjunction with specific embodiments of the present invention to jointly explain the invention and do not constitute a limitation thereof. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0033] like Figure 1 , 2 As shown, the main body 1 of the device is connected to the PC terminal 3 via serial port 2.

[0034] The body 1 includes: A petri dish platform is used to place petri dishes and initiate detection programs to create a stable environment for microbial growth. The detection module integrates three high-definition cameras arranged in a symmetrical "one vertical and two horizontal" layout to capture images of the culture dishes from above and the sides, respectively, and provides uniform illumination through a backlight panel; the detection module can detect at least 16 culture dishes simultaneously; the image data collected by the detection module is transmitted to the MCU microcontroller for preliminary processing before being transmitted to the computer; MCU (Microcontroller Unit) is used for control, communication, complex algorithm calculations, and high-speed data transmission, and features low power consumption. A voltage conversion module is used to convert a wide range of externally input voltages into various stable voltage levels. The motor drive module is used to drive the high-definition camera to move horizontally and adjust its position. The automatic temperature control module based on the PID algorithm includes a temperature sensor and a heating plate located inside a U-shaped copper tube. It is used to collect temperature data in real time and make precise adjustments to avoid condensation and ensure that the temperature of the culture dish platform is constant. The wireless Bluetooth pass-through module adopts Bluetooth serial communication technology and is compatible with RS232 protocol to realize bidirectional communication between the host computer and the slave computer. The image processing system employs the RAFT-Stereo deep learning stereo matching method. It utilizes a "three-camera parallax array and closed-loop calibration strategy" (a multi-view parallax array composed of three cameras—vertical (C0) and lateral (C1, C2)—is used to obtain the parallax map for each pair of views through stereo matching. Then, through multi-parallax fusion and closed-loop calibration that minimizes reprojection error, iterative compensation for extrinsic / intrinsic parameter errors and parallax deviation is applied to obtain accurate colony depth (thickness) and volume estimates. Confidence weighting and time smoothing are used to improve stability). This process processes the acquired multi-angle images to identify diameter changes in individual colonies, monitor colony height, and calculate volume, achieving accurate colony identification and counting. The image processing system calculates the intrinsic and extrinsic parameters for each pair of cameras using OpenCVsteroCalibrate, with distortion correction error controlled to ≤0.15 px. The system uses a ceramic checkerboard-dot dual-mode calibration board for Zhang calibration and epipolar correction, with reprojection error ≤0.08 px. The system also supports images with a thickness greater than 0.6 mm. Colonies with a thickness of 0.2 mm were used to trigger lateral white light secondary imaging and were verified by shadow length. For colonies with a thickness of less than 0.2 mm, the blue light bottom transmittance OD attenuation model was used for verification.

[0035] The data processing system uses the YOLOv8 algorithm to extract features and predict class probabilities from images, and combines nonmaximum suppression technology to optimize the prediction results, outputting accurate reports on colony quantity, location, and class analysis. At the same time, it plots microbial counting results curves based on the appearance time and quantity changes of colonies.

[0036] Specifically, the data processing system, based on an improved YOLOv8 deep learning object detection network, performs feature extraction and object detection on petri dish images acquired from multiple angles. The system first preprocesses the input images with grayscale normalization and background suppression. Then, through a C2f-structured feature extraction network and a multi-scale feature fusion module, it outputs the spatial location, diameter estimate, and class confidence score for each candidate colony. The detection results are optimized using non-maximum suppression (NMS) to remove duplicate or clustered detection boxes, ultimately generating a table showing the colony count, location coordinates, class probability, and confidence score. The system dynamically tracks the detection results based on time series data, plotting colony growth curves and distribution heatmaps to achieve automated quantitative analysis of colony growth dynamics.

[0037] Furthermore, the data processing system employs U-Net++ as the core segmentation model, optimizing it to address the issues of blurred colony boundaries and severe adhesion in petri dishes, thereby improving edge segmentation accuracy. To further distinguish adhered colony individuals, MaskR-CNN is introduced to achieve instance-level segmentation, combined with SAM (Segment Anything Model) for interactive fine-tuning. The data processing system extracts color, texture, and shape features from colony images using convolutional neural networks, employing non-maximum suppression (NMS) to remove redundant detection boxes during instance segmentation, ultimately outputting colony quantity, area, location, and morphological parameters. It integrates with classification models, enabling expanded support for colony category prediction. The system also supports visualization of time-series-based colony counting results, dynamically plotting microbial growth curves to achieve accurate analysis of colony quantity, spatial distribution, and morphological characteristics.

[0038] More specifically, this invention uses a PC-based host computer as the user interface, providing not only experimental setup and data collection but also result display and storage functions. The MCU microcontroller, as the core of the system, is responsible for parsing host computer instructions and precisely controlling the connected hardware modules to ensure accurate execution of the experimental process. The automatic temperature control system uses a DS18B20 temperature sensor to acquire temperature data inside the sample carrier device. Through heating elements and the DS18B20 temperature sensor, the internal temperature of the petri dish platform is monitored and adjusted in real time, providing stable temperature conditions for the growth of microbial colonies in the petri dish. Motor-driven control of mechanical components, such as automatic sample replacement or adjustment of sampling position, reduces human intervention and improves experimental efficiency. A U-shaped photoelectric switch is used to determine the position of the chamber entering / exiting and the detection start position. In addition, the system includes an image receiving and processing module and a data processing module to perform necessary processing and analysis on the raw images, ultimately generating easy-to-understand reports and charts.

[0039] like Figure 2 As shown, the device body 1 of this invention, which is based on a multi-angle high-definition camera to monitor the growth of microbial colonies in petri dishes in real time, is connected to a PC 3 via two serial ports. The device encapsulates the microbial colony growth data collected by the instrument into a communication protocol and transmits it to the PC's COM port for reception. The COM port then processes the data and plots the analyzed data into a counting result graph. Alternatively, a wireless Bluetooth or network module can be used to achieve wireless data transmission.

[0040] like Figure 3As shown, the base 4 is embedded inside the instrument. The frame includes a petri dish placement platform 5, a high-definition camera I6, a high-definition camera II7, a high-definition camera III10, a stepper motor 8, a moving track 9, and a heating element 11. The high-definition camera I6 is the main camera (C0), which is vertically fixed directly above the petri dish platform to obtain a 0° baseline image. The high-definition cameras III10 and II7 are auxiliary cameras C1 and C2, respectively, positioned at the center of the left side and the center of the right side of the petri dish placement platform 5 at 90° and 270°, forming a 120° baseline. Specifically, a culture dish placement platform 5 is mounted on the base 4 and fixed under the sliding rail 9. The culture dish placement platform 5 is mounted on the cell culture plate 12. The cell culture plate 12 is equipped with an X-axis movable frame 13, a power switch 14, a first U-shaped photoelectric switch 15, a second U-shaped photoelectric switch 16, and a culture platform fixing clip 17. The power switch 14 and the first U-shaped photoelectric switch 15 are mounted on one side of the stepper motor 8, and the first U-shaped photoelectric switch 15 and the second U-shaped photoelectric switch 16 are symmetrically mounted on both sides of the base 4. The culture platform fixing clip 17 is mounted on one side of the bottom of the cell culture plate 12. When the power is turned on, the control circuit board is started and performs a self-test to ensure that all components are working properly. A culture dish containing a microbial sample is placed on the culture dish placement platform. Using the sliding rail 9 and the stepper motor 8, the culture dish is moved to a preset position below the camera. The camera lens is adjusted to ensure that the microbial sample in the culture dish is in clear focus, capturing an image of the microorganisms in the culture dish. The captured image is sent to an external computer or processing unit through a data transmission interface.

[0041] like Figure 4 As shown, the culture dish platform includes a cell culture plate 12, an X-axis movable frame 13, a power switch 14, a first U-shaped photoelectric switch 15, a second U-shaped photoelectric switch 16, a culture table fixing clip 17, a high-definition camera I 6, a high-definition camera II 7, a high-definition camera III 10, and a camera bracket 18.

[0042] The stepper motor 8 is connected to the control motherboard via a wired connection. The stepper motor 8 is equipped with a moving track 9, a camera movable bracket 18, etc. The stepper motor 8 and the control motherboard are connected via a wired connection and can communicate and control each other through a serial port or a specific control protocol. The stepper motor 8 is controlled to move along the moving track 9 by programming, thereby driving the camera bracket 18 and the high-definition camera 6 to move.

[0043] Both the first U-shaped photoelectric switch 15 and the second U-shaped photoelectric switch 16 are wired to the control main board. The first U-shaped photoelectric switch 15 is combined with the outlet baffle. When the outlet baffle blocks the photoelectric switch, it sends a successful outlet flag to the culture dish placement platform 5. The second U-shaped photoelectric switch 16 is combined with the baffle to provide the system with the original position information and the detection start flag.

[0044] like Figure 5 As shown, the bottom is equipped with a petri dish placement platform moving device 23, a U-shaped copper tube 24, a temperature sensor 25, a control motherboard 22, a power interface 21, an integrated circuit chip 20, and a data interface 19. The entire system receives power through the power interface 21 to power the high-definition cameras I 6, II 7, III 10, the control motherboard 22, the integrated circuit chip 20, and other electronic components. The integrated circuit chip 20 in the control motherboard 22 is responsible for processing image data, executing image recognition algorithms, and identifying and counting microbial colonies in the petri dish. The number of identified colonies and related information can be transmitted to an external computer 3 through the data interface 19 for further data analysis and recording. The petri dish on the cell culture plate 12 can be moved by the petri dish placement platform moving device 23 (in this embodiment, the petri dish placement platform moving device 13 can move along the X and Y axes; the X-axis movable frame 13 drives the high-definition camera to move, achieving the effect that both the camera and the petri dish can move, and the X-axis movable frame 13 drives the high-definition camera to move), so that the high-definition cameras II 7 and III 10 on the side can capture images of the petri dish at different heights. The U-shaped copper tube 24 is connected to the temperature sensor 25 and the heating element 11 to provide real-time temperature feedback and maintain the set temperature cycle within the petri dish platform to achieve constant temperature cultivation in the petri dish.

[0045] like Figure 6 As shown, the detection principle of the detection device includes a high-definition camera I 6, a high-definition camera II 7, a high-definition camera III 10, a stepper motor 8, a moving slide rail 9, an X-axis movable frame 13, a base 26, and a photoelectric detector 27.

[0046] The control motherboard sends control commands to the stepper motor 8 according to a preset program. The stepper motor 8 then drives the X-axis movable frame 13 on the moving slide rail 9, moving the high-definition camera I 6 to a predetermined position. When the camera reaches the predetermined position, the photodetector 27 detects the position change and sends a signal to the control motherboard as feedback confirming the position. Upon receiving the position confirmation signal, the control motherboard instructs high-definition cameras I 6, II 7, and III 10 to take pictures.

[0047] Images captured by HD Camera I (6), HD Camera II (7), and HD Camera III (10) are transmitted to the control motherboard via a data interface for image processing and colony identification. The number of identified colonies and related information are transmitted to an external computer (3) via the data interface. Users can control the cameras to take pictures, move the platform, and view the analysis results through the interface on the control motherboard or a connected external device (PC 3).

[0048] like Figure 7 As shown, the overall software flow of this invention begins with the user placing the culture dish, setting the culture temperature, and detecting the number of culture dishes. After these preparations are completed, colony growth photos are taken, and the lower-level computer sends the photos to the upper-level computer and determines whether the transmission was successful. If the photo transmission fails, it is retransmitted. If the photo transmission is successful, it determines whether the detection is over. If the detection is over, the entire process will terminate. If there are still remaining detection tasks, the instrument will cycle through the detection steps.

[0049] like Figure 8 As shown, the control system of this invention uses an STM32F103VCT6 chip equipped with a Cortex-M3 core. This chip has a 32-bit width and a 72MHz main frequency, and integrates various peripheral interfaces. The peripheral circuit design of this chip includes, but is not limited to, a crystal oscillator circuit, a reset circuit, and an ST-LINK programming and debugging interface. The chip's built-in analog-to-digital converter (ADC) module can simultaneously acquire image data from 16 culture dishes and transmit the data to the processing unit. By configuring interrupt service routines and timers, it can generate pulse width modulation (PWM) signals to precisely control the motor's operation and respond to commands from the host computer. The system also integrates a Bluetooth communication module to realize wireless data exchange and remote control functions between the host computer and the slave computer. Various sensors, such as temperature sensors, are connected through general purpose input / output (GPIO) ports to read environmental parameters in real time and adjust the temperature accordingly to ensure the stability of the culture conditions.

[0050] like Figure 9 As shown, the main control board voltage control circuit is mainly used to provide operating voltage support for the main control board. The circuit uses the URA2409 chip to convert the overall input 24V power supply voltage to ±12V through related circuits. Then, through the second and third stage voltage conversion circuits, the 12V voltage is further converted to 5V and 3.3V to ensure that the main control board and other peripheral sensors and circuit boards connected to the main control board can operate normally.

[0051] like Figure 10As shown, the main control board printed circuit board (PCB) design involves precisely mapping each chip and functional component from the circuit board schematic to the PCB layout according to their model and package type. This process follows the electrical connection methods in the schematic, implementing the layout and routing of each component in PCB design software. After routing is completed, copper pouring is applied to different layers of the PCB. After the PCB design is completed, prototyping is performed, i.e., creating a prototype PCB board for functional and reliability testing. After passing the tests, each chip is precisely soldered onto the PCB board according to its package type, and different functional areas are connected by wires to achieve the final assembly of the circuit board.

[0052] like Figure 11 As shown, the data interaction circuit of this invention uses a high-definition camera to capture images of microorganisms in a petri dish. The camera converts light signals into electrical signals, which are then converted into digital signals for further processing. The analog electrical signals generated by the camera are processed by an image recognition algorithm to identify and count microbial colonies. A microcontroller chip sends control signals to a stepper motor through its output port to achieve precise control of the moving slide rail. The data interface is used to transmit the processed image data and other information to an external computer for further analysis and recording. like Figure 12 As shown, the data acquisition PCB diagram of this invention consists of a photodiode, an OPA2192 operational amplifier chip, and an ADS8688 chip. Upon system power-up, the main control chip initializes, including a camera module, a stepper motor drive module, a temperature sensor, and a heating element control module. The high-definition camera module captures images of microorganisms in the petri dish and transmits the image data to the image processing chip. The processed image data is transmitted to the main control chip via the SPI communication interface. The received image data is transmitted back to the computer via USB or a network interface. Further analysis and processing of the images are performed to identify the number of microbial colonies.

[0053] like Figure 13 As shown, the temperature sensor used in this invention is the DS18B20 digital temperature sensor, which communicates using a single-bus protocol. Through instructions from the main control chip, it receives data transmitted from the heating element and sends it back to the main control chip, comparing the real-time temperature with the set temperature to achieve feedback and precise temperature control of the culture dish platform.

[0054] like Figure 14 As shown, the adjustment button of this invention is used to adjust the temperature of the heating element for precise temperature management. The system receives a 24V DC power supply and converts it to 12V DC power via a TPS5430 to power the full-bridge drive circuit composed of an IR2104 and an LR7843. According to the instructions of the main control chip, this circuit can adjust the output voltage, thereby controlling the heating effect of the heating element to meet different temperature requirements.

[0055] Example 2: Image Processing System and Data Processing System The image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate, and the distortion correction error is controlled within ≤0.15 px. The image processing system uses a ceramic checkerboard-dot dual-mode calibration board to perform Zhang calibration and epipolar correction, with a reprojection error ≤0.08 px. The image processing system triggers lateral white light secondary imaging for colonies with a thickness greater than 0.6 mm and verifies it through shadow length, and verifies it for colonies with a thickness less than 0.2 mm using a blue light bottom transmittance OD attenuation model.

[0056] The image processing system first performs dual-camera calibration using three cameras to obtain initial extrinsic parameters. Then, during the stereo reconstruction process, high-confidence 3D points are selected in real time and projected onto the planes of each camera. A reprojection error function is constructed based on the projection deviation, and nonlinear least squares optimization is used to perturb and correct the camera extrinsic parameters and disparity bias. The extrinsic parameter optimization and deviation compensation are performed iteratively in a closed-loop manner, so that the reprojection error gradually converges, and finally the joint calibration parameters of the three cameras are obtained. The image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate. The specific process is as follows: Before performing dual-camera calibration, Zhang's calibration was performed on the three cameras C0, C1, and C2 using a ceramic checkerboard-dot dual-mode calibration board to obtain their respective intrinsic parameter matrices and distortion parameters. SCSS C0 → K0, D0 C1 → K1, D1 C2 → K2, D2 Where: scss represents the code block label, K0, K1, and K2 are the intrinsic parameter matrices of the three cameras; D0, D1, and D2 are the corresponding distortion coefficients. When stereoCalibrate performs extrinsic parameter estimation for each pair of cameras, the flag CALIB_FIX_INTRINSIC is enabled to ensure that stereoCalibrate no longer optimizes any camera intrinsic parameters or distortion coefficients, thereby ensuring that the three cameras maintain a consistent intrinsic parameter model in all dual-camera calibrations and avoiding calibration drift.

[0057] Step 1: Initial Static Calibration Phase First, using a ceramic checkerboard / dot dual-mode calibration board, dual-calibration was performed on the three cameras (C0, C1, C2) respectively: SCSS (C0, C1) → yields K0, K1, R 01 , T01 ; (C0, C2) → yields K0, K2, R 02 , T 02 ; (C1, C2) → yields K1, K2, R 12 ,T 12 ; Where scss represents a code block tag, K0 is the intrinsic parameter of camera C0; K1 is the intrinsic parameter of camera C1; K2 is the intrinsic parameter of camera C2; R 01 T 01 =External parameters of C0→C1; R02, T 02 =External parameters of C0→C2; R 12， T 12 =C1→C2 is an external parameter that is necessary for the three-camera chain geometric constraint.

[0058] Obtain initial calibration results: R 01 (0), T 01 (0); R 02 (0), T 02 (0); R 12 (0), T 12 (0); Among them, R 01 (0) represents the initial value of the rotation matrix from C0 to C1, T 01 (0) represents the initial value of the translation vector from C0 to C1; R 02 (0) represents the initial value of the rotation matrix from C0 to C2, T 02 (0) represents the initial value of the translation vector from C0 to C2; R 12 (0) represents the initial value of the rotation matrix from C1 to C2, T 12 (0) represents the initial value of the translation vector from C1 to C2.

[0059] Step 2: Consistency Constraints of Extrinsic Parameters for Three Cameras Based on the geometric chain relationship of the three-camera system, the following extrinsic parameter consistency condition is established: ; Among them, R 01 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C1; R 12 T 12 This indicates that the extrinsic parameter of camera C1 points to that of camera C2; R 02 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C2; Through joint optimization, the following can be achieved: (C0, C1); (C0, C2); (C1, C2); The extrinsic parameters of the three pairs of cameras remain strictly consistent in the same coordinate system; Step 3: Online reprojection error calculation After the image processing system runs, once each depth reconstruction is complete, N high-confidence stable 3D points are selected from the 3D point cloud or depth map: ; in, This represents the i-th 3D point, which is represented by 3D coordinates. Standard Xi, Yi, Zi composition.

[0060] Projected onto the planes of the three cameras respectively: vbnet

[0061] Where vbnet represents the language tag for the code block, It is the projection of camera C0, representing the pixel coordinates of the i-th 3D point projected onto camera C0; It is a projection onto camera C1, representing the pixel coordinates of the i-th 3D point projected onto camera C1. It is the projection of camera C2, representing the pixel coordinates of the i-th 3D point projected onto camera C2; Let "the i-th point" be the reference point. The intrinsic parameter matrices of camera C0 at center K0, camera C1 at K1, camera C2 at K2, and the poses of camera C0 at center R0 and T0 are typically set as references in the world coordinate system. The errors between these three projections and the actual observation point coordinates are calculated.

[0062] Where pi is the actual pixel seen, and pi′ is the pixel obtained by projection based on the current extrinsic parameters; And calculate the average error:

[0063] Among them, N is the number of reprojection errors in the tender documents. If E > 0.15 px, the online compensation mechanism will be triggered. Step 4: External parameter perturbation optimization The reprojection error is constructed as a nonlinear least-squares objective function. By fine-tuning the camera's rotation ΔR and translation ΔT, the reprojection error is minimized, enabling online dynamic calibration of the three-camera extrinsic parameters.

[0064] Where ΔR represents the rotation perturbation, which represents a small adjustment to the current camera rotation matrix R; ΔT represents the translation perturbation, which represents a small correction to the current camera displacement vector T; π represents the camera projection function, which means "projecting a 3D point onto a 2D pixel plane"; K represents the intrinsic parameter matrix; R represents the current rotation matrix; T represents the current translation vector; and (P_i) represents the i-th 3D point. Solve for the optimal perturbation using Levenberg–Marquardt or Ceres Solver: nginx R 01 ← R 01 + ΔR 01 ; T 01 ← T 01 + ΔT 01 ; R 02 ← R 02 + ΔR 02 ; T 02 ← T 02 + ΔT 02 ; Where nginx represents the code block language tag, extrinsic parameter perturbation update, ΔR 01 It is the camera's rotation correction for (C0→C1), ΔT 01 It is the camera's translation correction for (C0→C1), ΔR 02 It is the camera's rotation correction for (C0→C2), ΔT 02 It is the camera's translation correction for (C0→C2); Step 5: Dynamic parallax offset compensation The depth error of stereo matching is decomposed as follows:

[0065] Wherein, the original disparity d raw There is a systematic error, and this step obtains the bias through geometric error feedback. Δd After compensation, a more accurate true parallax is obtained. d true ; The system estimates the parallax bias based on the reprojection error:

[0066] in, , This represents the sensitivity of depth Z to parallax d; E is the average reprojection error, representing the pixel deviation of 3D points projected onto the image; and the depth is updated: Zcorrected = Zraw + ; The original depth Zraw is dynamically corrected by a parallax bias Δd to obtain a more accurate depth Zcorrected, which is used for 3D reconstruction and extrinsic parameter closed-loop optimization. Indicates the depth correction amount; Step Six: Dynamic Closed-Loop Iteration The entire system continuously executes the following during operation: Deep Reconstruction → Reprojection error estimation → Extrinsic parameter perturbation optimization → Parallax offset compensation → Parameter Update → A new round of reconstruction Until satisfied: Or Δ parameter < threshold (0.0001) Ultimately, this forms an adaptive, self-correcting closed-loop calibration system.

[0067] The data processing system employs the RAFT-Stereo deep learning stereo matching method. This method utilizes a binocular combination with a central camera and left and right cameras forming two different baselines. Through multi-baseline disparity estimation, disparity confidence modeling and fusion, and reflection suppression and feature enhancement mechanisms, it achieves high-precision 3D reconstruction. This method is particularly suitable for 3D measurement and depth recovery in scenarios with low texture, reflection, blurred edges of biological colonies, and minute-scale depth gradient changes.

[0068] Specifically as follows: (1) Introduction of multi-baseline input structure The following three cameras are deployed: -Center camera C0; - The left-side cameras C1 and C0 form a short baseline B01; - The right-side cameras C2 and C0 form a long baseline B02; Using the same central camera as a reference can effectively reduce the cumulative error caused by three independent viewpoints and improve the geometric consistency of multiple views. (2) Independent calculation of the two sets of disparity While maintaining the original distributed gradient iteration and cost volume construction mechanism of RAFT-Stereo, disparity is solved for the two baselines respectively:

[0069] in: -D 01 Short baseline parallax results in high precision but is susceptible to interference in reflective or low-texture areas; -D02 Long baseline parallax offers higher depth resolution but requires greater robustness in matching. The two complement each other and can form multiple baseline redundancy information, which significantly improves the stability of depth estimation.

[0070] (3) Modeling based on the confidence level of cost volume consistency and gradient magnitude For the disparity obtained for each baseline, confidence weights are constructed based on the following metrics: Cost body consistency Measure the convergence and internal consistency of cost volume updates during the bidirectional iteration of RAFT; gradient magnitude It reflects the local texture complexity and feature strength, and the disparity reliability is higher in high gradient regions.

[0071] Calculate the weights based on the above indicators. :

[0072] Where f can be a normalized exponential function, the output of a confidence network, or a piecewise linear function; CV 01 G is the cost volume consistency index for disparity D01. 01 It is the gradient intensity of the pixel corresponding to C0→C1; CV 02 G is the cost-volume consistency index for disparity D02. 02 C0→C2 represents the gradient intensity of the corresponding pixel; CV, the cost volume, represents the cost distribution of each pixel matching different disparity values ​​in RAFT or other stereo matching methods; the sharper the cost volume and the more obvious the peak, the more reliable the matching; conversely, the matching uncertainty is high and the confidence is low; G, the gradient magnitude, measures the local texture intensity of the image or feature map; in low-texture regions, the gradient is small, the matching is prone to errors, and the confidence should be reduced; high gradient regions are more reliable. (4) Parallax fusion model Two sets of disparities are fused using a confidence-weighted method:

[0073] (5) Use local RANSAC or least squares fitting for noisy regions; In the fused parallax map, for areas prone to local anomalies such as reflections, liquid surface refraction, and colony edges, the following is introduced: - Local RANSAC plane fitting; where local least-squares smooth fitting. Example

[0074] like Figure 15As shown, this invention employs a device based on a multi-angle high-definition camera to monitor the growth of bacterial colonies in petri dishes in real time, automating the photographing and counting of *E. coli*. The experimental group used this automated device to photograph and count *E. coli* in the petri dishes, while the control group used the traditional manual counting method. The experiment was repeated three times to ensure the accuracy of the results, and the average value was plotted for analysis. When the curve shows shape 15a, it indicates that under the same culture conditions, there is a large fluctuation between the results of manual counting for the same concentration of microorganisms. This fluctuation may be due to the subjectivity and operational errors of manual counting, thus affecting the consistency and reliability of the results. When the detection results show the shape of curve 15b, it indicates that under the same culture conditions, the data obtained from the automated counting device for the same concentration of microorganisms has smaller fluctuations, showing higher stability. This demonstrates that the automated counting device has significant advantages in accuracy and repeatability, providing more stable and reliable counting results.

[0075] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A device for real-time monitoring of bacterial colony growth in petri dishes using a multi-angle high-definition camera, characterized in that, The device body (1) is connected to the PC (3) via a serial port (2). The body (1) encapsulates the collected microbial colony growth data from the culture dish into a communication protocol and transmits it to the PC's host computer COM port via wired or wireless means. The COM port then processes the data and plots the parsed data into a counting result graph. The body (1) includes: A petri dish platform is used to place petri dishes and initiate detection programs to create a stable environment for microbial growth. The detection module integrates three high-definition cameras arranged in a symmetrical "one vertical and two horizontal" layout to capture images of the culture dishes from above and the sides, respectively, and provides uniform illumination through a backlight panel; the detection module can detect at least 16 culture dishes simultaneously; the image data collected by the detection module is transmitted to the MCU microcontroller for preliminary processing before being transmitted to the computer; MCU (Microcontroller Unit) is used for control, communication, complex algorithm calculations, and high-speed data transmission, and features low power consumption. A voltage conversion module is used to convert a wide range of externally input voltages into various stable voltage levels. The motor drive module is used to drive the high-definition camera to move horizontally and adjust its position. The automatic temperature control module based on the PID algorithm includes a temperature sensor and a heating plate located inside a U-shaped copper tube. It is used to collect temperature data in real time and make precise adjustments to avoid condensation and ensure that the temperature of the culture dish platform is constant. The wireless Bluetooth pass-through module adopts Bluetooth serial communication technology and is compatible with RS232 protocol to realize bidirectional communication between the host computer and the slave computer. The image processing system employs the RAFT-Stereo deep learning stereo matching method. It processes the acquired multi-angle images through a "three-camera parallax array and closed-loop calibration strategy" to identify the diameter changes of individual colonies, monitor colony height, and calculate volume, thereby achieving accurate colony identification and counting. The data processing system uses the YOLOv8 algorithm to extract features and predict class probabilities from images, and combines nonmaximum suppression technology to optimize the prediction results, outputting accurate reports on colony quantity, location, and class analysis. At the same time, it plots microbial counting results curves based on the appearance time and quantity changes of colonies.

2. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 1, characterized in that, The detection module includes a high-definition camera I (6), a high-definition camera II (7), and a high-definition camera III (10). The high-definition camera I (6) is mounted on a sliding rail (9) via an X-axis moving frame (13) and is driven by a stepper motor (8) of the motor drive module. A photodetector (27) is set above the camera movable support (18). The high-definition camera I (6) is the main camera (C0), which is vertically fixed above the petri dish platform to obtain a 0° reference image. The high-definition camera III (10) and the high-definition camera II (7) are the auxiliary cameras C1 and C2, respectively, which are located at the middle of the left side and the middle of the right side of the petri dish placement platform (5) at 90° and 270°, respectively, forming a 120° baseline.

3. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 2, characterized in that, The base (4) is embedded inside the device body (1). A culture dish placement platform (5) is set on the base (4). The culture dish placement platform (5) is fixed under the moving slide rail (9). A cell culture plate (12) is also set on the culture dish placement platform (5). An X-axis movable frame (13), a power switch (14), a first U-shaped photoelectric switch (15), a second U-shaped photoelectric switch (16), and a culture table fixing clip (17) are also set on the cell culture table (12). The power switch (14) and the first U-shaped photoelectric switch (15) are installed on the cell culture table (12). On one side of the stepper motor (8), the first U-shaped photoelectric switch (15) and the second U-shaped photoelectric switch (16) are symmetrically installed on both sides of the base (4); the culture table fixing clip (17) is installed on one side of the bottom of the cell culture plate (12); in addition, the first U-shaped photoelectric switch (15) and the second U-shaped photoelectric switch (16) are wired to the control motherboard. The first U-shaped photoelectric switch (15) is combined with the outlet baffle. When the outlet baffle blocks the photoelectric switch, the outlet success flag is sent to the culture dish placement platform (5). The second U-shaped photoelectric switch (16) combined with the baffle provides the system with the original position information and the detection start flag.

4. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 1, characterized in that, The bottom of the device body (1) is equipped with a petri dish placement platform moving device (23), a U-shaped copper tube (24), a temperature sensor (25) for the petri dish placement platform (5), a control motherboard (22), a power interface (21), an integrated circuit chip (20), and a data interface (19). The entire system receives power through the power interface (21) to power the high-definition camera I (6), high-definition camera II (7), high-definition camera III (10), the control motherboard (22), and the integrated circuit chip (20). The integrated circuit chip (20) in the control motherboard (22) is responsible for processing image data and performing image recognition. The algorithm identifies and counts microbial colonies in the culture dish; the number and information of the identified colonies are transmitted to an external computer (3) through the data interface (19) for data analysis and recording; the cell culture plate (12) moves the culture dish on the culture dish placement platform (5) through the culture dish placement platform moving device (23) so that the high-definition camera II (7) and high-definition camera III (10) can capture different heights of the culture dish; the U-shaped copper tube (24) is connected to the temperature sensor (25) and the heating plate (11) to provide real-time temperature feedback and maintain the set temperature cycle in the culture dish platform to achieve constant temperature culture of the culture dish.

5. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 1, characterized in that, The MCU microcontroller uses the STM32F103VCT6 chip, which has a 32-bit width and a 72MHz clock frequency, and has a variety of built-in peripheral interfaces.

6. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 1, characterized in that, The image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate, and the distortion correction error is controlled within ≤0.15 px. The image processing system uses a ceramic checkerboard-dot dual-mode calibration board to perform Zhang calibration and epipolar correction, with a reprojection error ≤0.08 px. The image processing system triggers lateral white light secondary imaging for colonies with a thickness greater than 0.6 mm and verifies it through shadow length, and verifies it for colonies with a thickness less than 0.2 mm using a blue light bottom transmittance OD attenuation model.

7. The device for real-time monitoring of bacterial colony growth in petri dishes based on a multi-angle high-definition camera according to claim 6, characterized in that, The image processing system first performs dual-target localization using three cameras to obtain initial extrinsic parameters. Then, during the stereo reconstruction process, it selects high-confidence 3D points in real time and projects them onto the planes of each camera. Based on the projection deviation, it constructs a reprojection error function and uses nonlinear least squares optimization to perform perturbation correction on the camera extrinsic parameters and parallax bias. The extrinsic parameter optimization and bias compensation are performed iteratively in a closed-loop manner, so that the reprojection error gradually converges and finally the joint calibration parameters of the three cameras are obtained. The image processing system calculates the intrinsic and extrinsic parameters of each pair of cameras using OpenCV stereoCalibrate. The specific process is as follows: Before performing dual-camera calibration, Zhang's calibration was performed on the three cameras C0, C1, and C2 using a ceramic checkerboard-dot dual-mode calibration board to obtain their respective intrinsic parameter matrices and distortion parameters. SCSS C0 → K0, D0 C1 → K1, D1 C2 → K2, D2 Where: scss represents code block label, K0, K1, K2 are the intrinsic parameter matrices of the three cameras; D0, D1, D2 are the corresponding distortion coefficients; when stereoCalibrate performs extrinsic parameter estimation for each pair of cameras, the flag CALIB_FIX_INTRINSIC is enabled to ensure that stereoCalibrate no longer optimizes any camera intrinsic parameters and distortion coefficients, thereby ensuring that the three cameras maintain a consistent intrinsic parameter model in all dual-camera calibrations and avoiding calibration drift; Step 1: Initial Static Calibration Phase First, using a ceramic checkerboard / dot dual-mode calibration board, dual-calibration was performed on the three cameras (C0, C1, C2) respectively: Scss; (C0, C1) → yields K0, K1, R 01 , T 01 ; (C0, C2) → yields K0, K2, R 02 , T 02 ; (C1, C2) → yields K1, K2, R 12 ,T 12 ; Where scss represents a code block tag, K0 is the intrinsic parameter of camera C0; K1 is the intrinsic parameter of camera C1; K2 is the intrinsic parameter of camera C2; R 01 T 01 =External parameters of C0→C1; R 02 T 02 =External parameters of C0→C2; R 12， T 12 =C1→C2 is the extrinsic parameter that is necessary for the three-camera chain geometric constraint; Obtain initial calibration results: R 01 (0), T 01 (0); R 02 (0), T 02 (0); R 12 (0), T 12 (0); Among them, R 01 (0) represents the initial value of the rotation matrix from C0 to C1, T 01 (0) represents the initial value of the translation vector from C0 to C1; R 02 (0) represents the initial value of the rotation matrix from C0 to C2, T 02 (0) represents the initial value of the translation vector from C0 to C2; R 12 (0) represents the initial value of the rotation matrix from C1 to C2, T 12 (0) represents the initial value of the translation vector from C1 to C2; Step 2: Consistency Constraints of Extrinsic Parameters for Three Cameras Based on the geometric chain relationship of the three-camera system, the following extrinsic parameter consistency condition is established:

8. Among them, R 01 T 01 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C1; R 12 T 12 This indicates that the extrinsic parameter of camera C1 points to that of camera C2; R 02 T 02 This indicates the extrinsic parameter of the center camera C0 pointing to the camera C2; Through joint optimization, the following can be achieved: (C0, C1); (C0, C2); (C1, C2); The extrinsic parameters of the three pairs of cameras remain strictly consistent in the same coordinate system; Step 3: Online reprojection error calculation After the image processing system runs, once each depth reconstruction is complete, N high-confidence stable 3D points are selected from the 3D point cloud or depth map:

9. Among them, This represents the i-th 3D point, which is represented by 3D coordinates. Standard Xi, Yi, Zi composition; Projected onto the planes of the three cameras respectively: Vbnet; 10. Among them, vbnet represents the language tag for a code block. It is the projection of camera C0, representing the pixel coordinates of the i-th 3D point projected onto camera C0; It is a projection onto camera C1, representing the pixel coordinates of the i-th 3D point projected onto camera C1. It is the projection of camera C2, representing the pixel coordinates of the i-th 3D point projected onto camera C2; Let "the i-th point" be the reference point. The intrinsic parameter matrices of camera C0 at center K0, camera C1 at K1, camera C2 at K2, and the pose of camera C0 at center R0 and T0 are typically set as the reference of the world coordinate system. Calculate the errors between the three projections and the actual observation point coordinates: ; Where pi is the actual pixel seen, and pi′ is the pixel obtained by projection based on the current extrinsic parameters; And calculate the average error: ； Among them, N is the number of reprojection errors in the tender documents. If E > 0.15 px, the online compensation mechanism will be triggered. Step 4: External parameter perturbation optimization The reprojection error is constructed as a nonlinear least-squares objective function. By fine-tuning the camera's rotation ΔR and translation ΔT, the reprojection error is minimized, enabling online dynamic calibration of the three-camera extrinsic parameters. ； Where ΔR represents the rotation perturbation, which represents a small adjustment to the current camera rotation matrix R; ΔT represents the translation perturbation, which represents a small correction to the current camera displacement vector T; π represents the camera projection function, which means "projecting a 3D point onto a 2D pixel plane"; K represents the intrinsic parameter matrix; R represents the current rotation matrix; T represents the current translation vector; and (P_i) represents the i-th 3D point. Solve for the optimal perturbation using Levenberg–Marquardt or Ceres Solver: nginx R 01 ← R 01 + ΔR 01 ; T 01 ← T 01 + ΔT 01 ; R 02 ← R 02 + ΔR 02 ; T 02 ← T 02 + ΔT 02 ; Where nginx represents the code block language tag, extrinsic parameter perturbation update, and ΔR 01 It is the camera's rotation correction for (C0→C1), ΔT 01 It is the camera's translation correction for (C0→C1), ΔR 02 It is the camera's rotation correction for (C0→C2), ΔT 02 It is the camera's translation correction for (C0→C2); Step 5: Dynamic parallax offset compensation The depth error of stereo matching is decomposed as follows: ； Wherein, the original disparity d raw There is a systematic error, and this step obtains the bias through geometric error feedback. Δd After compensation, a more accurate true parallax is obtained. d true ; The system estimates the parallax bias based on the reprojection error: ; in, , This represents the sensitivity of depth Z to parallax d. E is the average reprojection error, representing the pixel deviation of 3D points projected onto the image; and the depth is updated accordingly. ； The original depth Zraw is dynamically corrected by parallax bias Δd to obtain a more accurate depth Zcorrected, which is used for 3D reconstruction and extrinsic parameter closed-loop optimization. Here, ΔZ represents the depth correction amount. Step Six: Dynamic Closed-Loop Iteration The entire system continuously executes the following during operation: Deep Reconstruction → Reprojection error estimation → Extrinsic parameter perturbation optimization → Parallax offset compensation → Parameter Update → A new round of reconstruction Until satisfied: Or Δ parameter < threshold (0.0001) Ultimately, this forms an adaptive, self-correcting closed-loop calibration system.

11. The device for real-time monitoring of bacterial colony growth in a petri dish based on a multi-angle high-definition camera according to claim 1, characterized in that, The data processing system employs the RAFT-Stereo deep learning stereo matching method. This method utilizes a binocular combination consisting of a central camera and left and right cameras forming two different baselines. Through multi-baseline disparity estimation, disparity confidence modeling and fusion, as well as reflection suppression and feature enhancement mechanisms, it achieves high-precision 3D reconstruction. This method is particularly suitable for 3D measurement and depth recovery in scenarios with low texture, reflection, blurred edges of biological colonies, and minute-scale depth gradient changes. Specifically as follows: (1) Introduction of multi-baseline input structure The following three cameras are deployed: -Center camera C0; - The left-side cameras C1 and C0 form a short baseline B01; - The right-side cameras C2 and C0 form a long baseline B02; Using the same central camera as a reference can effectively reduce the cumulative error caused by three independent viewpoints and improve the geometric consistency of multiple views. (2) Independent calculation of the two sets of disparity While maintaining the original distributed gradient iteration and cost volume construction mechanism of RAFT-Stereo, disparity is solved for the two baselines respectively:

12. Among them: -D 01 Short baseline parallax results in high precision but is susceptible to interference in reflective or low-texture areas; -D 02 Long baseline parallax offers higher depth resolution but requires greater robustness in matching. The two complement each other and can form multi-baseline redundancy information, which significantly improves the stability of depth estimation. (3) Modeling based on the confidence level of cost volume consistency and gradient magnitude For the disparity obtained for each baseline, confidence weights are constructed based on the following metrics: Cost body consistency Measure the convergence and internal consistency of cost volume updates during the bidirectional iteration of RAFT; gradient magnitude It reflects the local texture complexity and feature intensity; the disparity reliability is higher in high gradient regions. Calculate the weights based on the above indicators. :

13. Where f can be a normalized exponential function, the output of a confidence network, or a piecewise linear function; CV 01 G is the cost volume consistency index for disparity D01. 01 It is the gradient intensity of the pixel corresponding to C0→C1; CV 02 G is the cost volume consistency index for disparity D02. 02 C0→C2 represents the gradient intensity of the corresponding pixel; CV, the cost volume, represents the cost distribution of each pixel matching different disparity values ​​in RAFT or other stereo matching methods; the sharper the cost volume and the more obvious the peak, the more reliable the matching; conversely, the matching uncertainty is high and the confidence is low; G, the gradient magnitude, measures the local texture intensity of the image or feature map; in low-texture regions, the gradient is small, the matching is prone to errors, and the confidence should be reduced; high gradient regions are more reliable. (4) Parallax fusion model Two sets of disparities are fused using a confidence-weighted method:

14. (5) Use local RANSAC or least squares fitting for the noisy region; In the fused parallax map, for areas prone to local anomalies such as reflections, liquid surface refraction, and colony edges, the following is introduced: - Local RANSAC plane fitting; where local least-squares smooth fitting.

15. The device for real-time monitoring of bacterial colony growth in a petri dish based on a multi-angle high-definition camera according to claim 1, characterized in that, The data processing system employs U-Net++ as the core segmentation model, optimizing it to address the issues of blurred colony boundaries and severe adhesion in petri dishes, thus improving edge segmentation accuracy. To further distinguish adhered colony individuals, Mask R-CNN is introduced for instance-level segmentation, combined with SAM for interactive fine-tuning. The system extracts color, texture, and shape features from colony images using convolutional neural networks. During instance segmentation, Non-Maximum Suppression (NMS) is used to remove redundant detection boxes, ultimately outputting colony quantity, area, location, and morphological parameters. It integrates a classification model, enabling expanded support for colony category prediction. The system also supports visualization of time-series-based colony counting results, dynamically plotting microbial growth curves to achieve accurate analysis of colony quantity, spatial distribution, and morphological characteristics. The "three-camera parallax array and closed-loop calibration strategy" involves using a multi-view parallax array composed of three cameras (vertical C0 and two side cameras C1 and C2) to obtain the parallax map for each pair of views through stereo matching, and then performing multi-parallax fusion... Closed-loop calibration with minimized reprojection error iteratively compensates for extrinsic / intrinsic errors and parallax bias, thereby obtaining accurate estimates of colony depth, thickness, and volume. Confidence weighting and time smoothing are used to improve stability.

16. The device for real-time monitoring of bacterial colony growth in a petri dish based on a multi-angle high-definition camera according to claim 1, characterized in that, The voltage conversion module is located on the main control circuit board and is used for voltage conversion. The main control board voltage control circuit is composed of URA2409 chip. The overall input 24V power supply voltage is converted to ±12V voltage through related circuits. Then, through secondary and tertiary voltage conversion circuits, the 12V voltage is further converted to 5V and 3.3V to ensure that the main control board and other peripheral sensors and circuit boards connected to the main control board can operate normally.

Images