Multimodal image acquisition and processing method for automated counting of streaming microbial colonies

Abstract

This invention relates to the field of image data processing technology, and discloses a multimodal image acquisition and processing method for automated counting of microbial colonies in a production line. The method includes the following steps: imaging a culture dish under test using a multimodal physical imaging module to acquire three-modal fused image data containing at least polarization characteristics, multispectral characteristics, and photometric stereoscopic characteristics; establishing a physical-neural hybrid reflection decoupling network and inputting the three-modal fused image data into the network; based on the diffuse reflection dominant image and the specular reflection dominant image, using a multi-scale colony generation and repair network to perform texture repair on reflective residue areas, wherein the repair network introduces a colony morphology prior constraint layer in the decoder section; outputting the repaired image and performing colony counting based on the repaired image. This invention possesses robustness in suppressing and repairing surface reflective interference and has strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of microbial detection and image data processing technology, and more specifically, to a multimodal image acquisition and processing method, system and device for automated counting of microbial colonies in a production line. Background Technology

[0002] Microbial colony counting is a core component of microbial testing in fields such as food, pharmaceuticals, cosmetics, and environmental monitoring, and is widely used in product quality control, hygiene and safety assessment, and pathogen monitoring. Traditional colony counting relies primarily on manual visual identification and counting. Operators place petri dishes on a colony counter, using a magnifying glass and a manual counting pen to mark and count each colony. However, manual counting is time-consuming and labor-intensive. Different operators have subjective differences in interpretation standards, and the counting results for the same petri dish may vary from person to person, affecting the consistency and comparability of test results. Furthermore, long-term colony counting requires extremely high levels of eyesight and concentration from operators, resulting in high human resource costs.

[0003] In recent years, with the rapid development of automation and machine vision technologies, more and more research institutions and testing centers have begun to explore automated production line solutions for microbial colony counting. Existing automated colony counting equipment typically includes the following basic modules: an automatic plate-pushing and positioning mechanism, an image acquisition system, and an image processing-based colony recognition algorithm. The plate-pushing mechanism often uses cylinders, conveyor belts, or robotic arms to achieve automatic sample introduction and positioning of the plates. The image acquisition system strives to obtain stable and clear plate images by setting up a closed dark chamber and controlling the light source angle. The image processing algorithm extracts colony regions from the background and completes the counting through filtering, segmentation, and morphological operations.

[0004] In image acquisition, existing technologies generally employ a dark box structure with a fixed light source to improve image quality and reduce ambient light interference. However, because petri dishes are typically made of transparent polystyrene or glass with smooth surfaces, and the agar medium itself contains moisture, they are prone to strong specular reflection under light illumination. This specular reflection not only creates bright spots in the image, obscuring the true morphology and edge information of the colonies, but may also produce false spots similar to the colony morphology, leading to missed or false detections in subsequent image segmentation and recognition algorithms. Although optimizing the light source angle and using diffused illumination can alleviate the reflection problem to some extent, it is difficult to fundamentally eliminate the complex reflection phenomena caused by individual differences in petri dishes, variations in agar surface humidity, and differences in culture medium composition.

[0005] To address reflective interference, some existing technologies have incorporated polarization-based optical imaging methods. However, these technologies still have significant shortcomings. First, existing techniques typically acquire only one set of polarization images (such as cross-polarized images) for reflective suppression, failing to fully utilize multi-angle polarization information for quantitative analysis and separation of reflective components. Second, even after polarization processing, localized reflective areas may remain on the surface of the culture dish, especially when the colony surface is moist, the culture medium has a high water content, or the dish has scratches. Existing technologies lack targeted processing methods for these residual reflective areas, often discarding the image information of these areas directly or ignoring them in subsequent counting, leading to lower colony counts. Third, the image processing algorithms used in existing technologies are mostly traditional filtering and morphological operations (such as median filtering, erosion, and dilation). For colonies adhering to each other in complex backgrounds, tiny colonies, and colonies morphologically similar to impurities, the recognition accuracy is limited, making it difficult to meet the accuracy requirements of high-throughput detection scenarios.

[0006] In image processing algorithms, existing technologies have attempted to apply deep learning to colony recognition. For example, some studies have used convolutional neural networks (CNNs) for colony image classification or target detection. However, these methods typically treat colony recognition as a purely end-to-end learning problem, ignoring the physical and optical mechanisms of the imaging process. This results in insufficient generalization ability of the model when faced with different light source conditions, different petri dish materials, and different culture medium types. Furthermore, for the problem of missing information in reflective areas, existing deep learning methods often use simple image completion or interpolation techniques for repair, lacking prior constraints on the biological morphological characteristics of colonies. The repair results may produce artifacts that do not conform to the actual colony morphology, affecting the accuracy of counting.

[0007] Overall, existing automated microbial colony counting technologies struggle to completely resolve reflection interference issues during image acquisition, lack physical modeling and separation of reflection components in image processing, lack targeted generative methods for repairing residual reflective areas, and the overall system fails to achieve closed-loop optimization of imaging control and image analysis. Summary of the Invention

[0008] The main objective of this invention is to provide a multimodal image acquisition and processing method for automated counting of microbial colonies in automated production lines, which can effectively suppress and repair reflective interference and possesses high precision and strong generalization ability, in order to solve the following technical problems:

[0009] Firstly, during the microbial colony counting process, the surface of the petri dish is affected by specular reflection caused by light source illumination. Existing technologies rely solely on single polarization imaging or a fixed light source angle, which makes it difficult to completely eliminate reflection, resulting in the colony information being obscured or false light spots being generated.

[0010] Secondly, the residual reflective areas cause a loss of image information, which traditional image processing methods cannot effectively repair, leading to missed colony detection or a decrease in counting accuracy.

[0011] Third, existing deep learning models ignore the physical mechanism of imaging, have insufficient generalization ability, and are difficult to adapt to different types of culture media, different humidity conditions, and individual differences in culture dishes. Moreover, static imaging systems cannot adaptively adjust the acquisition parameters according to the reflectivity of the sample, and have limited ability to process samples with extreme reflectivity.

[0012] Based on the first main aspect of the present invention, a multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line is provided, the method comprising one or a combination of the following steps:

[0013] The culture dish under test is imaged using a multimodal physical imaging module to obtain three-modal fused image data that includes at least polarization characteristics, multispectral characteristics, and photometric stereoscopic characteristics. The polarization characteristics include at least three polarization images at different polarization angles, the multispectral characteristics include at least two images in different spectral bands, and the photometric stereoscopic characteristics include at least three images illuminated by light sources at different angles.

[0014] A physical-neural hybrid reflex decoupling network is established, and the trimodal fused image data is input into the physical-neural hybrid reflex decoupling network. The structure of the network includes:

[0015] A physical constraint layer is used for the Stokes vector calculation model based on polarization imaging and the surface normal reconstruction model of photometric stereo to calculate the physical initial values ​​of the specular reflection component and the diffuse reflection component, respectively.

[0016] A deep separation network is used to receive the physical initial value and the original multimodal image, and output a high-precision diffuse reflection dominant image and specular reflection dominant image through multi-scale feature extraction and fusion.

[0017] The physical-neural hybrid reflex decoupling network is trained using a joint loss function, which includes a physical consistency loss term and an image reconstruction loss term.

[0018] Based on the diffuse dominant image and the specular dominant image, a multi-scale colony generation and repair network is used to repair the texture of the reflective residue area. The repair network introduces a colony morphology prior constraint layer in the decoder part. The prior constraint layer is used to constrain the generation result to conform to the geometric morphological characteristics of the colony.

[0019] Output the repaired image and perform colony counting based on the repaired image.

[0020] The overall concept of this invention is to first acquire a fused three-modal image of a culture dish—polarization, multispectral, and photometric stereoscopic—using a multimodal physical imaging module, providing rich optical information for subsequent reflection separation. This fused image is then input into a physical-neural hybrid reflection decoupling network composed of a physical constraint layer and a deep separation network. The physical constraint layer calculates initial physical values ​​for specular and diffuse reflection based on Stokes vectors, photometric stereoscopic methods, and Fresnel theory. The deep separation network refines these initial physical values ​​through multi-scale feature extraction and fusion, outputting high-precision diffuse and specular reflection dominant images. Finally, a multi-scale colony generation and repair network is used to repair the texture of residual reflective areas, and colony counting is performed based on the repaired image.

[0021] Its main effect is that through a hybrid architecture of physical model and deep learning, it achieves systematic suppression and accurate separation of reflective interference, significantly improves the accuracy and generalization ability of colony counting, and solves the technical problem of missed detection and false detection caused by reflective interference in existing technologies.

[0022] Optionally, the multimodal physical imaging module includes at least:

[0023] A rotatable polarizer assembly for acquiring images at four polarization angles: 0°, 45°, 90°, and 135°.

[0024] A multispectral light source array contains light sources in at least three bands: 405nm, white light, and red light, and images of each band are acquired separately using time-division multiplexing.

[0025] A three-dimensional LED light source array contains at least three independently controlled light sources, each of which can independently adjust its brightness and switching sequence, to achieve photometric stereo imaging and acquire images from at least three different lighting directions;

[0026] An electric rotating stage is used to rotate the petri dish to accommodate different light angles for collection.

[0027] The above scheme provides complete input data for subsequent reflection separation through hardware fusion of polarization, multispectral and photometric stereo, ensuring the reflection suppression effect from the source of imaging, and providing the necessary Stokes vector and normal map input for the calculation of the physical constraint layer.

[0028] As a further preferred embodiment, the calculation process of the physical constraint layer includes:

[0029] Based on the images at different polarization angles, the Stokes vector is calculated, and the degree of linear polarization and polarization angle are further calculated.

[0030] Based on the images illuminated by at least three different light sources, the normal map of the petri dish surface is reconstructed using a photometric stereo method;

[0031] Based on the linear polarization degree and normal diagram, the initial values ​​of the specular reflection component and the diffuse reflection component are separated using the Fresnel reflection model.

[0032] This physical constraint layer does not contain trainable parameters and is implemented entirely based on the physical principles of optical imaging. It can quickly generate initial values ​​of reflection components that conform to physical laws without the need for labeled data, providing high-quality initial features for deep separation networks, guiding the training direction of subsequent deep learning models, and enhancing the interpretability and physical consistency of network output.

[0033] Optionally, the deep separation network adopts an encoder-decoder structure, wherein:

[0034] The encoder section is used to extract multi-scale features of the original multimodal image and the physical initial values;

[0035] The decoder section fuses encoder features through skip connections to output the final separation result;

[0036] The network introduces a cross-modal attention mechanism between the encoder and decoder to establish correlation weights between different modal features, thereby enhancing the ability to identify reflective regions.

[0037] The encoder-decoder architecture, through the synergistic effect of multi-scale feature extraction and cross-modal attention mechanism, enables the deep separation network to effectively fuse heterogeneous optical information, refine the initial physical values, and output high-precision diffuse reflection-dominant and specular reflection-dominant images, thereby improving the accuracy and robustness of reflection separation.

[0038] Optionally, the joint loss function includes at least one of the following three losses or a combination thereof:

[0039] Physical consistency loss: Calculate the difference between the diffuse reflection component of the network output and the diffuse reflection component of the physical initial value, as well as the difference between the specular reflection component and the specular reflection component of the physical initial value;

[0040] Image reconstruction loss: Calculates the difference between the recombined diffuse-dominant image and specular-dominant image output by the network and the original input image;

[0041] Adversarial loss: The discriminator network determines whether the separation results match the data distribution of the real colony image.

[0042] The joint optimization of the three loss functions enables the network to take into account physical laws, image integrity and visual realism during training. Compared with the traditional method that uses only a single loss function, it can obtain more accurate and natural reflection separation results, while enhancing the model's generalization ability under different imaging conditions.

[0043] Optionally, the multi-scale colony-generating repair network includes at least one or a combination of the following components:

[0044] Multi-scale encoder: used to extract texture features of reflective areas at different scales;

[0045] Colony morphology prior constraint layer: Located at different levels of the decoder, it constrains the generated results through a pre-trained colony shape discriminator. The colony shape discriminator is trained based on historical colony images and is used to determine whether the generated region conforms to the morphological features of colonies such as roundness, area distribution, and edge gradient.

[0046] Spatial adaptive normalization layer: used as modulation parameters to control the generation process of the repair network by using the reflection intensity information in the specular reflection dominant image.

[0047] By introducing prior constraints based on colony morphology, the repaired area not only transitions naturally in texture but also conforms to the biological characteristics of actual colonies in geometry. This effectively avoids artifacts or non-colony morphologies that may be produced by traditional repair methods, thus ensuring the accuracy of the repaired image for colony counting.

[0048] As a further preferred embodiment, the physical-neural hybrid reflex decoupling network is established and deployed through the following steps:

[0049] A physical constraint layer without trainable parameters is constructed, which is implemented through programming based on the physical principles of optical imaging.

[0050] A deep separation network with trainable parameters is constructed. The deep separation network is trained on a training dataset using a deep learning framework to learn a nonlinear mapping from physical initial values ​​and original multimodal images to accurate reflection separation results.

[0051] The trained deep separation network is cascaded with the physical constraint layer to form a complete physical-neural hybrid reflex decoupling network, which is then deployed to the edge computing unit built into the microbial colony counting device for real-time processing of the acquired multimodal image data.

[0052] The overall concept of the physics-neural hybrid reflection decoupling network is as follows: First, a physical constraint layer without trainable parameters is constructed, which is implemented programmatically based entirely on the physical principles of optical imaging. Then, a deep separation network with trainable parameters is built. This network is trained on a training dataset using a deep learning framework, learning a nonlinear mapping from initial physical values ​​and raw multimodal images to accurate reflection separation results. Finally, the trained deep separation network is concatenated with the physical constraint layer to form a complete reflection decoupling network, which is then deployed to the device's built-in edge computing unit.

[0053] Its effectiveness lies in the fact that by combining a physical constraint layer with a deep separation network, an interpretable physical model and a deep learning model with strong fitting ability are organically integrated. This ensures the physical rationality of the network output and makes up for the shortcomings of the pure physical model in complex scenarios by using data-driven approaches, thus forming a reflection separation solution that combines interpretability and high accuracy.

[0054] Optionally, the construction of the physical constraint layer includes pre-constructing a deterministic calculation module for the physical constraint layer through programming, based on the physical principles of optical imaging. The physical constraint layer has the following calculation process: calculating the Stokes vector and linear polarization degree based on multi-angle polarization images; reconstructing the surface normal map based on multi-angle light source images using photometric stereo method; and analytically calculating the initial physical values ​​of specular reflection and diffuse reflection components according to Fresnel reflection theory. The physical constraint layer does not contain trainable parameters.

[0055] The construction of the deep separation network includes building a neural network architecture for the deep separation network on a high-performance computing server based on the PyTorch or TensorFlow deep learning framework. The deep separation network adopts an encoder-decoder structure and introduces a cross-modal attention mechanism. The constructed physical constraint layer is cascaded with the deep separation network to form a complete physical-neural hybrid reflection decoupling network. Multimodal image data containing at least polarization characteristics, multispectral characteristics, and photometric stereo characteristics are collected as training samples. The hybrid architecture model of the physical constraint layer and the deep separation network is trained end-to-end using a joint loss function containing a physical consistency loss term and an image reconstruction loss term. The physical consistency loss term constrains the output of the deep separation network to maintain consistency with the physical initial value of the physical constraint layer output until the model converges. The trained network weight parameter file is then output.

[0056] The deep separation network and physical constraint layer concatenation involves exporting the trained network weight parameter file into an intermediate format using the Open Neural Network Exchange Format (ONNX), and then using the TensorRT inference optimization tool for precision quantization and layer fusion optimization to generate an optimized inference engine file suitable for edge computing platforms. The optimized inference engine file is then loaded into the edge computing unit built into the microbial colony counting device. The edge computing unit is selected from NVIDIA Jetson series embedded platforms or industrial control computers supporting CUDA. The edge computing unit interacts with the device's main control program through a Java native interface or inter-process communication, receives the collected multimodal image data, and returns the reflection decoupling results in real time.

[0057] The complete setup and deployment process described above ensures efficient migration of the network from the development environment to the production environment. Model quantization and optimization techniques significantly reduce inference latency, enabling complex physical-neural hybrid models to run in real time on resource-constrained edge devices. Meanwhile, JNI bridging ensures seamless integration with the existing Java technology stack.

[0058] Optionally, after performing colony counting based on the repaired image, the method further includes:

[0059] A repair confidence score is calculated for each repaired region. The repair confidence score is calculated based on the output probability of the colony morphology prior constraint layer, the texture consistency between the repaired region and the surrounding region, and the reconstruction error of multi-scale features.

[0060] If the repair confidence score is lower than the preset threshold, the petri dish will be automatically marked as a manually verified sample in the laboratory information management system, and the low-confidence repair area will be marked in the image.

[0061] By introducing a confidence assessment mechanism, the system can identify and mark areas with unreliable repair quality, and hand over samples that are difficult to judge automatically to human review. This not only gives full play to the high efficiency of automated processing, but also ensures the accuracy of the final results through human-machine collaboration, avoiding counting deviations caused by erroneous repairs, and meeting the strict requirements for result reliability in the field of medical testing.

[0062] Optionally, before imaging the culture dish to be tested using the multimodal physical imaging module, the method further includes acquiring a thermal map of the reflective area distribution of the culture dish through a pre-scanning mode, and dynamically adjusting the acquisition parameters of the multimodal physical imaging module based on the thermal map of the reflective area distribution. The acquisition parameters include polarization angle, light source angle, exposure time, and spectral band combination.

[0063] Its effectiveness lies in achieving closed-loop control of diagnosis before imaging through pre-scanning and dynamic parameter adjustment. This enables the imaging system to automatically optimize the acquisition strategy for highly reflective samples, reducing reflective interference from the source and avoiding the problem of insufficient processing capability for extreme samples by fixed parameter imaging. This significantly improves the system's adaptability and imaging quality.

[0064] Optionally, the dynamic adjustment includes: automatically increasing the number of polarization angle samples when the area of ​​the reflective region exceeds a preset threshold; and performing secondary fine scanning imaging on the severely reflective local area by controlling the electric rotating stage and the three-dimensional LED light source array to obtain higher resolution multimodal image data.

[0065] In the above scheme, when the area of ​​the reflective region identified by the pre-scan exceeds the preset threshold, the system automatically increases the number of polarization angle samples (e.g., from 4 angles to 8 or more) to provide higher precision Stokes vector calculation input for the physical constraint layer.

[0066] For localized areas with particularly severe reflections, the system controls the electric rotating stage to rotate the culture dish to a specific angle and controls the 3D LED light source array to perform a fixed-point secondary fine scanning imaging of the area, thereby acquiring higher resolution local multimodal image data.

[0067] This layered dynamic adjustment strategy can take differentiated processing measures according to the severity of reflection. For normal reflection, it can be effectively suppressed by increasing the polarization angle, while for extreme reflection, it can be targeted by local fine scanning. This ensures both processing efficiency and imaging quality for difficult samples.

[0068] Based on the second main aspect of the present invention, a multimodal image acquisition and processing system for implementing the aforementioned method is provided, comprising at least the following modules:

[0069] The multimodal physical imaging module is used to acquire fused image data of the polarization characteristics, multispectral characteristics, and photometric stereoscopic characteristics of the culture dish under test;

[0070] The dynamic imaging control module is used to adaptively adjust the acquisition parameters of the imaging module based on the pre-scan results, and to perform secondary fine scanning imaging on areas with severe reflection.

[0071] A physical-neural hybrid reflection decoupling module is used to separate diffuse reflection components and specular reflection components based on multimodal fused image data through a deep neural network with embedded physical constraints.

[0072] A multi-scale colony generation and repair module is used to perform texture repair on reflective residual areas based on separation results. The repair module includes a colony morphology prior constraint layer.

[0073] The confidence assessment and labeling module is used to calculate the confidence score of the repaired area and manually review and label low-confidence areas.

[0074] The colony counting module is used to count colonies based on the repaired image.

[0075] Optionally, the multimodal physical imaging module includes:

[0076] A rotatable polarizer assembly that supports image acquisition at at least four polarization angles;

[0077] A multispectral light source array containing at least three light sources in different wavelength bands;

[0078] A three-dimensional LED light source array, comprising at least three independently controllable light sources;

[0079] An electric rotating stage is used to collect data at different lighting angles.

[0080] Optionally, the physical-neural hybrid reflex decoupling module includes:

[0081] A physical constraint layer is used to generate initial physical values ​​for the reflection components based on polarization imaging Stokes vector calculation and photometric stereo surface reconstruction.

[0082] The deep separation network employs an encoder-decoder structure and introduces a cross-modal attention mechanism to output high-precision diffuse-dominant and specular-dominant images.

[0083] A joint optimization unit is used to train the deep separation network using a joint loss function that includes physical consistency loss and image reconstruction loss.

[0084] Optionally, the multi-scale colony generation and repair module adopts a conditional generative adversarial network structure and introduces a colony morphology prior constraint layer in the decoder part. The prior constraint layer constrains the generation result through a pre-trained colony shape discriminator.

[0085] Based on a third key aspect of the present invention, an intelligent microbial colony counting device based on automated operation is provided, including the aforementioned multimodal image acquisition and processing system.

[0086] Compared with existing technologies, in the imaging stage, the multimodal physical imaging module of this invention integrates polarization, multispectral, and photometric stereo information, and, in conjunction with dynamic adaptive imaging control, can automatically optimize the polarization angle, light source angle, and exposure parameters based on the pre-scanned reflective thermal map, thereby reducing specular reflection brightness and ensuring image quality from the source. In the reflection separation stage, the physical-neural hybrid reflection decoupling network integrates a physical constraint layer based on Stokes vectors, photometric stereo methods, and Fresnel theory with a deep separation network, ensuring the physical rationality of reflection separation and enabling refined correction of complex reflective scenes through data-driven processing.

[0087] In the colony identification and repair process, the multi-scale colony generation and repair network introduces a prior constraint layer of colony morphology. The pre-trained colony shape discriminator constrains the generated results to conform to the biological characteristics of colony, such as roundness, area distribution, and edge gradient. This avoids artifacts or non-colony morphologies that may be produced by traditional repair methods, and improves the recall rate of colonies in areas where information is missing due to reflection.

[0088] Furthermore, this invention utilizes model quantization and optimization techniques (ONNX export, TensorRT acceleration) to deploy complex physical-neural hybrid models on edge computing platforms such as NVIDIA Jetson, controlling inference latency to the millisecond level and meeting the real-time requirements of high-throughput detection. Seamless integration with the device's main control program via the Java Native Interface (JNI) maintains cross-platform compatibility with existing Java technology stacks.

[0089] This invention integrates a complete automated process for plate sample loading, precise positioning, multimodal imaging, intelligent counting, and sorting and recycling. A single unit can replace 3 to 5 manual operators, freeing up human resources and providing a high-throughput, intelligent, and information-based overall solution for microbiology testing laboratories, directly promoting technological upgrading and automation transformation in the field of microbiology testing. Attached Figure Description

[0090] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0091] Figure 1 An execution flowchart of one embodiment of the present invention is shown;

[0092] Figure 2 The basic appearance characteristics of a typical colony are shown in one embodiment of the present invention;

[0093] Figure 3 A basic example of colony labeling is shown in one embodiment of the present invention;

[0094] Figure 4 A microbial colony counting device integrating a multimodal physical imaging module is shown in one embodiment of the present invention. Detailed Implementation

[0095] The preferred embodiments of the present invention will be described in detail below to provide a clearer understanding of the purpose, features, and advantages of the invention. It should be understood that the following embodiments are not intended to limit the scope of the invention, but are merely illustrative of the essential spirit of the technical solution of the invention.

[0096] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known techniques associated with the invention may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0097] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0098] like Figure 1 As shown, in one embodiment, the execution flow of the multimodal image acquisition and processing method for automated microbial colony counting of the present invention mainly includes the following steps:

[0099] S100 uses a multimodal physical imaging module to image the culture dish under test and acquire three-modal fusion image data that includes at least polarization characteristics, multispectral characteristics and photometric stereo characteristics.

[0100] S200, establish a physical-neural hybrid reflex decoupling network, and input the trimodal fused image data into the physical-neural hybrid reflex decoupling network;

[0101] S300, based on the diffuse dominant image and the specular dominant image, a multi-scale colony generation and repair network is used to perform texture repair on the reflective residual area. The repair network introduces a colony morphology prior constraint layer in the decoder part.

[0102] S400 outputs the repaired image and performs colony counting based on the repaired image.

[0103] like Figure 2 As shown, in the following embodiments, the basic appearance characteristics of the colonies to be treated are: clear boundaries; translucent or opaque morphology; and typically radial.

[0104] like Figure 3 As shown, in the following embodiments, the basic principle of colony labeling is: a colony with a clear boundary is considered a single colony; two colonies that are close together are considered two colonies if there is a clear break or gap; a colony that is spindle-shaped is considered a single colony. No matter how faint the color, as long as there is a clear boundary, i.e., the boundary is not gradual but abrupt, it is considered a single colony.

[0105] like Figure 4 As shown, this embodiment employs a microbial colony counting device integrating a multimodal physical imaging module. This device includes a frame body 2 and a multimodal physical imaging module 1 housed within it. The multimodal physical imaging module 1 includes a motorized rotating stage 1a for supporting the culture dish 3 to be tested. It also includes a height-adjustable support 4 positioned above the motorized rotating stage 1a, comprising an image acquisition component 4a and a lifting component 4b. A multispectral light source array 1b is mounted on the image acquisition component 4a. Furthermore, the multimodal physical imaging module 1 includes a rotatable polarization component 1c and a 3D LED light source array 1d. The polarizer of the rotatable polarization component 1c is positioned below the camera of the image acquisition component 4a. The 3D LED light source array 1d comprises at least three independently controlled light sources.

[0106] As a complete device, it usually also includes mechanisms such as an automatic plate loading mechanism and an edge computing unit, all of which can utilize existing technologies.

[0107] The multimodal physical imaging module provided in this embodiment is mainly used to perform multi-dimensional imaging acquisition of the sample to be tested (such as biological cells, microorganisms, etc.) in the culture dish 3, and to obtain three-modal fused image data including polarization characteristics, multispectral characteristics and photometric stereo characteristics, so as to provide comprehensive physical imaging information for subsequent sample analysis and feature extraction.

[0108] In this embodiment, the multimodal physical imaging module can be electrically connected to the main control unit, which centrally controls the working timing and parameter adjustment of each component to ensure the synchronization and accuracy of imaging acquisition. The polarization angle of the rotatable polarizer assembly 1c can be manually adjusted in four positions or automatically adjusted by the main control unit.

[0109] The rotatable polarizer assembly 1c is used to acquire images at four polarization angles: 0°, 45°, 90°, and 135°, enabling multi-dimensional capture of the polarization characteristics of the sample. The polarizer is a high extinction ratio linear polarizer with an extinction ratio of no less than 1000:1, ensuring effective polarization filtering and reducing stray light interference.

[0110] The multispectral light source array 1b is circumferentially arranged on the image acquisition component 4a, and includes at least three bands: a 405nm band light source, a white light source, and a red light source. Time-division multiplexing is used to acquire images of each band separately to realize the acquisition of multispectral characteristics of the sample. The 405nm band is the ultraviolet band, which is mainly used to excite the fluorescence characteristics of the sample. The white light source is used to acquire the conventional morphological image of the sample, and the red light source is used to enhance the detail contrast of the sample to adapt to the imaging needs of different samples.

[0111] Each LED light source in each band is equipped with an independent driving circuit. The main control unit controls the on / off timing of the driving circuit to achieve time-division multiplexing control. That is, only one band of light source is started at a time. After the image of that band is acquired, the current band light source is turned off and the next band light source is started. This cycle is repeated to avoid light interference between different band light sources and ensure the purity of the images in each band.

[0112] In addition, the brightness of each band of light source can be independently adjusted by the main control unit. The illumination intensity of each band can be flexibly adjusted according to the optical characteristics of the sample to avoid excessive brightness causing saturation of sample details and excessive darkness causing blurring of details, thereby further improving the quality of multispectral images.

[0113] A 1d stereo LED light source array is used to achieve photometric stereo imaging and acquire images from at least three different lighting directions. It contains at least three independently controlled light sources, each of which can independently adjust its brightness and switching sequence. By controlling the illumination state of different light sources, different lighting directions are simulated, thereby acquiring the three-dimensional morphological information of the sample.

[0114] Specifically, the three light sources are arranged sequentially from top to bottom, and each light source consists of several evenly distributed LED beads. By lighting up the three light sources individually or in combination, sample images under three different lighting directions are obtained, providing basic data for the three-dimensional reconstruction of photometric stereo imaging.

[0115] The switching sequence of each light source layer is uniformly controlled by the main control unit and synchronized with the acquisition sequence of the imaging unit. This ensures that the imaging unit only acquires images after a certain light source layer has been lit and stabilized. The brightness of each light source layer can be adjusted independently to adapt to the reflective characteristics of different samples, avoiding excessively strong or weak reflections that could affect the accuracy of 3D feature extraction.

[0116] The electrically driven rotating stage 1a is used to rotate the culture dish, ensuring uniform illumination of all areas of the sample in conjunction with the acquisition of light from different angles. It also enables multimodal imaging of the sample from different orientations, improving the comprehensiveness of the imaging. The stage mainly consists of a rotating platform located inside the operating table of the main frame 2, a drive motor, and a culture dish fixing slot position sensor on the electrically driven rotating stage 1a. The drive motor is a servo motor, offering high rotational precision and enabling continuous 360° rotation. The rotational speed can be adjusted via the main control unit.

[0117] The operation of this multimodal physical imaging module is uniformly controlled by the main control unit, and the specific steps are as follows:

[0118] S101, place the culture dish 3 to be tested in the fixed slot of the electric rotating stage 1a, adjust the position of the culture dish 3 to ensure that the center of the culture dish 3 is aligned with the central axis of the optical path, start the module through the main control unit, complete the initialization of each component, adjust the polarizer to 0°, turn off the light source array, and reset the rotating platform to the initial position.

[0119] S102, Polarization Image Acquisition: The main control unit or manual control rotates the polarizer assembly sequentially to 0°, 45°, 90°, and 135°. After each angle stabilizes, the imaging unit is controlled to acquire the polarization image at the corresponding angle. After acquisition, the polarizer is reset to 0°.

[0120] S103, Multispectral Image Acquisition: Using time-division multiplexing, the main control unit sequentially starts the 405nm band light source, white light source, and red light source. After each band light source is started and stabilized, the imaging unit is controlled to acquire the corresponding band of multispectral image. After acquisition is completed, all multispectral light sources are turned off.

[0121] S104, Photometric Stereo Image Acquisition: The main control unit sequentially and individually illuminates the three layers of the 1d stereo LED light source array. After each layer of light source is illuminated and stabilized, the imaging unit is controlled to acquire the image of the corresponding illumination direction. If it is necessary to combine illumination directions, two or three layers of light sources can be controlled to be illuminated simultaneously to acquire the image under combined illumination.

[0122] S105, omnidirectional acquisition: The main control unit controls the electric rotating stage to rotate at a constant speed, or rotate step by step according to a preset angle, and repeats the above steps synchronously to acquire multimodal image data of the sample from different directions, ensuring that all areas of the sample can be fully acquired.

[0123] S106 After acquisition, the main control unit transmits all polarization images, multispectral images, and photometric stereo images to the data processing unit for three-modal fusion processing to form complete multimodal fused image data for subsequent analysis.

[0124] In this embodiment, it is necessary to acquire three-modal fused image data of polarization characteristics, multispectral characteristics, and photometric stereo characteristics. Specifically, polarization characteristic acquisition requires sequentially acquiring images at four polarization directions: 0°, 45°, 90°, and 135°. Let the intensities of the four acquired images be... , , , According to polarization optics theory, the Stokes vector is calculated as follows:

[0125]

[0126] Further calculations were performed on the degree of linear polarization (DoLP) and the angle of polarization (AoLP):

[0127]

[0128] Among them, DoLP ranges from 0 to 1. The higher the DoLP value, the stronger the polarization characteristic of the reflected light of the pixel, which usually corresponds to the specular reflection area; AoLP represents the angle of polarization direction and is used to help determine the source of reflection.

[0129] Multispectral characterization was performed using time-division multiplexing, sequentially illuminating light sources in three bands: 405nm, white light, and red light, and acquiring three single-band images for each band. The 405nm wavelength, with its short wavelength and weak penetration, was primarily used to enhance surface colony details; white light provided full-spectrum information; and red light enhanced contrast for specific colonies on certain chromogenic media. The three images are denoted as follows: , , .

[0130] Photometric stereo characteristic acquisition requires controlling the sequential illumination of three layers of a stereo LED light source array, with each layer corresponding to a specific light source direction vector when illuminated. ( Three images were captured under different lighting directions and denoted as follows: , , .

[0131] The physical-neural hybrid reflex decoupling network is further illustrated in the following embodiments.

[0132] First, the acquired multimodal images need to be input into the physical constraint layer. This layer is based on the principles of optical physics and first calculates the initial physical values ​​of the specular reflection component and the diffuse reflection component. The entire calculation process does not include trainable parameters.

[0133] S201, Photometric stereoscopic surface normal reconstruction. Based on three photometric stereoscopic images, assuming the surface of the petri dish is Lambertian, the image intensity... relative to the direction of the light source Surface normal and albedo satisfy:

[0134]

[0135] Written in matrix form:

[0136]

[0137] Since the direction of the light source is known and linearly independent, it can be solved using the least squares method. Then normalize to obtain the unit normal vector. Surface albedo Determined by the scaling factor during the solution process, it reflects the surface reflectivity.

[0138] S202, Initial value separation of fused reflection components. This invention proposes a method for fusing polarization information (DoLP) and geometric information, i.e., normal. The initial value calculation formula for the reflection component. Let the total radiation intensity be... Taken from white light image Alternatively, a weighted average of each band can be taken. Define the geometric factor. for:

[0139]

[0140] in, The line of sight is directed from the optical center of the camera towards the surface of the petri dish. It is a half-angle vector. This is the specular reflection index, typically ranging from 50 to 200, with a larger value used for smooth surfaces. This geometric factor originates from the Torrance-Sparrow micro-element model and is used to characterize the variation of specular reflection intensity with angle.

[0141] By fusing polarization and geometric information, the initial value of the diffuse reflection component is separated. Initial value of specular reflection component :

[0142]

[0143] in, This is an adjustable weighting coefficient (ranging from 0.5 to 1.0) used to balance the contributions of polarization and geometric information. The principle behind this formula is that specular reflection regions typically have high DoLP values ​​and satisfy specific geometric angular relationships, i.e. The DoLP value is relatively large, therefore multiplying the two can enhance the recognition sensitivity of specular reflection areas. Diffuse reflection areas, on the other hand, have low DoLP values. Also relatively small, the product approaches 0, therefore the initial value of the diffuse reflection component approaches 0. This aligns with physical intuition.

[0144] S203, Architecture and Training of Deep Separation Networks. Initial Physical Values ​​Output by the Physical Constraint Layer. , The original multimodal images are used as input to the deep separation network. The network employs an encoder-decoder structure. The encoder consists of five convolutional blocks, each containing a convolutional layer, a batch normalization layer, and a ReLU activation function to extract multi-scale features. The decoder progressively restores spatial resolution through transposed convolutions and fuses features from corresponding encoder layers via skip connections to preserve detail.

[0145] A cross-modal attention mechanism is introduced between the encoder and decoder. Assume the input features include... There are 10 modes, including polarization mode, multispectral mode, and photometric stereo mode. The features extracted for each mode are as follows: ( Cross-modal attention weights The calculation is as follows:

[0146]

[0147] Here, AvgPool is a global average pooling algorithm, and MLP is a multilayer perceptron containing two fully connected layers. The final fused features are: This mechanism enables the network to adaptively focus on the mode that contributes the most to the reflection separation of the current pixel.

[0148] The deep separation network is trained using a joint loss function. The training dataset contains approximately 5000 multimodal images, covering different culture medium types, colony densities, and humidity conditions. Joint loss function. Includes three items:

[0149]

[0150] in, As the weighting coefficient, this embodiment takes... , , .

[0151] Physical consistency loss This is a unique loss term in this invention, and its calculation formula is as follows:

[0152]

[0153] In the formula, and To deeply separate the diffuse and specular reflection components output by the network, This represents the L2 norm. This loss term forces the network output to be consistent with the calculation results of the physical constraint layer, enabling the network to learn both the data distribution characteristics and the laws of optical physics during training. The principle is that the initial physical values ​​already have high accuracy in most areas, and the network only needs to correct in complex areas where the physical model fails, such as scratches or extremely wet surfaces, rather than learning reflection separation from scratch, thereby improving training efficiency and generalization ability.

[0154] Image reconstruction loss for:

[0155]

[0156] This loss ensures that the original input image can be restored after the separation results are recombined, avoiding information loss.

[0157] Combating losses The discriminator loss in a standard generative adversarial network is adopted. The discriminator network judges whether the separation results match the data distribution of real colony images, thereby improving the realism of the generated images.

[0158] The training process uses the Adam optimizer, with an initial learning rate set to... The batch size is 8, and the training lasts for 200 epochs. After training is complete, the network weight parameter file is output.

[0159] S204, Model Conversion and Deployment. The trained PyTorch model is exported in ONNX format to generate an intermediate representation file. Then, NVIDIA TensorRT is used for FP16 precision quantization and layer fusion optimization, fusing convolutional layers, batch normalization layers, and ReLU activation layers into a single computation node, significantly reducing inference latency. The optimized inference engine file (.engine) is loaded onto the NVIDIA Jetson Orin NX platform. The device's main control program calls the underlying C++ inference interface through the Java Native Interface (JNI), receiving multimodal image data and returning reflection decoupling results in real time, with single-frame processing time controlled within 50 milliseconds.

[0160] The following embodiments further illustrate the multi-scale colony generation and repair network.

[0161] After reflection decoupling, diffuse reflection dominates the image. A small number of residual reflective areas may still remain, mainly appearing at the edges of the petri dish, along scratches, or on extremely wet surfaces. The system will use specular reflection to dominate the image. Areas with intensity exceeding a preset threshold are identified as reflective residual areas, and a binary mask is generated. .

[0162] Will and A multi-scale colony generation and repair network is proposed. This network adopts a conditional generative adversarial network (cGAN) architecture, with a U-Net generator containing a downsampling encoder and an upsampling decoder. Its innovation lies in introducing a colony morphology prior constraint layer and a spatial adaptive normalization layer in the decoder part.

[0163] The colony morphology prior constraint layers are located in layers 3, 4, and 5 of the decoder. Each constraint layer contains a pre-trained colony shape discriminator. This discriminator is trained based on historical colony images, taking a local image patch as input and outputting the probability of whether that image patch conforms to the colony morphology. The discriminator network structure is a small convolutional network, and the training data contains approximately 100,000 labeled individual colony image patches. After training, the discriminator parameters are fixed and used to constrain the generation process of the repair network.

[0164] Colony morphology prior constraint loss Defined as:

[0165]

[0166] in, Indicates scale level. This represents the number of image patches in the repaired area at this scale. For colony shape discriminators at corresponding scales, For the repair area of ​​the first Each image patch. This loss encourages the discriminator to identify the repaired region as conforming to colony morphology, i.e. It approaches 1. The principle of this loss term is: to embed the biological morphological prior in a differentiable manner into the training of the generative network, so that the network is subject to shape constraints while generating textures, thus avoiding the generation of irregular artifacts or non-colony structures.

[0167] Spatial adaptive normalization layers will make specular reflections dominate the image. The reflected light intensity information is used as a modulation parameter to control the generation process of the repair network. Let the features of a certain layer of the decoder be... Reflection intensity diagram After sampling to Same spatial dimensions. The formula for spatial adaptive normalization is:

[0168]

[0169] in, and Feature maps Mean and standard deviation along the channel dimension and To pass through convolutional layers from The learned modulation parameters enable the network to adaptively adjust the generation process based on the reflectivity distribution, employing a more conservative generation strategy in areas of high reflectivity and generating textures more freely in areas of low reflectivity.

[0170] The repair network training also employs a joint loss mechanism, including adversarial loss, L1 reconstruction loss, and the aforementioned colony morphology prior constraint loss. Training data consists of historical colony images, with damaged and intact images constructed as training pairs by artificially synthesizing reflective regions.

[0171] The following embodiments provide a detailed description of the repair confidence assessment and LIMS linkage process.

[0172] After the repair is completed, the system calculates a repair confidence score for each repaired region. Based on information from three dimensions:

[0173]

[0174] in:

[0175] The output probability of the colony morphology prior constraint layer is the probability that the discriminator determines that the repaired area conforms to the colony morphology. The value ranges from 0 to 1, and the closer it is to 1, the more realistic the morphology is.

[0176] To calculate the structural similarity index between the repaired region boundary and the surrounding normal region, a ring-shaped region extending 10 pixels outward from the repaired region is used as a reference. The closer to 1, the more natural the texture transition;

[0177] The reconstruction error of multi-scale features is obtained by inputting the repaired region into a pre-trained autoencoder and calculating the L2 distance with the reconstruction result. The smaller the error, the more likely the repaired region conforms to the characteristics of normal colonies. The pre-trained autoencoder is trained on normal colony images.

[0178] As the weighting coefficient, this embodiment takes... , , ,satisfy .

[0179] This confidence score integrates three dimensions: biological morphological plausibility, texture continuity, and feature consistency, to comprehensively assess the restoration quality. If the system determines that the repaired area lacks reliability, it sends a labeling message to the Laboratory Information Management System (LIMS) via API, including the petri dish number, the coordinates of the low-confidence area, and the confidence score. The LIMS system automatically marks the sample in the testing task list as "recommended for manual review" and highlights the low-confidence area on the device's touchscreen to prompt the operator to conduct a thorough review.

[0180] The colony counting and result output process is described in the following examples.

[0181] For confidence scores For areas with high confidence, the system incorporates the repair results into the final image; for low-confidence areas, the system retains the unrepaired state in the original diffuse dominant image, awaiting manual review by the operator for identification. The final repaired image is then input into the colony counting module.

[0182] The colony counting module employs a deep learning-based instance segmentation network (Mask R-CNN) to detect and segment colonies in the repaired image. The segmentation results undergo post-processing, such as removing noise regions smaller than 3 pixels and merging adjacent segmented regions less than 5 pixels apart, before the total colony count is determined. The counting results automatically calculate the colony forming units (CFU) in the original sample based on the dilution factor.

[0183] Finally, the counting results, restoration confidence scores, original and restored images, and heatmaps of reflective area distribution are all uploaded to the LIMS system, automatically generating original records and test reports. The entire process achieves a high degree of automation and intelligence in microbial colony counting.

[0184] The technical terms, principles, or means related to the technical solutions of the present invention mentioned in the above embodiments, which are not described in detail above, are all well-known technologies or common practices that are known to those skilled in the art.

[0185] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for multi-modal image acquisition and processing for pipelined microbial colony automated counting, the method comprising: The method includes one or a combination of the following steps: ​ The culture dish under test is imaged using a multimodal physical imaging module to obtain three-modal fused image data that includes at least polarization characteristics, multispectral characteristics, and photometric stereoscopic characteristics. The polarization characteristics include at least three polarization images at different polarization angles, the multispectral characteristics include at least two images in different spectral bands, and the photometric stereoscopic characteristics include at least three images illuminated by light sources at different angles. A physical-neural hybrid reflex decoupling network is established, and the trimodal fused image data is input into the physical-neural hybrid reflex decoupling network. The structure of the network includes: A physical constraint layer is used for the Stokes vector calculation model based on polarization imaging and the surface normal reconstruction model of photometric stereo to calculate the physical initial values ​​of the specular reflection component and the diffuse reflection component, respectively. A deep separation network is used to receive the physical initial value and the original multimodal image, and output a high-precision diffuse reflection dominant image and specular reflection dominant image through multi-scale feature extraction and fusion. The physical-neural hybrid reflex decoupling network is trained using a joint loss function, which includes a physical consistency loss term and an image reconstruction loss term. Based on the diffuse dominant image and the specular dominant image, a multi-scale colony generation and repair network is used to repair the texture of the reflective residue area. The repair network introduces a colony morphology prior constraint layer in the decoder part. The prior constraint layer is used to constrain the generation result to conform to the geometric morphological characteristics of the colony. Output the repaired image and perform colony counting based on the repaired image.

2. The method for multi-modal image acquisition and processing for automated counting of microbial colonies according to claim 1, wherein, The multimodal physical imaging module includes at least: A rotatable polarizer assembly for acquiring images at four polarization angles: 0°, 45°, 90°, and 135°. A multispectral light source array contains light sources in at least three bands: 405nm, white light, and red light, and images of each band are acquired separately using time-division multiplexing. A three-dimensional LED light source array contains at least three independently controlled light sources, each of which can independently adjust its brightness and switching sequence, to achieve photometric stereo imaging and acquire images from at least three different lighting directions; An electric rotating stage is used to rotate the petri dish to accommodate different light angles for collection.

3. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 1, characterized in that, The calculation process for the physical constraint layer includes: Based on the images at different polarization angles, the Stokes vector is calculated, and the degree of linear polarization and polarization angle are further calculated. Based on the images illuminated by at least three different light sources, the normal map of the petri dish surface is reconstructed using a photometric stereo method; Based on the linear polarization degree and normal diagram, the initial values ​​of the specular reflection component and the diffuse reflection component are separated using the Fresnel reflection model.

4. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 1, characterized in that, The physical-neural hybrid reflex decoupling network is established and deployed through the following steps: A physical constraint layer without trainable parameters is constructed, which is implemented through programming based on the physical principles of optical imaging. A deep separation network with trainable parameters is constructed. The deep separation network is trained on a training dataset using a deep learning framework to learn a nonlinear mapping from physical initial values ​​and original multimodal images to accurate reflection separation results. The trained deep separation network is cascaded with the physical constraint layer to form a complete physical-neural hybrid reflex decoupling network, which is then deployed to the edge computing unit built into the microbial colony counting device for real-time processing of the acquired multimodal image data.

5. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 4, characterized in that, The construction of the physical constraint layer includes a deterministic calculation module pre-built by programming based on the physical principles of optical imaging. The physical constraint layer has the following calculation process: calculating the Stokes vector and linear polarization degree based on multi-angle polarization images, reconstructing the surface normal map based on multi-angle light source images using photometric stereo method, and analytically calculating the initial physical values ​​of specular reflection and diffuse reflection components according to Fresnel reflection theory. The physical constraint layer does not contain trainable parameters. The construction of the deep separation network includes building a neural network architecture for the deep separation network on a high-performance computing server based on the PyTorch or TensorFlow deep learning framework. The deep separation network adopts an encoder-decoder structure and introduces a cross-modal attention mechanism. The constructed physical constraint layer is cascaded with the deep separation network to form a complete physical-neural hybrid reflection decoupling network. Multimodal image data containing at least polarization characteristics, multispectral characteristics, and photometric stereo characteristics are collected as training samples. The hybrid architecture model of the physical constraint layer and the deep separation network is trained end-to-end using a joint loss function containing a physical consistency loss term and an image reconstruction loss term. The physical consistency loss term constrains the output of the deep separation network to maintain consistency with the physical initial value of the physical constraint layer output until the model converges. The trained network weight parameter file is then output. The deep separation network and physical constraint layer concatenation involves exporting the trained network weight parameter file into an intermediate format using the Open Neural Network Exchange Format (ONNX), and then using the TensorRT inference optimization tool for precision quantization and layer fusion optimization to generate an optimized inference engine file suitable for edge computing platforms. The optimized inference engine file is then loaded into the edge computing unit built into the microbial colony counting device. The edge computing unit is selected from NVIDIA Jetson series embedded platforms or industrial control computers that support CUDA. The edge computing unit interacts with the device's main control program through a Java native interface or inter-process communication, receives the collected multimodal image data, and returns the reflection decoupling results in real time.

6. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 1, characterized in that, After performing colony counting based on the restored image, the method further includes: A repair confidence score is calculated for each repaired region. The repair confidence score is calculated based on the output probability of the colony morphology prior constraint layer, the texture consistency between the repaired region and the surrounding region, and the reconstruction error of multi-scale features. If the repair confidence score is lower than the preset threshold, the petri dish will be automatically marked as a manually verified sample in the laboratory information management system, and the low-confidence repair area will be marked in the image.

7. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 1, characterized in that, Before imaging the culture dish to be tested using the multimodal physical imaging module, the method further includes obtaining a thermal map of the reflective area distribution of the culture dish through a pre-scanning mode, and dynamically adjusting the acquisition parameters of the multimodal physical imaging module based on the thermal map of the reflective area distribution. The acquisition parameters include polarization angle, light source angle, exposure time, and spectral band combination.

8. The multimodal image acquisition and processing method for automatic counting of microbial colonies in an automated production line according to claim 7, characterized in that, The dynamic adjustment includes: automatically increasing the number of polarization angle samples when the area of ​​the reflective region exceeds a preset threshold; and performing secondary fine scanning imaging on the severely reflective local area by controlling the electric rotating stage and the three-dimensional LED light source array to obtain higher resolution multimodal image data.

9. A multimodal image acquisition and processing system for implementing the method of any one of claims 1-8, characterized in that, It should include at least the following modules: The multimodal physical imaging module is used to acquire fused image data of the polarization characteristics, multispectral characteristics, and photometric stereoscopic characteristics of the culture dish under test; The dynamic imaging control module is used to adaptively adjust the acquisition parameters of the imaging module based on the pre-scan results, and to perform secondary fine scanning imaging on areas with severe glare. A physical-neural hybrid reflection decoupling module is used to separate diffuse reflection components and specular reflection components based on multimodal fused image data through a deep neural network with embedded physical constraints. A multi-scale colony generation and repair module is used to perform texture repair on reflective residual areas based on separation results. The repair module includes a colony morphology prior constraint layer. The confidence assessment and labeling module is used to calculate the confidence score of the repaired area and manually review and label low-confidence areas. The colony counting module is used to count colonies based on the repaired image.

10. An intelligent microbial colony counting device based on assembly line operation, characterized in that, It includes the multimodal image acquisition and processing system as described in claim 9.

Images