Bubble detection method based on multi-modal input and improved target detection network

Abstract

The invention belongs to the field of target detection, particularly relates to a bubble detection method based on multi-modal input and an improved target detection network, and aims to solve the problems that an existing bubble detection model is sensitive to environmental parameter changes, depends on a large amount of annotated data, is poor in migration adaptability, lacks multi-modal information fusion and physical law constraints, and is poor in migration efficiency. The precision is low and the cost is high. According to the method, gas-liquid two-phase flow images and temperature, pressure and flow velocity parameters are collected, a target detection network is constructed, and multi-scale visual features are extracted. After physical parameters are coded into high-dimensional features, a comprehensive feature map is generated through dynamic weighted fusion, and a detection head is input to output a bounding box and convert the bounding box into an actual size. An ideal gas equation and fluid mechanics are adopted to construct a physical constraint loss function correction size, environment self-adaption is achieved in combination with a dynamic weight adjustment and fine adjustment mechanism, and the detection precision and physical consistency are improved. According to the method, the bubble detection precision and the environmental adaptability are improved, and the labeling cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of target detection, and in particular relates to a bubble detection method based on multimodal input and an improved target detection network. Background Art

[0002] Gas-liquid two-phase flow, a complex flow pattern, is widely found in aerospace, nuclear engineering, shipbuilding and weaponry, energy and chemical engineering, environmental protection, petroleum, and light industries. Gas-liquid two-phase flow, defined as the simultaneous presence of gas and liquid phases, has a profound impact on the understanding, prediction, manipulation, control, and optimization of related physical phenomena.

[0003] Bubbles and liquid, as key components of gas-liquid two-phase flow, determine the contact area, heat exchange, and material exchange between the two phases, directly influencing reaction rates in industrial production. For example, in the energy and chemical industries, bubbles are often used to accelerate the mixing rate of substances in a reaction vessel, increasing the reaction area and heat exchange per unit time, and shortening chemical reaction times. Understanding the growth and motion of bubbles can effectively control these chemical processes. In the nuclear industry, studying the motion of bubbles can provide insights into the mechanisms of nucleate pool boiling heat transfer models. Existing models, starting with a single bubble, investigate its formation, growth, and detachment within nucleate pool boiling, as well as the transient heat exchange associated with this process. The models also include methods for predicting bubble dynamics parameters during nucleate pool boiling heat transfer, including bubble detachment diameter and detachment frequency. Therefore, accurate monitoring of bubble flow provides a deeper understanding of multiphase flow conditions and holds significant theoretical and practical value for establishing multiphase flow theory and guiding reactor design and scale-up.

[0004] Bubble detection methods can be categorized as invasive and non-invasive. Invasive methods frequently calculate physical quantities such as temperature, pressure, or velocity within the flow region, thereby determining their average and fluctuation values. Their advantage is that sensors can be placed in hard-to-reach locations. These methods include capillary aspiration probes, conductivity probes, fiber optic probes, and wire mesh sensors. Non-invasive methods, on the other hand, primarily utilize optical techniques, which have become increasingly popular with the advancement of laser and computer technology. These methods include interferometric particle imaging, phase Doppler velocimetry, and digital image analysis. Unlike invasive probe-based methods, non-invasive methods do not disturb the fluid under investigation, offering improved measurement accuracy and reliability. Among these non-invasive methods, digital image analysis can accurately measure irregular or non-spherical bubbles over a wide range, making it a widely used alternative to traditional bubble motion detection. Both invasive and non-invasive methods involve signal analysis and visualization of the acquired data through image processing. However, these methods suffer from low detection efficiency and large errors. Furthermore, they require a certain level of expertise from the inspector, making real-time measurement and visualization of multiphase flow dynamic parameters difficult.

[0005] Deep learning can be used efficiently for image processing and has developed rapidly in recent years. Currently, deep learning-based target detection algorithms have been used for bubble image detection in the field of gas-liquid two-phase flow. In the field of target detection, the YOLO (You Only LookOnce) series of algorithms are widely used in various scenarios due to their high efficiency and real-time performance. YOLOv8 focuses on the balance between accuracy and speed, using CSPDarknet as the backbone network. It can extract multi-scale visual features from the input image and generate the bounding box, confidence, and category prediction of the target through the detection head. This method performs well in standard image detection tasks, such as detecting targets such as people and vehicles in natural scenes. However, in specific industrial scenarios, existing technologies face some limitations.

[0006] Currently, bubble detection in gas-liquid two-phase flow relies primarily on computer vision and deep learning technologies, particularly object detection algorithms such as YOLOv8. These methods process image data of gas-liquid two-phase flows to identify and extract characteristic information such as bubble location, size, and shape, providing important quantitative tools for studying gas-liquid two-phase flow dynamics and for industrial process monitoring and control.

[0007] Existing bubble detection models are generally highly sensitive to changes in environmental conditions. When environmental physical parameters such as temperature, pressure, and flow rate change, the optical properties (such as refractive index and reflectivity) and physical properties (such as size, shape, and distribution) of bubbles change significantly, leading to a sharp decline in model performance. For example, when the parameters of the training environment and the application environment differ significantly, the detection accuracy of the model decreases significantly, and the model lacks the ability to adapt to changes in bubble characteristics under extreme conditions (such as high temperature, high pressure, or high flow rate).

[0008] Existing bubble detection technology primarily relies on image feature extraction and target recognition, failing to fully integrate multi-source information. It ignores the direct impact of key environmental physical parameters such as temperature, pressure, and flow rate on bubble morphology, and lacks clear constraints on the basic physical laws of gas-liquid two-phase flow. Furthermore, model migration adaptability is poor. When the application scenario changes, existing models face serious transfer learning issues, requiring the collection of large amounts of labeled data and retraining for each new environmental condition. Furthermore, the reliance of purely visual deep learning methods on large amounts of labeled data leads to high application costs. Sufficient labeled data must be collected under different environmental conditions, which is a huge workload.

[0009] Given these shortcomings, there is an urgent need to develop a bubble detection method that organically combines multimodal information input with physical constraints, achieving high adaptability and high-precision detection in changing environments while ensuring the physical consistency of the detection results. This method should significantly reduce the reliance on labeled data and possess strong knowledge transfer capabilities to meet the requirements of gas-liquid two-phase flow analysis in various industrial scenarios and scientific research needs.

[0010] Based on this, the present invention proposes a bubble detection method based on multimodal input and improved target detection network. Summary of the Invention

[0011] In order to solve the above-mentioned problems in the prior art, namely, the existing bubble detection model is sensitive to changes in environmental parameters, relies on a large amount of labeled data, has poor migration adaptability, and lacks multimodal information fusion and physical law constraints, resulting in low accuracy and high cost, the present invention provides a bubble detection method based on multimodal input and an improved target detection network, which includes:

[0012] Synchronously collecting image data of the gas-liquid two-phase flow and environmental physical parameters and performing preprocessing operations, wherein the environmental physical parameters include at least temperature, pressure and flow rate;

[0013] Extracting multi-scale visual features of the pre-processed image data based on a pre-built and trained object detection network, and encoding the pre-processed environmental physical parameters to generate high-dimensional physical features; wherein the object detection network is constructed based on multiple groups of convolutional downsampling layers, multiple cross-stage partial network modules, and a spatial pyramid pooling fusion module, and extracts multi-scale visual features from shallow, middle, and deep layers respectively;

[0014] Based on the physical parameters of the environment, the dynamic weighted feature fusion module is input, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map;

[0015] Inputting the comprehensive feature map into a detection head network, outputting a bubble detection bounding box, extracting bubble parameters of the bubble based on the bubble detection bounding box, and converting them into actual physical dimensions based on a calibrated scale factor; the bubble parameters include at least size, shape, and position parameters;

[0016] A physical constraint loss function is constructed based on the ideal gas state equation and fluid mechanics principles to correct the actual physical size of the bubble, and the network parameters of the detection head network are corrected through the constraint loss.

[0017] Output the corrected actual physical size and physical consistency score, and adapt to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms.

[0018] Furthermore, the pre-processing operation includes:

[0019] The image data is subjected to denoising, contrast enhancement, background removal and normalization processing, and the environmental physical parameters are subjected to smoothing processing, outlier removal and standardization.

[0020] Furthermore, the target detection network specifically includes:

[0021] Five groups of convolutional downsampling layers, four cross-stage partial network modules, and a spatial pyramid pooling fusion module;

[0022] The five groups of convolutional downsampling layers include a first convolutional downsampling layer, a second convolutional downsampling layer, a third convolutional downsampling layer, a fourth convolutional downsampling layer, and a fifth convolutional downsampling layer;

[0023] The four cross-stage partial network modules include a first cross-stage partial network module, a second cross-stage partial network module, a third cross-stage partial network module and a fourth cross-stage partial network module;

[0024] The first convolution downsampling layer is used to receive the preprocessed image data, the output of the first convolution downsampling layer is connected to the input of the second convolution downsampling layer, the output of the second convolution downsampling layer is connected to the input of the first cross-stage partial network module, the output of the first cross-stage partial network module is connected to the input of the third convolution downsampling layer, the output of the third convolution downsampling layer is connected to the input of the second cross-stage partial network module, the output of the second cross-stage partial network module is connected to the input of the fourth convolution downsampling layer, the output of the fourth convolution downsampling layer is connected to the input of the third cross-stage partial network module, the output of the third cross-stage partial network module is connected to the input of the fifth convolution downsampling layer, the output of the fifth convolution downsampling layer is connected to the input of the fourth cross-stage partial network module, and the output of the fourth cross-stage partial network module is connected to the spatial pyramid pooling fusion module;

[0025] The second cross-stage partial network module, the third cross-stage partial network module and the spatial pyramid pooling fusion module respectively output shallow features, middle features and deep features to the multi-scale feature fusion module, and output multi-scale visual features.

[0026] Furthermore, the pre-processed environmental physical parameters are encoded to generate high-dimensional physical features, and the method is as follows:

[0027] The environmental physical parameters are arranged in a set order to form an input vector, which is input into a pre-built multi-layer perceptron to output high-dimensional physical features; the multi-layer perceptron includes an input layer, a first fully connected layer, a first normalization layer, a first activation function, a Dropout layer, a second fully connected layer, a second normalization layer, a second activation function and an output layer connected in sequence.

[0028] Furthermore, the environmental physical parameters are input into a dynamic weighted feature fusion module, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map. The method is as follows:

[0029] Input the high-dimensional physical feature vector into the fully connected layer to generate initial weights, and perform normalization to obtain visual feature weights and physical feature weights; the sum of the visual feature weights and the physical feature weights is 1;

[0030] Reshape the physical feature vector into a single-pixel multi-channel feature map and expand it to the same spatial size as the visual feature map through interpolation;

[0031] The multi-scale visual features are adjusted according to the weights and then spliced ​​with the spatialized physical features of the environment. The number of channels is compressed through convolution operations to generate the final comprehensive feature map.

[0032] Furthermore, the detection head network comprises three convolutional layers connected in sequence, the first two layers use a three-by-three convolution kernel, the number of input and output channels is the same and maintains a preset number, and each layer performs edge padding;

[0033] The third layer uses a single-channel convolution kernel, and the number of output channels is set according to the bounding box coordinates, detection confidence, and the total number of bubble categories;

[0034] After each convolution layer, batch normalization and nonlinear activation function operations are performed in sequence;

[0035] The network output is decoded into bounding box coordinates, confidence, and category information. Based on the non-maximum suppression algorithm, the bubble detection results are filtered by pre-setting the intersection-over-union overlap threshold and confidence threshold to ensure that a single bubble corresponds to a unique detection box.

[0036] Furthermore, bubble parameters of the bubble are extracted based on the bubble detection bounding box, and converted into actual physical size based on a calibrated scale factor, and the method is as follows:

[0037] Calculating the pixel area of ​​the bubble bounding box based on the detected bubble bounding box;

[0038] Multiplying the pixel area by a pre-calibrated scale factor to convert it into an actual physical area;

[0039] Correct the perspective distortion of bubbles in the edge area of ​​the image according to the camera calibration parameters;

[0040] Calculating a ratio of the width to the height of the bubble's bounding box, and determining whether the bubble is horizontally flattened or vertically stretched based on the ratio;

[0041] According to the proportional relationship between the contour area and the area of ​​its minimum circumscribed circle, the degree to which the bubble is close to a perfect circle is evaluated based on the proportional relationship;

[0042] Performing ellipse fitting on the contour, and quantifying the flattening degree of the bubble shape by the length ratio of the major axis to the minor axis of the ellipse;

[0043] The bubble's horizontal flattening or vertical stretching state, the bubble's degree of closeness to a perfect circle, and the degree of flattening of the bubble shape are taken as bubble shape parameters;

[0044] Wherein, for the case where the bubble shape within the bounding box is irregular, adaptive threshold segmentation is used to extract the precise outline; the area of ​​the pixel region actually occupied is calculated based on the outline, and the proportional coefficient conversion is applied again.

[0045] Furthermore, a physical constraint loss function is constructed based on the ideal gas state equation and the principles of fluid mechanics to correct the actual physical size of the bubble. The method is as follows:

[0046] Based on the ideal gas law, the correlation constraints between bubble volume, temperature and pressure are established, and the difference between the detected volume and the theoretical volume is calculated as the size constraint loss;

[0047] Based on the principles of fluid dynamics, a mapping relationship between the bubble aspect ratio and flow velocity is established, and the difference between the shape prediction value and the theoretical value is calculated as the shape constraint loss;

[0048] Based on the balance between gravity and buoyancy, a statistical model of bubble vertical position distribution and flow velocity is constructed, and the distribution deviation is calculated as the distribution constraint loss.

[0049] Combining size, shape, and distribution constraint losses into a comprehensive physical constraint loss function according to preset weights; wherein the preset weights are dynamically adjusted according to environmental parameters;

[0050] The model parameters of the detection head network are modified based on the results of the calculation of the comprehensive physical constraint loss function.

[0051] Furthermore, dynamic weights or small sample fine-tuning mechanisms can be adjusted to adapt to environmental changes. The method is as follows:

[0052] Determine whether changes in environmental conditions exceed a preset threshold;

[0053] If it does not exceed the limit, the weight of the physical features in the dynamic weighted fusion module is automatically increased according to the degree of deviation of the environmental parameters; the weight ratio of each constraint item in the physical constraint loss function is adjusted to adapt it to the current environmental characteristics; the confidence threshold and non-maximum suppression threshold in the post-detection processing parameters are adaptively adjusted;

[0054] If it exceeds, the parameters of the target detection network and the detection head network are kept fixed, and only the parameters of the dynamic weighted fusion module are updated; fine-tuning is performed using a learning rate lower than that in the basic training stage; and model adaptation is completed within a limited number of training rounds based on a small amount of new environment annotated data.

[0055] Furthermore, a consistency score of the calculation result is calculated based on each physical constraint value, and the consistency score is multiplied by the detection confidence to generate a comprehensive reliability score. Comprehensive reliability scores below a threshold are screened for manual review or automatic correction.

[0056] Beneficial effects of the present invention:

[0057] (1) This invention significantly improves the adaptability and stability of bubble detection in changing environments. By using environmental physical parameters (temperature, pressure, flow rate, etc.) and image data as model inputs, the system can perceive environmental changes and adjust detection strategies accordingly.

[0058] (2) The present invention ensures the physical consistency of the test results and improves their reliability and interpretability. By introducing a physical constraint loss function based on the principles of fluid mechanics, the system can autonomously assess whether the test results conform to objective physical laws and adjust any results that do not conform to the laws.

[0059] (3) The present invention achieves optimal integration of visual and physical information through a dynamic weighted feature fusion mechanism. This mechanism can adaptively adjust the importance weight of each modal information according to different environmental conditions, relying more on visual features in normal environments and more on physical features in abnormal environments.

[0060] (4) The present invention greatly reduces the dependence on labeled data and significantly reduces the cost of model application. Due to the introduction of physical constraints as additional supervisory signals, the system can learn more generalized feature representations from a small amount of labeled data. Experiments have shown that when using less labeled data, the present invention can still achieve the performance level of traditional methods using full data. This feature greatly reduces the cost of data collection and annotation, making the technology easier to quickly deploy in new scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0062] Figure 1 This is a schematic diagram of the overall process of a bubble detection method based on multimodal input and an improved target detection network of the present invention;

[0063] Figure 2 1 is a schematic diagram of a target detection network of a bubble detection method based on multimodal input and an improved target detection network according to the present invention;

[0064] Figure 3 It is a structural diagram of a multi-layer perceptron of a bubble detection method based on multimodal input and an improved target detection network of the present invention;

[0065] Figure 4 1. It is a flow chart of a physical constraint loss function of a bubble detection method based on multimodal input and an improved target detection network of the present invention;

[0066] Figure 5 Schematic diagram of the overall training process of a bubble detection method based on multimodal input and improved target detection network of the present invention;

[0067] Figure 6 This is a schematic diagram comparing the bubble recognition confidence of the algorithm before and after improvement of the bubble detection method based on multimodal input and improved target detection network of the present invention. DETAILED DESCRIPTION

[0068] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0069] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0070] The present invention provides a bubble detection method based on multimodal input and an improved target detection network, the method comprising:

[0071] Synchronously collecting image data of the gas-liquid two-phase flow and environmental physical parameters and performing preprocessing operations, wherein the environmental physical parameters include at least temperature, pressure and flow rate;

[0072] Extracting multi-scale visual features of the pre-processed image data based on a pre-built and trained object detection network, and encoding the pre-processed environmental physical parameters to generate high-dimensional physical features; wherein the object detection network is constructed based on multiple groups of convolutional downsampling layers, multiple cross-stage partial network modules, and a spatial pyramid pooling fusion module, and extracts multi-scale visual features from shallow, middle, and deep layers respectively;

[0073] Based on the physical parameters of the environment, the dynamic weighted feature fusion module is input, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map;

[0074] Inputting the comprehensive feature map into a detection head network, outputting a bubble detection bounding box, extracting bubble parameters of the bubble based on the bubble detection bounding box, and converting them into actual physical dimensions based on a calibrated scale factor; the bubble parameters include at least size, shape, and position parameters;

[0075] A physical constraint loss function is constructed based on the ideal gas state equation and fluid mechanics principles to correct the actual physical size of the bubble, and the network parameters of the detection head network are corrected through the constraint loss.

[0076] Output the corrected actual physical size and physical consistency score, and adapt to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms.

[0077] In order to more clearly illustrate the bubble detection method based on multimodal input and improved target detection network of the present invention, the following is combined with Figure 1Each step in the embodiment of the present invention is described in detail as follows:

[0078] Synchronously collecting image data of the gas-liquid two-phase flow and environmental physical parameters and performing preprocessing operations, wherein the environmental physical parameters include at least temperature, pressure and flow rate;

[0079] In this example, high frame rates and short exposure times are key to capturing fast-moving bubbles. Bubbles in gas-liquid two-phase flow move quickly, and conventional cameras can cause motion blur, affecting subsequent edge detection and shape analysis. Using a macro lens ensures that even small bubbles are clearly magnified.

[0080] The equipment uses a high-speed camera to capture bubble images. Because bubbles may move very quickly, a high-frame-rate camera can capture clear dynamic images. Therefore, the frame rate is required to be above 1000fps and the resolution is greater than 1920×1080 to ensure clear image details and an exposure time of less than 1ms to reduce blur caused by bubble movement.

[0081] The shooting environment is to place two layers of sulfuric acid paper behind a transparent vertical circular tube and shoot with a uniform flat soft light source for backlighting. This ensures clear images and reduces shadow interference. Image acquisition must be synchronized with the collection of environmental physical parameters. This can be achieved through timestamp alignment to ensure that each image frame has the corresponding environmental physical parameter data.

[0082] The image data preprocessing includes denoising, contrast enhancement, background removal, normalization and scaling. Specifically:

[0083] Denoising: Apply a Gaussian filter or a median filter to remove image noise. Sensor noise and environmental interference can introduce random noise into the image, affecting bubble edge detection. A Gaussian filter smoothes the image by weighted averaging the values ​​of surrounding pixels. A median filter replaces the central pixel with the median value of the surrounding pixels, making it particularly effective against salt and pepper noise.

[0084] Contrast Enhancement: Uses adaptive histogram equalization (CLAHE) to enhance bubble boundaries. In gas-liquid two-phase flow images, bubble boundaries can lack contrast due to uneven illumination or refraction. Adaptive histogram equalization significantly enhances edge contrast by performing histogram equalization within localized image regions while avoiding the over-enhancement of noise that can occur with standard histogram equalization.

[0085] Background Subtraction: Use background subtraction to eliminate static background interference. In gas-liquid two-phase flow experiments, static structures such as flow channel boundaries can interfere with bubble detection. Background subtraction highlights the moving bubble portion by subtracting a previously acquired background frame from the current frame.

[0086] Normalization: Normalizes pixel values ​​to the range [0, 1]. Deep learning models are often sensitive to the scale of the input data. Normalization reduces training instability caused by numerical range differences by scaling the original pixel values ​​(typically 0-255) to the range 0-1, accelerating network convergence.

[0087] Scaling: Scale the image to the model input size (e.g., 640×640). Deep learning models require a fixed input size. Image scaling ensures that original images of varying resolutions are converted to the uniform size required by the network while preserving the integrity of the image content.

[0088] This preprocessing step significantly improves image quality and consistency, providing an excellent data foundation for subsequent multimodal feature extraction and bubble detection.

[0089] Environmental physical parameter collection, the specific method is as follows:

[0090] In the present invention, the environmental physical parameter acquisition step is used to obtain key physical data in the gas-liquid two-phase flow environment in real time, providing an accurate basis for subsequent multimodal information fusion and physical constraints of the detection results. The physical sensors are mainly installed in the flow channel measurement section that is monitored synchronously with the high-speed camera. The specific locations include:

[0091] (1) Temperature sensor: It should be installed in the main body of the fluid, at the same or similar location as the bubble imaging area, to ensure the temporal and spatial consistency of the temperature data and the collected image, thereby accurately reflecting the temperature state of the local fluid environment. Using an immersion thermocouple or a wall-mounted non-contact infrared sensor, the measurement accuracy can reach ±0.1°C.

[0092] (2) Pressure sensor: Installed at the inlet and outlet of the measuring section, it forms a pressure gradient measurement. The pressure information plays an important role in the subsequent correction of bubble size according to the gas state equation and provides a basis for the calculation of physical confinement loss.

[0093] (3) Flow velocity sensor: The ultrasonic flow meter is installed upstream of the flow channel measurement section to avoid interfering with the flow field in the imaging area. Flow velocity data directly affects the dynamic behavior and shape changes of bubbles in the fluid.

[0094] The sensor sampling frequency must match or be an integer multiple of the image frame rate. For example, if the image frame rate is 100 fps, the sensor sampling frequency can be set to 100 Hz or higher. Each set of environmental physical parameters is timestamped with the corresponding image frame to ensure time alignment of multimodal data.

[0095] Environmental physical parameter preprocessing includes smoothing, outlier removal, and standardization, specifically:

[0096] In order to ensure the stability of environmental physical parameter data, the collected raw environmental physical parameter data such as temperature, pressure, flow rate, etc. are preprocessed. The specific steps include:

[0097] Data smoothing: A low-pass filter is used to smooth fluctuations in sensor readings. Sensor readings often contain high-frequency noise and fluctuations. The low-pass filter retains the low-frequency components of the signal (real physical changes) while filtering out the high-frequency components (noise and transient fluctuations), improving the reliability of environmental physical parameters.

[0098] Outlier detection and removal: Use the 3σ criterion to remove outliers. Sensors may experience transient anomalies that can cause outliers. The 3σ criterion, based on normal distribution theory, considers readings that are more than three standard deviations from the mean to be outliers. These values ​​are removed or replaced to reduce interference with subsequent analysis.

[0099] Parameter Standardization: Z-score normalization is performed on each environmental physical parameter. Different environmental physical parameters have different units and magnitudes. Z-score normalization converts all parameters to the same scale with a mean of 0 and a standard deviation of 1 by subtracting the mean and dividing by the standard deviation. This allows the model to fairly account for the impact of each parameter.

[0100] Through the above-mentioned data parameter preprocessing steps, the quality and stability of environmental physical parameter data can be significantly improved, providing reliable input data for multimodal information fusion and model training, thereby ensuring that the final detection results are physically reasonable and consistent.

[0101] Extracting multi-scale visual features of the pre-processed image data based on a pre-built and trained object detection network, and encoding the pre-processed environmental physical parameters to generate high-dimensional physical features; wherein the object detection network is constructed based on multiple groups of convolutional downsampling layers, multiple cross-stage partial network modules, and a spatial pyramid pooling fusion module, and extracts multi-scale visual features from shallow, middle, and deep layers respectively;

[0102] like Figure 2 As shown, in this embodiment, the target detection network specifically includes:

[0103] Five groups of convolutional downsampling layers, four cross-stage partial network modules, and a spatial pyramid pooling fusion module;

[0104] The five groups of convolutional downsampling layers include a first convolutional downsampling layer, a second convolutional downsampling layer, a third convolutional downsampling layer, a fourth convolutional downsampling layer, and a fifth convolutional downsampling layer;

[0105] The four cross-stage partial network modules include a first cross-stage partial network module, a second cross-stage partial network module, a third cross-stage partial network module and a fourth cross-stage partial network module;

[0106] The first convolution downsampling layer is used to receive the preprocessed image data, the output of the first convolution downsampling layer is connected to the input of the second convolution downsampling layer, the output of the second convolution downsampling layer is connected to the input of the first cross-stage partial network module, the output of the first cross-stage partial network module is connected to the input of the third convolution downsampling layer, the output of the third convolution downsampling layer is connected to the input of the second cross-stage partial network module, the output of the second cross-stage partial network module is connected to the input of the fourth convolution downsampling layer, the output of the fourth convolution downsampling layer is connected to the input of the third cross-stage partial network module, the output of the third cross-stage partial network module is connected to the input of the fifth convolution downsampling layer, the output of the fifth convolution downsampling layer is connected to the input of the fourth cross-stage partial network module, and the output of the fourth cross-stage partial network module is connected to the spatial pyramid pooling fusion module;

[0107] The second cross-stage partial network module, the third cross-stage partial network module and the spatial pyramid pooling fusion module respectively output shallow features, middle features and deep features to the multi-scale feature fusion module, and output multi-scale visual features.

[0108] The network input is a processed gas-liquid two-phase flow image, typically resized to 640×640 pixels with three color channels. Through the first convolutional layer, the input image is downsampled to 320×320, and the number of channels is expanded to 32. Subsequently, the network uses a series of convolutional layers and CSP modules to gradually reduce the feature map dimension to 20×20 while increasing the number of channels to 512, completing the process of extracting global semantic features from local texture features.

[0109] The network consists of five groups of convolutional downsampling layers, each using a 3×3 convolution kernel with a stride of 2. This halves the spatial dimensions of the feature map while increasing the number of channels. These layers gradually expand the receptive field while preserving important features, enabling the network to understand the characteristics of bubbles in a wider context. Each convolutional layer is followed by batch normalization and a ReLU activation function to enhance the network's nonlinear representation capabilities and training stability.

[0110] The cross-stage partial network module (CSPLayer) is the core component of this network. It employs a split-transform-aggregate strategy, splitting input features into two branches for processing, reducing computational redundancy and improving gradient flow efficiency. The first cross-stage partial network module contains multiple bottleneck structures and uses residual connections to enhance feature transfer, effectively alleviating the vanishing gradient problem in deep networks.

[0111] At different depths of the network, the structural parameters of the CSP module are adjusted: in the first cross-stage partial network module, fewer Bottleneck blocks (n=1) are used to focus on basic feature extraction; in the second and third cross-stage partial network modules, the number of Bottleneck blocks is increased (n=2 to n=3) to enhance feature transformation capabilities; in the fourth cross-stage partial network module, the complexity is reduced again (n=1) to focus on high-level semantic feature integration.

[0112] The Spatial Pyramid Pooling Fusion (SPPF) module, located at the end of the network, aggregates feature representations across different receptive fields through multi-scale max pooling. This module first compresses the input features through convolutional channels, then performs three consecutive max pooling operations to generate feature maps with four different receptive fields. Finally, these feature maps are concatenated and channel-wise integrated. This design enables the network to simultaneously focus on bubble features at different scales, enhancing the robustness of bubble detection in changing environments.

[0113] Multi-scale feature output mechanism

[0114] An important design of this network is to extract multi-scale features from three different depth positions:

[0115] ① Shallow features (F1): obtained from the output of CSPLayer2, retaining a high spatial resolution (about 1 / 8 of the original image), mainly including low-level visual features such as edges and contours, which help detect small-sized bubbles and accurately locate bubble boundaries.

[0116] ②Mid-level features (F2): obtained from the output of CSPLayer3, with medium spatial resolution (about 1 / 16 of the original image), integrating local texture and mid-level semantic information, and suitable for detecting medium-sized bubbles.

[0117] ③ Deep features (F3): obtained from the output of the SPPF module, with low spatial resolution (about 1 / 32 of the original image) but rich semantic information. They are mainly used to detect large bubbles and understand complex scene structures.

[0118] The characteristic information at these three scales together constitutes a complete spectrum of bubble visual representation, laying the foundation for subsequent fusion with environmental physical parameters and accurate bubble detection.

[0119] The pre-processed environmental physical parameters are encoded to generate high-dimensional physical features, and the method is as follows:

[0120] The environmental physical parameters are arranged in a set order to form an input vector, which is then input into a pre-built multi-layer perceptron to output high-dimensional physical features; the multi-layer perceptron includes an input layer, a first fully connected layer, a first normalization layer, an activation function, a Dropout layer, a second fully connected layer, and an output layer connected in sequence.

[0121] like Figure 3 As shown, in this embodiment, the environmental physical parameter feature extraction step uses a multi-layer perceptron (MLP) network to encode the pre-processed temperature, pressure, flow rate, and other optional physical data to generate a high-dimensional physical feature representation for subsequent multimodal fusion. At the same time, through the nonlinear transformation of the neural network, the complex interrelationships between environmental physical parameters are captured to create a more expressive feature representation. The implementation method is as follows:

[0122] (1) Input vector construction

[0123] The pre-processed environmental physical parameters (such as temperature T, pressure P, flow rate V, etc.) are used to form the input vector X in a predetermined order.

[0124]

[0125] Among them, each parameter is standardized to eliminate dimension and value differences and improve the overall stability of the model.

[0126] (2) Multi-layer perceptron structure design

[0127] An MLP network consisting of multiple fully connected layers is constructed. Each layer sequentially performs linear transformation and nonlinear activation operations to extract potential correlations and high-order features within the physical data. The specific process can be expressed as:

[0128]

[0129] Among them, W i with b i are the weight matrix and bias term of the i-th layer, σ(·) represents the nonlinear activation function (such as ReLU), H i is the intermediate feature of the i-th layer, and finally outputs the high-dimensional physical feature vector F phys .

[0130] The specific structure includes:

[0131] Input layer: Receives a d-dimensional normalized environmental physical parameter vector (X), where d represents the number of environmental physical parameters (such as temperature, pressure, flow rate, etc.).

[0132] The first fully connected layer maps the d-dimensional input parameters to a 128-dimensional hidden feature space to increase the expressive power of the features.

[0133] The first normalization layer: normalizes the 128-dimensional features to accelerate training convergence and improve stability.

[0134] ReLU activation function: introduces nonlinear transformation to enhance the expressive power of the network.

[0135] Dropout layer: randomly discards 10% of neurons to prevent model overfitting and improve generalization ability.

[0136] The second fully connected layer further maps the 128-dimensional features to 256 dimensions to further enhance the expressive power of the features.

[0137] Second layer normalization and ReLU activation: feature normalization and nonlinear transformation are performed again.

[0138] Output layer: The last fully connected layer maps the 256-dimensional features to the m-dimensional output feature space to generate the final physical feature vector. Usually m is set to 256 or 512 to match the dimension of the visual features.

[0139] This progressively expanded dimensionality design enables environmental physical parameters to more fully express the impact of environmental conditions on bubble morphology in a high-dimensional feature space. The encoded physical feature vectors not only directly contribute to the subsequent feature fusion process but are also used to generate dynamic fusion weights, guiding the model to adaptively adjust the relative importance of visual and physical features under varying environmental conditions. This design enables the model to flexibly adjust its feature fusion strategy based on changes in the physical environment, relying more on physical prior knowledge when environmental conditions change significantly and placing greater emphasis on visual features under standard conditions, thereby achieving highly adaptable bubble detection in diverse environments.

[0140] By extracting the features of environmental physical parameters, this method establishes a mapping relationship between environmental physical conditions and bubble visual representation, laying the foundation for subsequent dynamic weighted feature fusion and physical constraint evaluation. It effectively solves the problem that existing pure visual bubble detection methods are sensitive to changes in environmental conditions, and improves the applicability and reliability of the model in actual industrial environments.

[0141] In this embodiment, the environmental physical parameters are input into a dynamic weighted feature fusion module, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map. The method is as follows:

[0142] Input the high-dimensional physical feature vector into the fully connected layer to generate initial weights, and perform normalization to obtain visual feature weights and physical feature weights; the sum of the visual feature weights and the physical feature weights is 1;

[0143] Reshape the physical feature vector into a single-pixel multi-channel feature map and expand it to the same spatial size as the visual feature map through interpolation;

[0144] The multi-scale visual features are adjusted according to the weights and then spliced ​​with the spatialized physical features of the environment. The number of channels is compressed through convolution operations to generate the final comprehensive feature map.

[0145] Dynamic weighted feature fusion is the core innovation of this physically constrained multimodal deep learning bubble detection method. It achieves adaptive integration of visual and physical features, significantly improving the model's adaptability to changing environments. This mechanism, through three key steps: attention weight generation, physical feature spatialization, and weighted feature fusion, constructs a feature fusion framework that flexibly responds to environmental changes.

[0146] Attention weight generation primarily involves mapping the input physical feature vector into a modal weight vector through a dedicated attention weight network. The weights are then normalized using the Softmax function to ensure their sum is 1. The advantage of this attention mechanism is that it automatically adjusts the model's dependency strategy based on varying operating conditions, rather than employing a fixed fusion strategy. This allows the model to better adapt to diverse environmental conditions, improving detection robustness and accuracy.

[0147] In order to ensure that the physical features and image features have the same spatial resolution and channel dimension when fused, the physical features must be spatialized, that is, the low-dimensional physical feature vector (usually a one-dimensional vector) is converted into a high-dimensional feature map (usually a two-dimensional or three-dimensional tensor) that matches the visual features. This gives the physical features that originally had no spatial dimension a spatial structure so that they can interact with the visual features with spatial characteristics. The physical feature vector is reshaped into the initial form of the spatial feature map, that is, converted from a one-dimensional vector to a small-sized feature map with channel dimensions (initially a 1×1 spatial size). Then, this small-sized feature map is expanded to the same spatial resolution as the visual features through an upsampling operation.

[0148] The principle of weighted feature fusion is to dynamically adjust the importance weights of visual and physical features based on the characteristics of the current environmental conditions. The two features are then fused according to their respective weights to obtain a fused feature that integrates information from both modalities. First, the weights are applied to the visual features and the spatialized physical features, respectively. The weighted features are then concatenated in the channel dimension. Finally, a 1×1 convolution operation is used to fuse the concatenated features into a unified feature representation.

[0149] The fused features not only retain the spatial structure and edge information of the visual features but also incorporate environmental conditions represented by physical parameters, enabling the model to accurately identify and characterize bubble morphology under varying temperature, pressure, and flow rate conditions. This dynamic weighting mechanism significantly enhances the model's robustness to changing environmental conditions, reduces reliance on large amounts of annotated data, improves the physical rationality and reliability of the detection results, and provides more accurate and adaptable technical support for gas-liquid two-phase flow analysis.

[0150] Inputting the comprehensive feature map into a detection head network, outputting a bubble detection bounding box, extracting bubble parameters of the bubble based on the bubble detection bounding box, and converting them into actual physical dimensions based on a calibrated scale factor; the bubble parameters include at least size, shape, and position parameters;

[0151] The detection head network consists of three convolutional layers connected in sequence. The first two layers use a three-by-three convolution kernel. The number of input and output channels is the same and maintains a preset number. Each layer performs edge padding.

[0152] The third layer uses a single-channel convolution kernel, and the number of output channels is set according to the bounding box coordinates, detection confidence, and the total number of bubble categories;

[0153] After each convolution layer, batch normalization and nonlinear activation function operations are performed in sequence;

[0154] The network output is decoded into bounding box coordinates, confidence, and category information. Based on the non-maximum suppression algorithm, the bubble detection results are filtered by pre-setting the intersection-over-union overlap threshold and confidence threshold to ensure that a single bubble corresponds to a unique detection box.

[0155] In this embodiment, bubble detection and parameter extraction are key steps in the implementation of this physically constrained multimodal deep learning method. It uses fused features to accurately locate and characterize bubbles in gas-liquid two-phase flows. This process primarily involves designing a detection head network and extracting bubble physical parameters. Through a systematic process, the visual output of the deep learning model is converted into physically meaningful bubble characteristic parameters.

[0156] Design and implementation of detection head network

[0157] The present invention uses an improved YOLOv8 detection head network to detect bubble targets. The detection head receives the fused feature map F fused As input, a series of convolution operations are performed to output the bubble location, size, and confidence information. The structure of the detection head network is optimized to adapt to the special morphological characteristics of bubbles in gas-liquid two-phase flow images.

[0158] The detection head network consists of three convolutional layers. The first two layers use 3×3 convolution kernels to extract local features, and the last layer uses a 1×1 convolution kernel to output the detection results. The specific structure is as follows: the first convolutional layer uses C_s input channels and C_s output channels, with a convolution kernel size of 3×3 and a padding of 1; the second convolutional layer maintains the same configuration; the third convolutional layer uses C_s input channels and outputs (4+1+num_classes) channels, corresponding to the bounding box coordinates (4), confidence (1), and bubble category (num_classes). Batch normalization and SiLU activation function are applied after each convolution layer to accelerate convergence and improve nonlinear expression capabilities.

[0159] After decoding, the output of the detection head forms preliminary bubble detection results. For each detected bubble, its bounding box coordinates [x1, y1, x2, y2], detection confidence score, and category class_id are recorded. To eliminate redundant detections, this paper uses the non-maximum suppression (NMS) algorithm, setting appropriate IoU thresholds (typically 0.5) and confidence thresholds (typically 0.25) for screening, ensuring that each bubble is detected only once.

[0160] The bubble parameters of the bubble are extracted based on the bubble detection bounding box, and converted into actual physical size based on the calibrated scale factor, and the method is as follows:

[0161] Calculating the pixel area of ​​the bubble bounding box based on the detected bubble bounding box;

[0162] Multiplying the pixel area by a pre-calibrated scale factor to convert it into an actual physical area;

[0163] Correct the perspective distortion of bubbles in the edge area of ​​the image according to the camera calibration parameters;

[0164] Calculating a ratio of the width to the height of the bubble's bounding box, and determining whether the bubble is horizontally flattened or vertically stretched based on the ratio;

[0165] According to the proportional relationship between the contour area and the area of ​​its minimum circumscribed circle, the degree to which the bubble is close to a perfect circle is evaluated based on the proportional relationship;

[0166] Performing ellipse fitting on the contour, and quantifying the flattening degree of the bubble shape by the length ratio of the major axis to the minor axis of the ellipse;

[0167] The bubble's horizontal flattening or vertical stretching state, the bubble's degree of closeness to a perfect circle, and the degree of flattening of the bubble shape are taken as bubble shape parameters;

[0168] Wherein, for the case where the bubble shape within the bounding box is irregular, adaptive threshold segmentation is used to extract the precise outline; the area of ​​the pixel region actually occupied is calculated based on the outline, and the proportional coefficient conversion is applied again.

[0169] In this embodiment, after obtaining accurate bubble detection results, a complete bubble parameter extraction method is further developed to obtain the key physical characteristics of the bubbles. These characteristics are not only used to characterize the bubbles themselves, but also provide a basis for subsequent physical constraint evaluation.

[0170] The present invention calculates the pixel area of ​​each bubble based on the detected bounding box and converts it to its actual physical size using a pre-calibrated scaling factor. The specific calculation method is as follows: First, the pixel area of ​​each bubble's bounding box is calculated. Then, the pixel area is converted to a physical area using a pixel-to-physical unit conversion factor, K. The conversion factor, K, is obtained by calibrating the camera on a calibration plate of known physical dimensions. The present invention also considers the impact of viewpoint distortion on area calculation and employs a distortion correction method based on camera intrinsic parameters to ensure high accuracy of bubble area calculations at different image locations.

[0171] For irregularly shaped bubbles, this method further improves area calculation accuracy by employing a contour fitting method. First, adaptive threshold segmentation is applied within the detection bounding box to extract the precise bubble outline. Then, the actual pixel area enclosed by the outline is calculated. Finally, the same conversion coefficient is applied to obtain the physical area.

[0172] Bubble shape parameter extraction:

[0173] Bubble shape is an important indicator of the force applied to the bubble in the flow field. First, the bubble's aspect ratio is calculated as a basic shape parameter: an aspect ratio greater than 1 indicates a horizontally flattened bubble, less than 1 indicates a vertically stretched bubble, and a ratio of 1 indicates a nearly circular bubble.

[0174] Secondly, for the bubbles extracted with precise contours, the circularity parameter is calculated: The circularity value ranges from 0 to 1, with values ​​closer to 1 indicating that the bubble shape is closer to a perfect circle.

[0175] Third, calculate the bubble's eccentricity: This is determined by fitting an ellipse to the bubble's outline and calculating the ratio of the ellipse's major and minor axes. An eccentricity closer to 0 indicates a more circular bubble shape, while an eccentricity closer to 1 indicates a flatter bubble shape.

[0176] Through this systematic bubble detection and parameter extraction process, this method transforms the visual output of the deep learning model into bubble characteristic parameters with clear physical meaning, providing accurate and comprehensive quantitative characterization data for gas-liquid two-phase flow dynamics research and industrial process optimization. These parameters are not only used directly for flow state assessment but also provide input for subsequent physical constraint assessment, ensuring that the detection results conform to objective physical laws.

[0177] like Figure 4 As shown, a physical constraint loss function is constructed based on the ideal gas state equation and the principle of fluid mechanics to correct the actual physical size of the bubble, and the network parameters of the detection head network are corrected through the constraint loss;

[0178] The core innovation of this invention lies in the introduction of a physical constraint evaluation and adjustment mechanism to ensure that bubble detection results conform to objective physical laws, significantly improving the model's detection reliability and generalization capabilities under changing environments. Based on fundamental principles of fluid mechanics, this mechanism establishes constraints across three dimensions: bubble size, shape, and spatial distribution, compensating for the limitations of purely visual methods.

[0179] Based on the ideal gas law (PV = nRT), constraints on the relationship between bubble size, temperature, and pressure are established. Under constant gas conditions, bubble volume is directly proportional to temperature and inversely proportional to pressure. By calculating the difference between the predicted bubble size distribution and theoretical expectations, a size constraint loss is constructed. The size constraint loss calculation process first obtains the relative size ratio of all detected bubbles and then compares it with the theoretical size ratio calculated based on the current temperature and pressure.

[0180] Based on the principles of fluid dynamics, a constraint is established regarding the relationship between bubble shape and flow velocity. Bubbles in a fluid exhibit different shapes at different flow velocities—flatter (increased aspect ratio) at high flow rates and more spherical at low flow rates. By comparing the predicted bubble shape with the theoretical shape, a shape constraint loss is constructed.

[0181] Considering the effects of gravity and buoyancy on the spatial distribution of bubbles, constraints on the vertical position distribution of bubbles are established. In static or low-velocity conditions, bubbles tend to float upward due to buoyancy; in high-velocity or downward-flowing conditions, the bubble distribution may change.

[0182] To comprehensively consider the importance of each physical constraint, the present invention designs a comprehensive physical constraint loss function. This function forms a unified evaluation metric by weightedly combining the three physical constraint losses described above. The weight coefficients are used to balance the influence of different physical constraints and can be flexibly adjusted according to the specific application scenario. For example, in a high-pressure environment, the weight of the size constraint can be appropriately increased; in a high-flow environment, the weight of the shape constraint can be increased. This flexible weighting mechanism enables the model to adapt to the specific needs of different working conditions.

[0183] The physical constraints of this invention are used not only to guide model learning during the training phase but also to evaluate and adjust results during the inference phase. During inference, the system calculates a physical consistency score for each test result as a key indicator of result reliability. This score is calculated based on a negative exponential function of each physical constraint value, with higher scores indicating greater conformance to physical laws. The system further multiplies the physical consistency score with the test confidence level to produce a comprehensive reliability score, providing users with a comprehensive assessment of the result quality.

[0184] like Figure 5 As shown, the present invention uses a three-stage training strategy to ensure stable model convergence. The first stage is pure visual training, the main purpose of which is to establish basic visual recognition capabilities. Only gas-liquid two-phase flow images and their bubble annotation data are used, and the standard YOLOv8 model architecture is used. The training objective is to optimize the standard detection loss function. At this stage, physical parameters and physical constraints have not yet been introduced. The goal is to allow the model to first learn basic bubble visual feature recognition capabilities. The second stage is feature fusion training. The training data is image data and supporting physical parameters. The training network architecture is the YOLOv8 backbone, physical parameter encoder, and dynamic weighted feature fusion module. The training objective is still to optimize the standard detection loss function. The key to this stage is training the dynamic weighted feature fusion module, which enables the model to adaptively adjust the weights of visual features and physical features according to different physical conditions. The third stage is physical constraint training, which introduces physical constraints to ensure that the detection results conform to the laws of fluid mechanics. The network architecture is a complete model, and the optimization objective is the overall loss function. The key to this stage is to balance the relationship between visual detection accuracy and physical constraints. By gradually increasing the weight of the physical constraints, it is ensured that the model can both accurately detect bubbles and conform to basic physical laws.

[0185] Through the above-mentioned physical constraint evaluation and adjustment mechanism, the present invention realizes the organic combination of deep learning and physical knowledge, significantly improves the accuracy, robustness and physical consistency of bubble detection in changing environments, and provides more reliable technical support for gas-liquid two-phase flow analysis.

[0186] In this embodiment, the dynamic weight or small sample fine-tuning mechanism is adjusted to adapt to environmental changes. The method is as follows:

[0187] Determine whether changes in environmental conditions exceed a preset threshold;

[0188] If it does not exceed the limit, the weight of the physical features in the dynamic weighted fusion module is automatically increased according to the degree of deviation of the environmental parameters; the weight ratio of each constraint item in the physical constraint loss function is adjusted to adapt it to the current environmental characteristics; the confidence threshold and non-maximum suppression threshold are adaptively adjusted;

[0189] If it exceeds, the parameters of the target detection network and the detection head network are kept fixed, and only the parameters of the dynamic weighted fusion module are updated; fine-tuning is performed using a learning rate lower than that in the basic training stage; and model adaptation is completed within a limited number of training rounds based on a small amount of new environment annotated data.

[0190] In this embodiment, the environmental adaptation mechanism is the key innovation of the present invention to cope with changing working conditions. In response to the challenge of frequently changing environmental conditions in industrial production, two complementary adaptation strategies are designed.

[0191] The first is a configuration adjustment strategy. When environmental conditions change slightly to moderately, the system employs a configuration adjustment strategy, adapting to the new environment by modifying internal parameters without retraining the model. The specific strategy includes the following aspects: First, the system automatically adjusts the feature weights in the dynamic weighted fusion module based on the degree of environmental change. When environmental conditions deviate from the training data distribution, the system tends to increase the weight of physical features and reduce reliance on visual features, thereby improving the system's adaptability to environmental changes. Second, the system adjusts physical constraint parameters to better align with the characteristics of the new environment. Third, the system adjusts post-detection processing parameters, such as the non-maximum suppression threshold and confidence threshold, to accommodate changes in bubble morphology under the new environment. The advantages of the configuration adjustment strategy lie in its low implementation cost and fast response, making it suitable for dynamically changing industrial environments. The system can automatically perform adjustments based on historical experience and also supports manual intervention by experts to address special circumstances.

[0192] The second mechanism is a rapid fine-tuning. When environmental conditions change moderately to severely, and configuration adjustments cannot meet performance requirements, the system employs a rapid fine-tuning mechanism, updating the model with a small amount of new environmental data to quickly adapt to the new environment. The specific mechanism includes: First, the system employs a parameter-selective fine-tuning strategy, keeping the backbone network and detection head fixed and fine-tuning only the parameters of the fusion module. This strategy significantly reduces the number of parameters that need to be updated (typically less than 10% of all parameters), significantly reducing the amount of data and computing resources required for fine-tuning. Second, the system employs a low learning rate for fine-tuning, typically set at 1 / 20-1 / 10 of the baseline training learning rate (e.g., 0.00005). This low learning rate ensures that the fine-tuning process does not destroy the model's already learned basic detection capabilities while allowing the model to adapt to new environmental characteristics. Third, the system employs short-cycle fine-tuning, typically completing adaptation in just 5-10 training epochs. This enables the system to complete updates within minutes, minimizing the impact of the adaptation process on production operations. Most importantly, the system possesses small-sample learning capabilities, requiring only 5-10 annotated images of the new environment for effective fine-tuning. This greatly reduces the data annotation cost for environmental adaptation and makes the system more practical in actual industrial environments.

[0193] Output the corrected actual physical size and physical consistency score, and adapt to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms:

[0194] A consistency score of the calculation result is calculated based on each physical constraint value, and the consistency score is multiplied by the detection confidence to generate a comprehensive reliability score. Comprehensive reliability scores below a threshold are screened for manual review or automatic correction.

[0195] The reliability assessment of the results is an important guarantee mechanism for the output link of the present invention, which provides users with quantitative indicators of the quality of the test results. The system has designed a three-tier evaluation system: the first is the detection confidence, which is directly output by the model and reflects the model's confidence in the existence of bubbles; the second is the physical consistency score, which is obtained by calculating the degree to which the test results meet various physical constraints. Specifically, it is a negative exponential function of the sum of the loss values ​​of each physical constraint. When the test results fully comply with the physical laws, the score is close to 1, otherwise it is close to 0; the last is the comprehensive reliability score, which is obtained by multiplying the detection confidence and the physical consistency score, which comprehensively reflects the quality of the test results. The system will assign a reliability score to each detected bubble, and can filter high-reliability results according to the threshold set by the user. If necessary, it can also mark low-reliability areas as areas for further confirmation to remind users to pay attention.

[0196] In practical applications, the result output and environmental adaptation mechanism of the present invention can cope with a variety of complex scenarios. For example, in chemical reaction process monitoring, when the reaction temperature rises from room temperature to high temperature, the system maintains detection accuracy by increasing the physical feature weights and adjusting the size constraint parameters; in multiphase fluid delivery systems, when the flow rate suddenly changes, the system quickly adapts to the new flow pattern by quickly fine-tuning the fusion module; in the material preparation process, when the addition of new surfactants changes the bubble interface characteristics, the system ensures the consistency of the detection results by injecting new physical constraint parameters.

[0197] Through the organic combination of the above mechanisms, the present invention achieves highly adaptable bubble detection in changing environments, significantly improves the practical value of the system in actual industrial applications, and provides strong technical support for gas-liquid two-phase flow analysis and related process optimization.

[0198] See also Figure 6 , Figure 6 This is a comparison of the bubble detection results before and after the algorithm improvement. The left picture is before the improvement, and the confidence of the identified bubbles is low. The right picture shows the confidence after the improvement.

[0199] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0200] A second embodiment of the present invention provides a bubble detection system based on multimodal input and an improved target detection network. This system is based on the bubble detection method based on multimodal input and an improved target detection network of the first embodiment. The system includes:

[0201] a parameter acquisition module configured to synchronously collect image data of the gas-liquid two-phase flow and environmental physical parameters and perform preprocessing operations, wherein the environmental physical parameters include at least temperature, pressure and flow rate;

[0202] A multi-scale visual feature construction module is configured to extract multi-scale visual features from the pre-processed image data based on a pre-built and trained object detection network, and encode the pre-processed environmental physical parameters to generate high-dimensional physical features; wherein the object detection network is constructed based on multiple groups of convolutional downsampling layers, multiple cross-stage partial network modules, and a spatial pyramid pooling fusion module, and extracts multi-scale visual features from shallow, middle, and deep layers respectively;

[0203] a comprehensive feature map generation module configured to input the environmental physical parameters into a dynamic weighted feature fusion module, output dynamic weights, and perform weighted fusion of multi-scale visual features and high-dimensional physical features based on the dynamic weights to generate a comprehensive feature map;

[0204] a bubble parameter calculation module configured to input the comprehensive feature map into a detection head network, output a bubble detection bounding box, extract bubble parameters of the bubble based on the bubble detection bounding box, and convert them into actual physical dimensions based on a calibrated scale factor; the bubble parameters include at least size, shape, and position parameters;

[0205] A constraint and correction module is configured to construct a physical constraint loss function based on the ideal gas state equation and fluid mechanics principles, correct the actual physical size of the bubble, and correct the network parameters of the detection head network through the constraint loss;

[0206] The environmental adaptation module is configured to output the corrected actual physical size and physical consistency score, and adapt to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms.

[0207] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0208] It should be noted that the bubble detection system based on multimodal input and improved target detection network provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps and are not regarded as improper limitations on the present invention.

[0209] An electronic device according to a third embodiment of the present invention includes:

[0210] at least one processor; and

[0211] a memory communicatively connected to at least one of the processors; wherein,

[0212] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned bubble detection method based on multimodal input and improved target detection network.

[0213] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are used to be executed by the computer to implement the above-mentioned bubble detection method based on multimodal input and improved target detection network.

[0214] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0216] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0217] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0218] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A bubble detection method based on multimodal input and improved target detection network, characterized in that: The method includes: Synchronously collecting image data of the gas-liquid two-phase flow and environmental physical parameters and performing preprocessing operations, wherein the environmental physical parameters include at least temperature, pressure and flow rate; Extracting multi-scale visual features of the pre-processed image data based on a pre-built and trained object detection network, and encoding the pre-processed environmental physical parameters to generate high-dimensional physical features; Based on the physical parameters of the environment, the dynamic weighted feature fusion module is input, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map; Inputting the comprehensive feature map into a detection head network, outputting a bubble detection bounding box, extracting bubble parameters of the bubble based on the bubble detection bounding box, and converting them into actual physical dimensions based on a calibrated scale factor; the bubble parameters include at least size, shape, and position parameters; A physical constraint loss function is constructed based on the ideal gas state equation and fluid mechanics principles to correct the actual physical size of the bubble, and the network parameters of the detection head network are corrected through the constraint loss. Output the corrected actual physical size and physical consistency score, and adapt to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms.

2. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: The pre-processing operation includes: The image data is subjected to denoising, contrast enhancement, background removal and normalization processing, and the environmental physical parameters are subjected to smoothing processing, outlier removal and standardization.

3. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: The target detection network specifically includes: Five groups of convolutional downsampling layers, four cross-stage partial network modules, and a spatial pyramid pooling fusion module; The five groups of convolutional downsampling layers include a first convolutional downsampling layer, a second convolutional downsampling layer, a third convolutional downsampling layer, a fourth convolutional downsampling layer, and a fifth convolutional downsampling layer; The four cross-stage partial network modules include a first cross-stage partial network module, a second cross-stage partial network module, a third cross-stage partial network module and a fourth cross-stage partial network module; The first convolution downsampling layer is used to receive the preprocessed image data, the output of the first convolution downsampling layer is connected to the input of the second convolution downsampling layer, the output of the second convolution downsampling layer is connected to the input of the first cross-stage partial network module, the output of the first cross-stage partial network module is connected to the input of the third convolution downsampling layer, the output of the third convolution downsampling layer is connected to the input of the second cross-stage partial network module, the output of the second cross-stage partial network module is connected to the input of the fourth convolution downsampling layer, the output of the fourth convolution downsampling layer is connected to the input of the third cross-stage partial network module, the output of the third cross-stage partial network module is connected to the input of the fifth convolution downsampling layer, the output of the fifth convolution downsampling layer is connected to the input of the fourth cross-stage partial network module, and the output of the fourth cross-stage partial network module is connected to the spatial pyramid pooling fusion module; The second cross-stage partial network module, the third cross-stage partial network module and the spatial pyramid pooling fusion module respectively output shallow features, middle features and deep features to the multi-scale feature fusion module, and output multi-scale visual features.

4. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: The pre-processed environmental physical parameters are encoded to generate high-dimensional physical features, and the method is as follows: The environmental physical parameters are arranged in a set order to form an input vector, which is input into a pre-built multi-layer perceptron to output high-dimensional physical features; the multi-layer perceptron includes an input layer, a first fully connected layer, a first normalization layer, a first activation function, a Dropout layer, a second fully connected layer, a second normalization layer, a second activation function and an output layer connected in sequence.

5. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: Based on the environmental physical parameters, the dynamic weighted feature fusion module is input, dynamic weights are output, and multi-scale visual features and high-dimensional physical features are weightedly fused based on the dynamic weights to generate a comprehensive feature map. The method is as follows: Input the high-dimensional physical feature vector into the fully connected layer to generate initial weights, and perform normalization to obtain visual feature weights and physical feature weights; the sum of the visual feature weights and the physical feature weights is 1; Reshape the physical feature vector into a single-pixel multi-channel feature map and expand it to the same spatial size as the visual feature map through interpolation; The multi-scale visual features are adjusted according to the weights and then spliced ​​with the spatialized physical features of the environment. The number of channels is compressed through convolution operations to generate the final comprehensive feature map.

6. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: The detection head network consists of three convolutional layers connected in sequence. The first two layers use a three-by-three convolution kernel. The number of input and output channels is the same and maintains a preset number. Each layer performs edge padding. The third layer uses a single-channel convolution kernel, and the number of output channels is set according to the bounding box coordinates, detection confidence, and the total number of bubble categories; After each convolution layer, batch normalization and nonlinear activation function operations are performed in sequence; The network output is decoded into bounding box coordinates, confidence, and category information. Based on the non-maximum suppression algorithm, the bubble detection results are filtered by pre-setting the intersection-over-union overlap threshold and confidence threshold to ensure that a single bubble corresponds to a unique detection box.

7. The bubble detection method based on multimodal input and improved target detection network according to claim 1, characterized in that: The bubble parameters of the bubble are extracted based on the bubble detection bounding box, and converted into actual physical size based on the calibrated scale factor, and the method is as follows: Calculating the pixel area of ​​the bubble bounding box based on the detected bubble bounding box; Multiplying the pixel area by a pre-calibrated scale factor to convert it into an actual physical area; Correct the perspective distortion of bubbles in the edge area of ​​the image according to the camera calibration parameters; Calculating a ratio of the width to the height of the bubble's bounding box, and determining whether the bubble is horizontally flattened or vertically stretched based on the ratio; According to the proportional relationship between the contour area and the area of ​​its minimum circumscribed circle, the degree to which the bubble is close to a perfect circle is evaluated based on the proportional relationship; Performing ellipse fitting on the contour, and quantifying the flattening degree of the bubble shape by the length ratio of the major axis to the minor axis of the ellipse; The bubble's horizontal flattening or vertical stretching state, the bubble's degree of closeness to a perfect circle, and the degree of flattening of the bubble shape are taken as bubble shape parameters; Wherein, for the case where the bubble shape within the bounding box is irregular, adaptive threshold segmentation is used to extract the precise outline; the area of ​​the pixel region actually occupied is calculated based on the outline, and the proportional coefficient conversion is applied again.

8. The bubble detection method based on multimodal input and improved target detection network according to claim 7, characterized in that: According to the ideal gas state equation and the principle of fluid mechanics, a physical constraint loss function is constructed to correct the actual physical size of the bubble. The method is as follows: Based on the ideal gas law, the correlation constraints between bubble volume, temperature and pressure are established, and the difference between the detected volume and the theoretical volume is calculated as the size constraint loss; Based on the principles of fluid dynamics, a mapping relationship between the bubble aspect ratio and flow velocity is established, and the difference between the shape prediction value and the theoretical value is calculated as the shape constraint loss; Based on the balance between gravity and buoyancy, a statistical model of bubble vertical position distribution and flow velocity is constructed, and the distribution deviation is calculated as the distribution constraint loss. Combining size, shape, and distribution constraint losses into a comprehensive physical constraint loss function according to preset weights; wherein the preset weights are dynamically adjusted according to environmental parameters; The model parameters of the detection head network are modified based on the calculation results of the comprehensive physical constraint loss function.

9. The bubble detection method based on multimodal input and improved target detection network according to claim 8, characterized in that: Adapting to environmental changes by adjusting dynamic weights or small sample fine-tuning mechanisms is done as follows: Determine whether changes in environmental conditions exceed a preset threshold; If it does not exceed the limit, the weight of the physical features in the dynamic weighted fusion module is automatically increased according to the degree of deviation of the environmental parameters; the weight ratio of each constraint item in the physical constraint loss function is adjusted to adapt it to the current environmental characteristics; Adaptively adjust confidence threshold and non-maximum suppression threshold; If it exceeds, the parameters of the target detection network and the detection head network are kept fixed, and only the parameters of the dynamic weighted fusion module are updated; Fine-tune the model using a lower learning rate than that used in the basic training phase. Adapt the model within a limited number of training rounds based on a small amount of labeled data from a new environment.

10. The bubble detection method based on multimodal input and improved target detection network according to claim 9, characterized in that: A consistency score of the calculation result is calculated based on each physical constraint value, the consistency score is multiplied by the detection confidence to generate a comprehensive reliability score, and comprehensive reliability scores below a threshold are screened for manual review or automatic correction.

Images