Laboratory reagent bottle visual identification method and system based on multi-modal perception data fusion

Abstract

The invention discloses a laboratory reagent bottle visual identification method and system based on multi-modal perception data fusion. The visual identification method comprises the steps of designing and constructing a multi-modal sensing module on a mechanical arm for operating reagent bottles, fusing multi-modal data, performing reagent bottle type identification and label information extraction by adopting a deep learning visual identification module based on a convolutional neural network and an image segmentation technology, and the like. The data of various sensors are aligned and synchronously processed through time and space coordinates to form complete environment sensing data at the same time point; in the data fusion process, the measurement errors of the various sensors are corrected by adopting a method of combining Kalman filtering and particle filtering. The deep learning visual identification module adopts a multilayer convolutional network model introducing a multi-scale attention mechanism. According to the invention, by integrating various sensor data and adopting a deep learning algorithm, accurate recognition of the reagent bottle is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of laboratory intelligent management, and more specifically, to a laboratory reagent bottle visual recognition method and system based on multimodal perception data fusion. Background Art

[0002] In modern laboratories, the automated management of reagents faces challenges such as difficult identification, low management efficiency, and unintelligent path planning. Traditional reagent management systems mostly rely on manual operation or semi-automated equipment, and mainly rely on single-modal visual recognition technology. They have poor adaptability to complex environments such as lighting changes, reagent bottle occlusion and stacking, resulting in low recognition accuracy. In addition, due to subtle differences in reagent bottle labels, similar colors, and diverse shapes, conventional methods are difficult to achieve efficient recognition in complex environments, which can easily lead to misidentification and misoperation. These problems make manual intervention and adjustment an indispensable part of experimental operations, greatly reducing overall management efficiency.

[0003] Although deep learning technology has made significant progress in image recognition, existing visual recognition methods are still susceptible to environmental noise interference and lack robustness when dealing with highly similar or fuzzy-labeled reagent bottles. Most existing systems rely only on single visual information and cannot integrate additional data provided by multimodal sensors (such as depth, torque, temperature and humidity), resulting in limited recognition results and poor practicality and adaptability of the system.

[0004] Therefore, developing a visual recognition method and system for laboratory reagent bottles with higher recognition accuracy is a technical problem that has not yet been overcome. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention proposes a laboratory reagent bottle visual recognition method based on multimodal perception data fusion, which realizes accurate recognition of reagent bottles by integrating multiple sensor data and adopting deep learning algorithms.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a laboratory reagent bottle visual recognition method based on multimodal perception data fusion, comprising:

[0007] Step S1, designing and constructing a multimodal perception module on a robotic arm that operates a reagent bottle: the multimodal perception module includes a variety of sensors including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor, which are used to collect multidimensional data of the laboratory environment and the status of the reagent bottle;

[0008] Step S2, multimodal data fusion: The data of the multiple sensors are aligned and synchronously processed through time and space coordinates to form complete environmental perception data at the same time point; during the data fusion process, the measurement errors of the multiple sensors are corrected by combining Kalman filtering with particle filtering to improve the accuracy of reagent bottle positioning and identification. The multimodal data fusion method combining Kalman filtering with particle filtering is used to reduce the measurement errors between sensors and effectively improve the accuracy and robustness of environmental perception.

[0009] Step S3: using a deep learning visual recognition module based on convolutional neural network and image segmentation technology to identify the type of reagent bottle and extract label information:

[0010] The deep learning visual recognition module adopts a multi-layer convolutional network model that introduces an attention mechanism; the attention mechanism can focus on important areas in the image (such as label information and bottle body features), thereby improving the ability to extract key features. The training process of the multi-layer convolutional network model includes the following sub-steps:

[0011] Step S31, collecting and annotating a large number of image data sets of reagent bottles, including images of reagent bottles in different lighting, angles, and shapes;

[0012] Step S32: preprocessing the image data, including image cropping, scaling, rotation and image enhancement, to enhance the robustness of the model;

[0013] Step S33: Combine the three-dimensional position information provided by the depth camera to jointly train the two-dimensional image data and the depth data to improve the recognition performance of the model in complex environments.

[0014] Furthermore, in step S2, the complete environmental perception data after data fusion is uploaded to the laboratory management central controller; the laboratory management central controller generates management instructions (such as robot arm pick-up and placement path planning, storage environment adjustment, alarm prompts, etc.) according to the complete environmental perception data information, and sends them to the corresponding execution module.

[0015] Furthermore, in step S2, the linear sensor data of the torque sensor and the temperature and humidity sensor are fused using a Kalman filter algorithm to eliminate measurement noise and improve data accuracy; the nonlinear visual data of the RGB camera and the depth camera are fused using a particle filter algorithm to process three-dimensional position information in a complex environment.

[0016] Furthermore, in step S2, the linear sensor data of the torque sensor and the temperature and humidity sensor are firstly denoised and predicted and updated using the Kalman filter algorithm; the state of the nonlinear visual data is then predicted and estimated using the particle filter, and the weight of the particle is updated according to the current observation value; finally, the information from different sensors is integrated into the state estimation at the same time by the weighted average method to form the complete state information of the reagent bottle. The combined application of Kalman filtering and particle filtering forms an efficient and accurate multimodal data fusion method.

[0017] Furthermore, in step S2, according to the measurement accuracy and reliability of each sensor in different scenarios, the weight factors of linear filtering and nonlinear filtering are dynamically adjusted during weighted averaging. This adaptive dynamic adjustment mechanism can further ensure accurate perception of the status of the reagent bottle.

[0018] Furthermore, in step S2, while integrating the information from different sensors into the state estimation at the same time, the residual analysis of each sensor data is performed to determine whether the data is abnormal and make corresponding weight adjustments. This mechanism can effectively prevent the overall perception failure caused by abnormal data from a single sensor.

[0019] Furthermore, in step S3, the deep learning visual recognition module includes three parts: an encoder, a decoder and a multi-scale attention mechanism; the encoder extracts high-level semantic features of the input image based on the EfficientNetB7 network structure; EfficientNetB7 adopts a compound scaling strategy and optimizes depth, width and resolution at the same time; the decoder includes a deconvolution module and a jump connection, and the jump connection directly transfers the feature map of the corresponding level in the encoder to the decoder for fusing the high-resolution feature information in the encoder; the multi-scale attention mechanism uses a channel attention mechanism and a spatial attention mechanism for feature weighting to enhance the ability of the deep learning visual recognition module to capture important areas and subtle features.

[0020] Furthermore, the EfficientNetB7 network is composed of several stacked MBConv modules; each MBConv module includes depth convolution, point-by-point convolution and SE modules.

[0021] Furthermore, the channel attention mechanism and the spatial attention mechanism are combined and applied to different stages of the encoder and decoder:

[0022] Encoder part: The channel attention mechanism is embedded in each MBConv module of the EfficientNetB7 network to perform weighted screening on the feature channels extracted at each layer;

[0023] Decoder: After the upsampling and deconvolution operations of the decoder, a spatial attention mechanism is introduced to weight the features in the spatial dimension.

[0024] In the decoder feature fusion process, multi-scale convolution and the multi-scale attention mechanism are introduced to extract multi-scale feature information through different convolution kernel sizes.

[0025] In a second aspect, the present invention provides a laboratory reagent bottle visual recognition system based on multimodal perception data fusion, which is used to implement the laboratory reagent bottle visual recognition method as described above, comprising:

[0026] The multimodal perception module is built on the robotic arm that operates the reagent bottle. It includes a variety of sensors, including RGB camera, depth camera, torque sensor, and temperature and humidity sensor, to collect multidimensional data of the laboratory environment and reagent bottle status.

[0027] The deep learning visual recognition module uses a multi-layer convolutional network model that introduces an attention mechanism to identify the type of reagent bottles and extract label information; combined with the three-dimensional position information provided by the depth camera, the two-dimensional image data and the depth data are jointly trained to train the deep learning visual recognition module;

[0028] The data fusion module adopts a multimodal data fusion method that combines Kalman filtering and particle filtering to reduce the measurement error between sensors; it fuses data from different sensors and deep learning visual recognition to generate high-dimensional comprehensive environmental information.

[0029] Compared with the prior art, the present invention has the following technical effects:

[0030] 1. Multimodal perception data fusion: The present invention adopts a multimodal perception module, integrates multidimensional data collected by multiple sensors (such as RGB cameras, depth cameras, torque sensors, and temperature and humidity sensors), and uses a combination of Kalman filtering and particle filtering to perform data fusion. Compared with traditional single-modal recognition methods, the present invention can better eliminate the measurement errors between different sensors, thereby improving the positioning accuracy and recognition stability of reagent bottles in complex environments, especially in the case of light changes, partial occlusion, and reagent bottle stacking, showing higher robustness and recognition rate.

[0031] 2. Deep learning visual recognition and multi-sensor information fusion: The present invention introduces convolutional neural network (CNN) and attention mechanism in the deep learning visual recognition module, and jointly processes visual recognition with depth sensor data. Compared with the traditional single visual recognition system, the present invention can use the three-dimensional information provided by the depth camera to effectively improve the recognition accuracy of reagent bottle label information and morphological features, avoid recognition confusion caused by similar or occluded reagent bottle shapes, and thus improve the overall recognition effect of the system.

[0032] 3. Efficient recognition and low error rate: The present invention introduces an attention mechanism and a multi-layer convolutional network structure in the visual recognition module, and combines the three-dimensional information of the depth sensor to effectively improve the accuracy and stability of reagent bottle recognition. Compared with traditional recognition systems, the present invention can effectively reduce the error rate in complex environments, greatly improving the recognition ability and application scope of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flow chart of a method for visually identifying laboratory reagent bottles in one embodiment of the present invention.

[0034] Figure 2 The figure is a flowchart of the execution of multimodal perception and data fusion in one embodiment of the present invention.

[0035] Figure 3 Flowchart of a multimodal data fusion algorithm in one embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the EfficientNetB7 network structure in one embodiment of the present invention.

[0037] Figure 5 Schematic diagram of the structure of a laboratory reagent bottle visual recognition system in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0039] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms are not shown in detail to avoid obscuring the subject matter of the present invention.

[0040] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0041] Example 1

[0042] See also Figure 1 This embodiment provides a laboratory reagent bottle visual recognition method based on multimodal perception data fusion, comprising the following steps:

[0043] Step S1, design and construct a multimodal perception module on the robotic arm that operates the reagent bottle: the multimodal perception module includes multiple sensors including RGB camera, depth camera, torque sensor and temperature and humidity sensor, which are used to collect multidimensional data of laboratory environment and reagent bottle status. Figure 2 , this embodiment uses a multimodal perception module to perform comprehensive and integrated perception of the laboratory environment and the status of reagent bottles. The module includes an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor. The RGB camera is used to capture the color, label information, and surface morphology of the reagent bottle, and generate two-dimensional image data; the depth camera is used to collect the depth information and relative distance of the reagent bottle in three-dimensional space; the torque sensor is used to monitor the contact force of the robot arm when grasping and moving the reagent bottle to ensure the safety of the operation process; the temperature and humidity sensor monitors the temperature and humidity changes in the laboratory environment in real time to ensure the environmental stability of the reagent during storage.

[0044] Step S2, multimodal data fusion: the data of the multiple sensors are aligned and synchronously processed through time and space coordinates to form complete environmental perception data at the same time point; during the data fusion process, the measurement errors of the multiple sensors are corrected by combining Kalman filtering and particle filtering.

[0045] This step generates high-dimensional comprehensive environmental information by fusing data from different sensors. Different sensors have different data characteristics. For example, the image data provided by the RGB camera is two-dimensional information, the depth camera provides three-dimensional spatial coordinates, and the torque sensor and temperature and humidity sensor output one-dimensional torque and environmental parameters respectively. Therefore, how to effectively fuse these heterogeneous data to form complete environmental perception data is the key technology of the multimodal perception module. This embodiment adopts a multimodal data fusion method that combines Kalman filtering with particle filtering to reduce the measurement error between sensors and effectively improve the accuracy and robustness of environmental perception.

[0046] Kalman filtering is a recursive optimal state estimation method commonly used in linear systems and is suitable for processing linear sensor data with Gaussian noise. This embodiment uses Kalman filtering to fuse the data of the torque sensor and the temperature and humidity sensor to eliminate measurement noise and improve data accuracy. Specifically, Kalman filtering is divided into two stages: prediction stage and update stage.

[0047] Prediction stage: Based on the state at the previous moment and the current control input, predict the state and state covariance matrix at the current moment:

[0048]

[0049]

[0050] in Indicates time The predicted status. is the state transfer matrix, which indicates the system from the previous moment To the current time The state transition relationship. B is the control matrix, which represents the control input Impact on system status. is the covariance matrix of the predicted state. is the process noise covariance matrix.

[0051] Update phase: When new sensor measurements are When it arrives, the state is updated according to the predicted state and the observed value:

[0052]

[0053]

[0054]

[0055] in: is the Kalman gain, which represents the weight distribution between the sensor measurement value and the predicted state, is the observation matrix, which represents the mapping relationship from state space to observation space. is the measurement noise covariance matrix. is the updated optimal state estimate. Kalman filtering can effectively reduce noise and make prediction corrections to linear sensor data, thereby improving the measurement accuracy of torque sensors and temperature and humidity sensors.

[0056] Particle filtering is a nonlinear state estimation technology based on the Monte Carlo method, which is suitable for processing complex data with nonlinear and non-Gaussian distribution. This embodiment uses particle filtering to fuse the visual data of the RGB camera and the depth camera to process the three-dimensional position information in a complex environment.

[0057] The particle filter represents the probability distribution of the state through a set of discrete particles and their weights, and resamples them according to the weight of each particle. The weight update steps of the particle are as follows:

[0058] Particle state prediction: Assume that the particle state at the previous moment is , predict the particle state at the current moment:

[0059]

[0060] Where f is the state transfer function, is the process noise.

[0061] Particle weight update: based on current observation value and the particle's predicted state , update the particle weight:

[0062]

[0063] in is the likelihood probability between the particle state and the observed value, which can be estimated by Gaussian distribution.

[0064] Particle resampling: Resample the particle set according to the particle weight distribution, retain the particles with higher weights, and remove the particles with lower weights to avoid particle degradation. State estimation: The final state estimation value is the weighted average of all particle states:

[0065]

[0066] in is the particle weight, The state of the particle.

[0067] By fusing visual data with particle filtering, it is possible to maintain a high-precision estimate of the position of reagent bottles in complex environments (such as changes in the position of reagent bottles, occlusion, stacking, etc.), thereby improving the robustness and recognition capabilities of the system in complex environments.

[0068] Kalman filtering and particle filtering are used together in the present invention to form an efficient and accurate multimodal data fusion method. The specific steps are as follows:

[0069] Linear data processing: Use Kalman filtering to reduce noise and predict and update linear sensor data (such as torque sensors and temperature and humidity sensors) to improve the accuracy and stability of measurement data.

[0070] Nonlinear data processing: Use particle filtering to predict and estimate the state of nonlinear sensor data (such as RGB cameras and depth cameras), and update the weights of particles according to the current observation values ​​to ensure the fusion accuracy of visual information and 3D depth information.

[0071] Data fusion and joint estimation: The linear data processed by Kalman filter and the nonlinear data processed by particle filter are jointly fused. By weighted averaging, the information from different sensors is integrated into the state estimation at the same time, thus forming the complete state information of the reagent bottle. Assume that the linear state estimation is , the nonlinear state is estimated as , then the final state is estimated to be:

[0072]

[0073] in, It is the weight factor for linear and nonlinear data fusion, which is dynamically adjusted according to the complexity of the laboratory environment and the noise level of the sensor to ensure that the fused state estimation can maintain high robustness in various complex environments.

[0074] Adaptive dynamic adjustment mechanism: This embodiment preferably designs an adaptive dynamic adjustment mechanism to dynamically adjust the fusion ratio of linear filtering and nonlinear filtering according to the measurement accuracy and reliability of each sensor in different scenarios. For example, when the ambient lighting conditions are good and the label is not blocked, the system will increase the weight of visual recognition data; when the lighting conditions are poor or the reagent bottle is blocked, the weight of the torque and depth information will be increased to ensure accurate perception of the reagent bottle status.

[0075] Data anomaly detection and correction: In the data fusion process, this embodiment preferably introduces a data anomaly detection and correction mechanism. The specific approach is to perform residual analysis on the measurement values ​​output by each sensor and the fused state estimate. If the measurement residual of a sensor exceeds the set threshold, the sensor output is considered abnormal. At this time, the system will reduce the weight of the sensor data until its output returns to normal. This mechanism can effectively prevent overall perception failure caused by abnormal data from a single sensor.

[0076] The overall process of the multimodal data fusion algorithm is as follows: Figure 3 As shown, the specific steps are as follows:

[0077] Data collection and preprocessing: Each sensor collects environmental data in real time and performs preprocessing, including data denoising, scale alignment, and time synchronization.

[0078] State prediction and observation update: Kalman filter processing is performed on the torque sensor and temperature and humidity sensor data to predict the current environmental state. Particle filter processing is performed on the RGB camera and depth camera data to generate the position and attitude estimation of the reagent bottle.

[0079] Data fusion and anomaly detection: Combine the sensor data to generate multi-modal state estimates. At the same time, perform residual analysis on the sensor data to determine whether the data is abnormal and make corresponding weight adjustments.

[0080] State output: Preferably, the fused state estimate is transmitted to the central control module, which generates management instructions (such as robot pick-up and placement path planning, storage environment adjustment, alarm prompts, etc.) based on the state information and sends them to the corresponding execution module.

[0081] Through the above steps, the multimodal data fusion module of the present invention can achieve high-precision identification, positioning and environmental status perception of reagent bottles in a complex laboratory environment, provide reliable data support for reagent management and operation, and significantly improve the perception accuracy and intelligence level of the system.

[0082] Step S3: using a deep learning visual recognition module based on convolutional neural network and image segmentation technology to identify the type of reagent bottle and extract label information:

[0083] The deep learning visual recognition module adopts a multi-layer convolutional network model that introduces an attention mechanism; the training process of the multi-layer convolutional network model includes the following sub-steps:

[0084] Step S31, collecting and annotating a large number of image data sets of reagent bottles, including images of reagent bottles in different lighting, angles, and shapes;

[0085] Step S32, preprocessing the image data;

[0086] Step S33: Combine the three-dimensional position information provided by the depth camera to jointly train the two-dimensional image data and the depth data.

[0087] More specifically, this embodiment uses a neural network structure based on EfficientNetB7 in the deep learning visual recognition module to identify and segment the type, label information and status of the reagent bottle. The module includes three main parts: encoder, decoder and attention mechanism. The specific design and implementation will be described in detail below.

[0088] The encoder part is based on the EfficientNetB7 network structure and is responsible for extracting high-level semantic features of the input image. EfficientNetB7 adopts a compound scaling strategy and optimizes depth, width, and resolution at the same time, which can achieve higher feature extraction capabilities than traditional networks with fewer parameters and computational complexity.

[0089] like Figure 4 As shown in the figure, EfficientNetB7 is composed of several MBConv modules (Mobile Inverted BottleneckConvolution) stacked together. Each MBConv module contains depthwise convolution, pointwise convolution, and SE module (Squeeze-and-Excitation). The following is the specific process:

[0090] Depthwise Convolution:

[0091] Deep convolution greatly reduces the number of model parameters by processing the features of each channel independently. This operation is expressed as follows:

[0092]

[0093] in, Indicates The output feature map of the channels is is the first input feature map channels, is the convolution kernel.

[0094] Pointwise Convolution:

[0095] Point-wise convolution uses The convolution kernel maps the features of each channel to the new channel space and integrates the information between channels. The formula for point-by-point convolution is as follows:

[0096]

[0097] in, is the output feature map after point-by-point convolution, is the convolution kernel for each channel.

[0098] SE module (Squeeze-and-Excitation):

[0099] The SE module improves the network's attention to important features by modeling global information for each channel. The specific process is as follows: First, global average pooling is performed to obtain the global information of each channel:

[0100]

[0101] in, For the The global information of each channel, and are the height and width of the feature map.

[0102] Generate channel weights through two fully connected layers and ReLU activation function:

[0103]

[0104] in, and is the learnable parameter matrix, is the ReLU activation function, is the Sigmoid activation function, is the channel weight. Each channel is weighted to generate a weighted feature map:

[0105]

[0106] The SE module can increase the network's attention to important features and effectively screen multi-level features. After stacking several MBConv modules, EfficientNetB7 gradually converts the input image into multiple high-dimensional feature maps and performs weighted processing on each channel through the SE module. Finally, the encoder outputs a highly compressed feature map with a significantly increased number of channels and a reduced spatial resolution of 1 / 32 of the original image.

[0107] The decoder part is as follows:

[0108] 1. Transposed Convolution:

[0109] Deconvolution is used to further refine the upsampled feature map and gradually restore it to the same resolution as the input image. Deconvolution learns a convolution kernel weight and performs a reverse convolution operation on the input features, so that richer edge details can be maintained when generating feature maps. The calculation formula for deconvolution is as follows:

[0110]

[0111] in, is the output feature map, is the input feature map, is the deconvolution kernel. The deconvolution kernel size is usually set to , the step size is 2 to ensure that each deconvolution operation can double the spatial size of the feature map.

[0112] 2. Skip Connections:

[0113] The skip connection directly transfers the feature map of the corresponding level in the encoder to the decoder to fuse the high-resolution feature information in the encoder. This connection method can effectively retain the detailed features of the input image and avoid information loss caused by multiple convolution and pooling operations. The calculation formula of the skip connection is:

[0114]

[0115] in, is the upsampled feature map in the decoder, is the feature map of the corresponding level in the encoder, and Concat represents the channel dimension concatenation of the feature map. The feature map fused by skip connection has richer spatial and semantic information, which is helpful for the generation of subsequent segmentation maps.

[0116] The decoder finally outputs a segmentation map of the same size as the input image, where each pixel in the segmentation map represents the category it belongs to (such as reagent bottle label, background, etc.). The decoder part achieves fine-grained reconstruction of the encoder feature map through multiple upsampling, deconvolution, and jump connections.

[0117] Introduction of multi-scale attention mechanism:

[0118] This embodiment introduces multiple attention mechanisms in each layer of the encoder and decoder to enhance the model's ability to capture important areas and subtle features. Channel Attention and Spatial Attention are mainly used for feature weighting.

[0119] 1. Channel Attention Mechanism:

[0120] The channel attention mechanism weights the features of each channel so that the model can focus on the feature channels that are more important for the segmentation task. The calculation process is as follows:

[0121] Global average pooling: Perform global average pooling on the feature map of each channel to generate a global feature vector:

[0122]

[0123] in, Indicates The global features of the channels, and are the height and width of the feature map.

[0124] Fully connected layer and activation function: Generate channel weights through two fully connected layers (FC) and ReLU activation function:

[0125]

[0126] in, and is the learnable fully connected layer weight matrix, is the ReLU activation function, The Sigmoid activation function is used to limit the weights to the range [0,1].

[0127] Channel weighting: The weight of each channel Multiply the original feature map element by element to achieve feature weighting:

[0128]

[0129] Through the channel attention mechanism, the model can effectively screen on different feature channels, so as to better capture the label information, morphological characteristics and other details of the reagent bottle.

[0130] 2. Spatial Attention Mechanism:

[0131] The spatial attention mechanism weights each spatial position of the feature map, allowing the model to focus on important areas. The specific process is as follows:

[0132] Channel fusion: First, the feature map is fused in the channel dimension (such as global average pooling or maximum pooling) to generate a two-dimensional spatial feature map.

[0133]

[0134] Among them, the Pool operation is used to perform global averaging or maximum pooling in the channel dimension. Indicates location The characteristic value of .

[0135] The convolution operation generates spatial weights: using a The convolution kernel convolves the fused spatial feature map to generate spatial weights

[0136]

[0137] in, is the convolution kernel weight, is the Sigmoid activation function, which is used to constrain the spatial weights to the range [0,1].

[0138] Spatial weighting: Multiply the spatial weight by the original feature map element by element to achieve weighted operation in the spatial dimension:

[0139]

[0140] By introducing the spatial attention mechanism, the model can focus more on key areas in the image (such as the reagent bottle label area) during the feature extraction process, thereby improving the segmentation accuracy.

[0141] Combination and application of hybrid attention mechanism:

[0142] In the visual recognition module of this embodiment, channel attention and spatial attention are combined and applied to different stages of the encoder and decoder:

[0143] Encoder part: The channel attention mechanism is embedded in each MBConv module of the EfficientNetB7 network to perform weighted screening on the feature channels extracted at each layer, ensuring that the model can focus on features that are more important for the segmentation task.

[0144] Decoder part: After the upsampling and deconvolution operations of the decoder, the spatial attention mechanism is introduced to improve the model's ability to focus on the target area by weighting the features in the spatial dimension.

[0145] Multi-scale attention mechanism: In the decoder feature fusion process, multi-scale convolution and multi-scale attention mechanism are introduced. The multi-scale feature information extracted by different convolution kernel sizes can further improve the segmentation effect of the model in complex scenes (such as reagent bottle occlusion and morphological similarity).

[0146] Through the above design, the deep learning visual recognition module of the present invention can realize efficient recognition and segmentation of reagent bottles in complex environments. The feature extraction capability of EfficientNetB7, the fine-grained reconstruction capability of the decoder, and the introduction of multiple attention mechanisms have significantly improved the recognition accuracy and robustness of the model, and can cope with reagent bottle recognition and management tasks in a variety of complex scenarios.

[0147] Example 2

[0148] See also Figure 5 This embodiment provides a laboratory reagent bottle visual recognition system based on multimodal perception data fusion, which is used to implement the laboratory reagent bottle visual recognition method as described in Example 1, including:

[0149] The multimodal sensing module 100 is built on the robotic arm that operates the reagent bottle and includes a variety of sensors including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor, which are used to collect multidimensional data of the laboratory environment and the status of the reagent bottle;

[0150] The deep learning visual recognition module 200 uses a multi-layer convolutional network model that introduces an attention mechanism to identify the type of reagent bottles and extract label information; combined with the three-dimensional position information provided by the depth camera, the two-dimensional image data and the depth data are jointly trained to train the deep learning visual recognition module;

[0151] The data fusion module 300 adopts a multimodal data fusion method combining Kalman filtering and particle filtering to reduce the measurement error between sensors; it fuses data from different sensors and deep learning visual recognition to generate high-dimensional comprehensive environmental information.

[0152] As a further technical solution, the comprehensive environmental information generated by the data fusion module 300 is output to the laboratory management central control unit. The laboratory management central control unit includes a reagent status tracking and intelligent management module. This module is a functional module closely integrated with the multimodal perception module and the deep learning visual recognition module, and is used to realize status monitoring, environmental monitoring, dynamic layout optimization and intelligent management of reagent bottles; by real-time tracking and recording of information such as the frequency of use, validity period, storage location and environmental conditions of reagent bottles, an intelligent reagent management solution is provided for the laboratory.

[0153] More specifically, first, the multimodal perception module collects various status information of reagent bottles, such as the reagent bottle ID and storage location recorded by the RFID tag, the environmental conditions fed back by the temperature and humidity sensor, and the reagent usage measured by the weight sensor. All collected data will be integrated through the data fusion module, and a status record of each reagent bottle will be generated, including storage location, remaining quantity, frequency of use and expiration date. The reagent status tracking and intelligent management module updates the reagent management database in real time based on these status records, and combines with the deep learning visual recognition module to realize automatic recognition and verification of reagent bottle label information, ensuring that the status data of all reagents is accurate and complete.

[0154] Secondly, based on the reagent status data, the reagent status tracking and intelligent management module can dynamically adjust the storage position of different reagent bottles on the reagent rack according to the frequency of use and expiration date. Move frequently used or about to expire reagent bottles to the front row for easy access, and move infrequently used reagent bottles or spare reagents to the back row or infrequently used areas, thereby improving the overall efficiency of the robot's access operations. This layout optimization strategy based on reinforcement learning can significantly reduce the ineffective movements of the robot and improve the overall efficiency of laboratory management.

[0155] In addition, the reagent status tracking and intelligent management module can also introduce environmental monitoring and anomaly detection functions. The temperature and humidity sensor monitors the temperature and humidity in the reagent rack environment in real time, and compares the data with the ideal storage conditions of the reagent bottles. When the storage environment of a reagent bottle does not meet the standards, the system will automatically trigger an alarm and record the abnormal information in the reagent management database. At the same time, it can be recommended to transfer the affected reagent bottles to other areas with more suitable environments based on the current environmental status to avoid reagent failure or deterioration due to poor storage conditions. In terms of frequency of use and expiration date, by dynamically tracking the usage of each reagent bottle, intelligently marking the reagent bottles that are about to expire, and giving priority to the experimenters to use them, to ensure that the reagents are used efficiently. In addition, for reagent bottles that are used less or stored for too long, the reagent status tracking and intelligent management module will generate low usage reminders and provide corresponding optimization suggestions, such as regular inspections or transfers to spare areas.

[0156] Through the collaborative work of the reagent status tracking and intelligent management module with the multimodal perception module and the deep learning visual recognition module, the present invention can also realize all-round intelligent reagent management, from real-time monitoring of reagents, environmental control, layout optimization to inventory management, which comprehensively improves the efficiency and accuracy of laboratory reagent management, reduces resource waste and safety hazards caused by poor management, and provides laboratories with efficient, intelligent and accurate management solutions.

[0157] The above-mentioned laboratory reagent bottle visual identification method can be embodied in the form of a computer program product or a software functional unit. If the above-mentioned laboratory reagent bottle visual identification method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the essence of the technical solution or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic system (which can be a personal computer, a server, or a network system, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0158] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0159] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0160] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the systems and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A laboratory reagent bottle visual recognition method based on multimodal perception data fusion, characterized in that: include: Step S1, designing and constructing a multimodal perception module on a robotic arm that operates a reagent bottle: the multimodal perception module includes a variety of sensors including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor, which are used to collect multidimensional data of the laboratory environment and the status of the reagent bottle; Step S2, multimodal data fusion: the data of the multiple sensors are aligned and synchronously processed through time and space coordinates to form complete environmental perception data at the same time point; During the data fusion process, a method combining Kalman filtering and particle filtering is used to correct the measurement errors of the multiple sensors; Step S3: using a deep learning visual recognition module based on convolutional neural network and image segmentation technology to identify the type of reagent bottle and extract label information: The deep learning visual recognition module adopts a multi-layer convolutional network model that introduces an attention mechanism; the training process of the multi-layer convolutional network model includes the following sub-steps: Step S31, collecting and annotating a large number of image data sets of reagent bottles, including images of reagent bottles in different lighting, angles, and shapes; Step S32, preprocessing the image data; Step S33: Combine the three-dimensional position information provided by the depth camera to jointly train the two-dimensional image data and the depth data.

2. The method for visual recognition of laboratory reagent bottles based on multimodal perception data fusion according to claim 1 is characterized in that: The complete environmental perception data and visual recognition data after data fusion are uploaded to the laboratory management central controller; the laboratory management central controller generates management instructions based on the complete environmental perception data information and sends them to the corresponding execution modules.

3. The laboratory reagent bottle visual recognition method based on multimodal perception data fusion according to claim 1 or 2, characterized in that: In step S2, the linear sensor data of the torque sensor and the temperature and humidity sensor are fused using the Kalman filter algorithm; the nonlinear visual data of the RGB camera and the depth camera are fused using the particle filter algorithm to process the three-dimensional position information in a complex environment.

4. The method for visual recognition of laboratory reagent bottles based on multimodal perception data fusion according to claim 3 is characterized in that: In step S2, the Kalman filter algorithm is first used to reduce noise and predict the linear sensor data of the torque sensor and the temperature and humidity sensor; then the particle filter is used to predict and estimate the state of the nonlinear visual data, and the weight of the particle is updated according to the current observation value; finally, the information from different sensors is integrated into the state estimation at the same time through the weighted average method to form the complete state information of the reagent bottle.

5. The method for visual recognition of laboratory reagent bottles based on multimodal perception data fusion according to claim 4 is characterized in that: In step S2, according to the measurement accuracy and reliability of each sensor in different scenarios, the weight factors of linear filtering and nonlinear filtering are dynamically adjusted during weighted averaging.

6. The laboratory reagent bottle visual recognition method based on multimodal perception data fusion according to claim 5 is characterized in that: In step S2, while integrating the information from different sensors into the state estimation at the same time, residual analysis is performed on the data of each sensor to determine whether the data is abnormal and to make corresponding weight adjustments.

7. The laboratory reagent bottle visual recognition method based on multimodal perception data fusion according to claim 1 or 2, characterized in that: In step S3, the deep learning visual recognition module includes three parts: an encoder, a decoder and a multi-scale attention mechanism; the encoder extracts high-level semantic features of the input image based on the EfficientNetB7 network structure; EfficientNetB7 adopts a compound scaling strategy and optimizes depth, width and resolution at the same time; the decoder includes a deconvolution module and a jump connection, and the jump connection directly transfers the feature map of the corresponding level in the encoder to the decoder for fusing the high-resolution feature information in the encoder; The multi-scale attention mechanism uses a channel attention mechanism and a spatial attention mechanism for feature weighting.

8. The laboratory reagent bottle visual recognition method based on multimodal perception data fusion according to claim 7 is characterized in that: The EfficientNetB7 network is composed of several stacked MBConv modules; each MBConv module contains depth convolution, point-by-point convolution and SE modules.

9. The laboratory reagent bottle visual recognition method based on multimodal perception data fusion according to claim 8 is characterized in that: The channel attention mechanism and spatial attention mechanism are combined and applied in different stages of the encoder and decoder: Encoder part: The channel attention mechanism is embedded in each MBConv module of the EfficientNetB7 network to perform weighted screening on the feature channels extracted at each layer; Decoder: After the upsampling and deconvolution operations of the decoder, a spatial attention mechanism is introduced to weight the features in the spatial dimension. In the decoder feature fusion process, multi-scale convolution and the multi-scale attention mechanism are introduced to extract multi-scale feature information through different convolution kernel sizes.

10. A laboratory reagent bottle visual recognition system based on multimodal perception data fusion, characterized in that: A method for visually identifying a laboratory reagent bottle according to any one of claims 1 to 9, comprising: The multimodal perception module is built on the robotic arm that operates the reagent bottle. It includes a variety of sensors, including RGB camera, depth camera, torque sensor, and temperature and humidity sensor, to collect multidimensional data of the laboratory environment and reagent bottle status. The deep learning visual recognition module uses a multi-layer convolutional network model that introduces an attention mechanism to identify the type of reagent bottles and extract label information; combined with the three-dimensional position information provided by the depth camera, the two-dimensional image data and the depth data are jointly trained to train the deep learning visual recognition module; The data fusion module adopts a multimodal data fusion method that combines Kalman filtering and particle filtering to reduce the measurement error between sensors; it fuses data from different sensors and deep learning visual recognition to generate high-dimensional comprehensive environmental information.

Images