AI-based pharmaceutical process density cross-modal measurement method and measurement and control robot system

Abstract

The invention belongs to the technical field of pharmaceutical process quality measurement and control, and provides an AI-based pharmaceutical process density cross-modal measurement method and a measurement and control robot system. Comprising the following steps: S1, collecting video data of a material surface in a multi-segment pharmaceutical process in real time; s2, performing framing processing on the video to obtain a plurality of image sets; s3, preprocessing each frame of image in the image set, and extracting density cross-modal image features to obtain feature vectors; s4, sorting and splicing the feature vectors to obtain a global feature matrix of the multiple video segments, and generating density label vectors; s5, performing standardization processing on the global feature matrix to obtain a standardized feature matrix; and S6, taking the standardized feature matrix and the corresponding density label vector as a data set, fitting a non-linear relationship between the image features and density measurement values, and constructing a humanoid visual depth multivariate correction model so as to measure the density of the pharmaceutical process in a cross-modal manner. According to the invention, non-contact dynamic monitoring of the density in the pharmaceutical process can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of pharmaceutical process quality measurement and control, and specifically relates to an AI-based pharmaceutical process density cross-modal measurement method and a measurement and control robot system. Background Art

[0002] In the pharmaceutical manufacturing process, real-time monitoring of material density is a key technology for ensuring product quality and batch-to-batch consistency. Traditional testing methods often rely on visual inspection based on manual experience or offline sensor detection equipment (such as handheld density testers, tuning fork concentration meters, differential pressure density meters, etc.), which have significant bottlenecks: a single manual measurement takes up to 5-10 minutes, resulting in a 2-3 hour delay in process parameter adjustment, which seriously slows down production; the experience of operators varies, which can easily lead to subjective judgment bias, causing the concentration endpoint determination result to deviate from the actual value. In addition, for high-viscosity and high-solids concentrates, the density meter is prone to clogging the sensor or sample flow channel due to drug solution adhesion, crystallization, or impurity deposition, resulting in measurement interruption or deviation. These combined problems of low efficiency, uncontrollable errors, and insufficient equipment reliability not only make it difficult to meet the rigid requirements of GMP regulations for real-time measurement and control and data traceability of pharmaceutical processes, but also create quality control blind spots in complex scenarios such as the concentration process of pastes and polysaccharide pharmaceutical products, highlighting the fundamental contradiction between traditional monitoring methods and the standardization needs of modern pharmaceuticals. Specifically, density detection in conventional pharmaceutical and chemical concentration processes mainly relies on contact sensing equipment, which measures physical parameters through direct contact with the liquid to infer density. This presents the following problems:

[0003] 1. The contact detection method is not ideal: the sensor needs to penetrate the drug solution, which is prone to introducing impurities and contamination, cannot meet aseptic production requirements, and is difficult to monitor dynamic changes in real time;

[0004] 2. The pharmaceutical process information perception method is single: it relies on a single physical parameter (such as temperature and pressure) and cannot simultaneously capture dynamic visual characteristics of the material surface (such as boiling state, foam morphology, and other key density image features);

[0005] 3. Response hysteresis: Poor real-time performance. Manual sampling is difficult to capture rapidly changing concentration states. The detection cycle is long (usually tens of minutes), making it difficult to track rapid changes in density.

[0006] 4. Unclear process quality change patterns: Conventional operations only test material density at the concentration endpoint, making it difficult to understand the process quality change patterns, which can easily lead to endpoint density deviation and make it impossible to accurately control the concentration endpoint;

[0007] 5. Energy waste: Failure to stop the energy supply equipment in the pharmaceutical process in a timely manner can easily lead to energy waste;

[0008] 6. Sensitive to environmental interference: Environmental factors such as steam and temperature can easily affect the accuracy of sensing equipment;

[0009] 7. Insufficient reliability of testing equipment: High-viscosity materials often contain solid particles or foam, which can easily clog the density meter's measurement channel or probe. Regular disassembly and cleaning, downtime for cleaning, or component replacement are required, increasing equipment maintenance costs and downtime, affecting production efficiency.

[0010] In summary, the existing difficulties in density measurement in pharmaceutical processes are:

[0011] 1. The gap between manual experience and measurement values: Manual experience in evaluating pharmaceutical process density is difficult to convert into quantitative indicators;

[0012] 2. Separation of visual features and physical parameters: Traditional methods lack the ability to collaboratively analyze cross-modal (visual / physical) surface features of materials;

[0013] 3. Insufficient dynamic process modeling: Manual sampling and single-point measurement cannot construct a continuous density change model;

[0014] 4. Lack of intelligent sensing technology: A non-contact cross-modal density measurement method based on deep learning and human-like vision has not yet been established.

[0015] These problems seriously restrict the precise measurement and control of pharmaceutical process density, and there is an urgent need for an innovative solution that integrates multimodal information, dynamic feature extraction and AI perception. Summary of the Invention

[0016] The present invention aims to address the technical problems existing in the prior art and provide an AI-based cross-modal density measurement method and measurement and control robot system for pharmaceutical processes that combines machine automation, computer vision, and multi-dimensional information fusion to achieve non-contact dynamic density monitoring and real-time feedback control of pharmaceutical processes.

[0017] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0018] An AI-based cross-modal density determination method for pharmaceutical processes includes the following steps:

[0019] S1, real-time acquisition of multiple segments of pharmaceutical process material surface video;

[0020] S2. Frame processing is performed on each collected video segment of the material surface of the pharmaceutical process to obtain multiple image sets of the multiple video segments;

[0021] S3, preprocessing and feature extraction are performed on each frame of the image in the plurality of image sets to obtain a feature vector of each frame of the image;

[0022] S4. Sort and concatenate the feature vectors of each frame of each video in chronological order to obtain a global feature matrix of multiple videos, and generate density label vectors of pharmaceutical process materials corresponding to different video segments;

[0023] S5, normalizing the global feature matrix obtained in step S4 to obtain a standardized feature matrix;

[0024] S6. Using the standardized feature matrix and the corresponding pharmaceutical process material density label vector as a training data set, fitting the nonlinear relationship between image features and pharmaceutical process material density, and constructing a cross-modal human-like vision depth multivariate calibration model that spans from image values ​​to density measurement values, thereby determining the pharmaceutical process material density;

[0025] Optionally, in step S1, the collected pharmaceutical process material surface video parameters are set to resolution W×H, frame rate F, W×H represents pixels, F represents the number of frames, and the output format is .mp4.

[0026] Optionally, in step S2, the frame interval time Δt is calculated according to the following formula:

[0027]

[0028] Where N represents the total number of frames in the video, and T represents the total duration of the collected video of the surface of a pharmaceutical process material.

[0029] According to the frame interval time Δt, the time point for frame division is determined using the following calculation formula:

[0030] t i =i×Δt(i=0,1,...,N-1),

[0031] Where i represents the i-th frame of the video;

[0032] Extract the video frames according to the time point of the frame division and obtain the framed image set {I1,I2,...,I N}, where {I1,I2,...,I N} indicates that the image set contains N frames of images, and I1 indicates the i-th frame of image.

[0033] Optionally, in step S3, the process of preprocessing each frame of the plurality of image sets specifically includes:

[0034] S301, assuming that a certain frame image is I i , image I i Scaling to pixels α×β, the formula is as follows:

[0035]

[0036] in, Represents the scaled image, Resized(·) represents the image scaling operation, and α×β represents the scaled image pixels;

[0037] S302: scaling the image Perform center cropping, the formula is as follows:

[0038]

[0039] in, represents the image after center cropping, CenterCrop(·) represents the image center cropping operation, γ×η represents the image pixels after center cropping, 0<γ<α, 0<η<β;

[0040] S303: Normalize the center-cropped image using the following formula:

[0041]

[0042] in, represents the normalized image, μ and σ represent the normal distribution parameters after normalization, (x, y, ) represents the pixel coordinates, ( , , c) represents the channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

[0043] Optionally, in step S3, the process of extracting features from the normalized image is specifically as follows:

[0044] The normalized image is input into the convolutional neural network, and features are extracted through multi-layer convolution operations. The output of the lth layer of convolution is expressed as:

[0045] Z l =ReLU(W l *Z l-1 +b l ),

[0046] Among them, W l represents the convolution kernel weight, Z l-1 represents the feature map of the l-1 layer, b l represents the bias of layer l;

[0047] After multiple layers of convolution operations, a feature map is obtained, which is compressed by global average pooling to generate the feature vector of the image:

[0048]

[0049] Among them, f i represents the feature vector of the image, (x',y') represents the pixel coordinates in the feature map, and ρ and υ both represent the size of the feature map.

[0050] Optionally, step S4 specifically includes:

[0051] Assume that the feature vector set of each frame image in a certain video image set is {f1,f2,...,f j}, sort the feature vector set in chronological order to obtain the temporal feature matrix F of the image set video as follows:

[0052]

[0053] Among them, R j×χ represents a matrix with j rows and x columns;

[0054] For a dataset containing M videos, the temporal feature matrices of the M videos are vertically concatenated to obtain the global feature matrix X of the M videos as follows:

[0055]

[0056] Among them, R (M×j)×χ represents a matrix of M×j rows and x columns;

[0057] The density label vector Y of the pharmaceutical process materials generated for different video segments is as follows:

[0058]

[0059] Among them, 1 j is a vector of all 1s, indicating that the j frames of each video share a pharmaceutical process material density label y k (k=1,2,...,M).

[0060] Optionally, in step S5, the process of normalizing the global feature matrix is ​​specifically as follows:

[0061] For the global feature matrix X containing M videos, normalize it by column. For the d-th column feature, calculate the mean μ of all samples d and standard deviation σ d , the calculation formula is as follows:

[0062]

[0063] Among them, F p,d Represents the d-th dimension eigenvalue of the p-th sample;

[0064] The standardized eigenvalues ​​are:

[0065]

[0066] Generate standardized feature matrix X norm ∈R(M×j)×χ .

[0067] Optionally, step S6 specifically includes:

[0068] Based on the standardized feature matrix X norm ∈R (M×j)×χ The LightGBM-based multivariate calibration model is trained with the pharmaceutical process material density label vector Y, and the nonlinear relationship between frame-level features and pharmaceutical process material density is fitted by the gradient boosting decision tree;

[0069] For the input feature z q , the output correction value is as follows:

[0070]

[0071] Where f(·) represents the multivariate calibration model established through training, Represents the density correction value of pharmaceutical process materials for a single frame image;

[0072] Collect the surface video of the material in the current pharmaceutical process and extract the feature matrix X of the θ frame image in the video test ∈R θ×ω , using the trained multivariate calibration model to estimate the θ pharmaceutical process material density calibration values ​​frame by frame The material density measurement result corresponding to the current pharmaceutical process material surface video is the average of the θ pharmaceutical process material density correction values, as follows:

[0073]

[0074] in, It represents the material density measurement result corresponding to the material surface video of the current pharmaceutical process, and τ represents the τth frame.

[0075] The present invention also provides an AI-based pharmaceutical process density cross-modal measurement and control robot system, which uses the aforementioned AI-based pharmaceutical process density cross-modal measurement method to measure and monitor pharmaceutical process density, including an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit, and a remote monitoring unit;

[0076] The industrial robot unit is used to carry the machine vision unit;

[0077] The machine vision unit is used to collect multiple segments of material surface videos during the pharmaceutical process in real time;

[0078] The edge computing unit is used to process multiple segments of pharmaceutical process material surface videos collected in real time by the machine vision unit, and continuously learn, calculate, and update the multivariate correction model for pharmaceutical process material density to measure the density of pharmaceutical process materials.

[0079] The feedback control unit is used to receive the pharmaceutical process material density signal output by the edge computing unit and output a control signal to the system;

[0080] The remote monitoring unit is used to analyze the sampled image data in real time and monitor the dynamic changes in the density of the pharmaceutical process materials. When the density of the pharmaceutical process materials reaches the preset judgment conditions, an alarm signal is issued to remind the operator to perform control intervention.

[0081] Optionally, the industrial robot unit is one of a fixed robot and a composite mobile robot;

[0082] The machine vision unit is an industrial camera installed on the industrial robot unit.

[0083] Compared with the prior art, the present invention has the following beneficial effects:

[0084] (1) Non-contact dynamic detection: The present invention realizes non-contact dynamic detection through an industrial robot unit and a machine vision unit. The industrial camera on the machine vision unit collects real-time surface video of the pharmaceutical process material, and combines the visual feature analysis of the edge computing unit to realize non-invasive density detection.

[0085] (2) Multi-dimensional information fusion: The present invention collects dynamic videos of the material surface during the pharmaceutical process and extracts the features of each frame in the video to comprehensively characterize the dynamics of the pharmaceutical process and realize dynamic and static multi-dimensional information fusion;

[0086] (3) Temporal feature retention: The present invention constructs a video frame-level feature matrix and retains the independence of the temporal sequence. Through the multi-dimensional feature extraction of video frames and the construction of the temporal feature matrix, a multivariate correction model based on LightGBM is constructed to fit the nonlinear mapping relationship between the video frame-level features and the density of pharmaceutical process materials, thereby realizing the density measurement based on the dynamic video of the pharmaceutical process material surface;

[0087] (4) Density measurement and real-time feedback: The present invention obtains image signals and extracts image features based on the currently collected surface video of pharmaceutical process materials. The image features are input into the multivariate correction model based on LightGBM, and the density measurement value of the pharmaceutical process materials is output. The data is transmitted in real time, and the control signal is output to the system through the feedback control unit. When the density measurement value of the pharmaceutical process material reaches the preset judgment condition, the energy supply can be stopped in time, thereby achieving the goal of accurately controlling the quality of the pharmaceutical process and saving energy and reducing consumption;

[0088] (5) High-precision detection: The present invention reduces the mean absolute error between the density measurement value of pharmaceutical process materials and the laboratory measurement value through multi-dimensional feature engineering of video frames, which is superior to traditional sensors. In addition, the detection is highly real-time and realizes automated feedback control based on intelligent judgment of concentration preset conditions, accurately controls the concentration endpoint, and promptly terminates the energy supply equipment of the concentration system, avoiding energy waste, reducing the deviation of the concentrate density between different batches, and improving the consistency of process quality.

[0089] (6) Strong real-time performance: The present invention has a short response time from video acquisition to density measurement, strong real-time performance, and supports online monitoring and control;

[0090] (7) Anti-interference capability: The ring-shaped LED fill light of the light source system on the machine vision unit is used to fill in the light, ensuring uniform illumination on the surface of the material. The standardized light source and video noise reduction technology effectively eliminate ambient light fluctuations and steam interference;

[0091] (8) Simplified process: The present invention eliminates the problems of easy clogging and contamination risks of contact sensing detection equipment, avoids tedious operations such as sensor cleaning, realizes automated density detection, and avoids the tedious process of manual density detection operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is a flow chart of a cross-modal density determination method for a pharmaceutical process based on AI according to an embodiment of the present invention;

[0093] Figure 2 This is a technical roadmap for an AI-based cross-modal density measurement method for pharmaceutical processes according to an embodiment of the present invention;

[0094] Figure 3 This is a workflow diagram of the AI-based cross-modal measurement and control robot system for pharmaceutical process density according to an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0096] Example 1

[0097] Combine Figure 1-Figure 2 As shown, the present invention provides an AI-based cross-modal density determination method for pharmaceutical processes, comprising the following steps:

[0098] S1, real-time acquisition of multiple segments of pharmaceutical process material surface video;

[0099] S2. Frame processing is performed on each collected video segment of the material surface of the pharmaceutical process to obtain multiple image sets of the multiple video segments;

[0100] S3, preprocessing and feature extraction are performed on each frame of the image in the plurality of image sets to obtain a feature vector of each frame of the image;

[0101] S4. Sort and concatenate the feature vectors of each frame of each video in chronological order to obtain a global feature matrix of multiple videos, and generate density label vectors of pharmaceutical process materials corresponding to different video segments;

[0102] S5, normalizing the global feature matrix obtained in step S4 to obtain a standardized feature matrix;

[0103] S6. Use the standardized feature matrix and the corresponding pharmaceutical process material density label vector as the training data set, fit the nonlinear relationship between image features and pharmaceutical process material density, and construct a cross-modal human-like visual depth multivariate correction model that spans from image values ​​to density measurement values, so as to determine the pharmaceutical process material density.

[0104] Example 2

[0105] Combine Figure 1-Figure 2 As shown, based on Example 1, in this embodiment, in step S1, the collected video parameters of the pharmaceutical process liquid surface are set to resolution W×H, frame rate F, and output format is .mp4; the pharmaceutical process is specifically a pharmaceutical chemical concentration process;

[0106] Specifically, an industrial camera is used to capture real-time dynamic video of the liquid surface during the pharmaceutical and chemical concentration process. The camera is positioned directly above the transparent observation window of the liquid concentration tank, with the lens focal length covering the boiling area. Video parameters are set to a resolution of W×H, a frame rate of F, and an .mp4 output format. A lighting system is incorporated into the camera, using a ring-shaped LED fill light to ensure uniform illumination of the liquid surface and avoid reflections. This non-contact video capture completely eliminates the risk of contamination caused by contact between traditional sensors and the liquid.

[0107] Furthermore, in step S2, assuming that the total duration of the collected video of the surface of the drug liquid in a pharmaceutical chemical concentration process is T (.mp4 format, resolution W×H, frame rate F, total duration T seconds, W×H represents pixels, F represents the number of frames), the frame interval time Δt is calculated using the following formula:

[0108]

[0109] Where N represents the total number of frames in the video;

[0110] According to the frame interval time Δt, the time point for framing is determined. The calculation formula is as follows:

[0111] t i =i×Δt(i=0,1,...,N-1),

[0112] Where i represents the i-th frame of the video;

[0113] Extract the video frames according to the time point of the frame division and obtain the framed image set {I1,I2,...,I N}, where {I1,I2,...,I N} means that the image set contains N frames of images, and I1 means the i-th frame of image;

[0114] Specifically, assuming N=30, determine each time point t for framing i =i×Δt(i=0,1,...,N-1), extract video frames at this interval, and obtain the framed image set {I1,I2,...,I 30 This step ensures that the dynamic changes of the concentration process are uniformly captured from the video.

[0115] Furthermore, in step S3, the process of preprocessing each frame of the plurality of image sets specifically includes:

[0116] S301, assuming that a certain frame image is I i , image I i Scaling to pixels α×β, the formula is as follows:

[0117]

[0118] in, Represents the scaled image, Resized(·) represents the image scaling operation, and α×β represents the scaled image pixels;

[0119] Specifically, the pixel α×β can be 256×256, that is,

[0120] S302: scaling the image Perform center cropping, the formula is as follows:

[0121]

[0122] in, represents the image after center cropping, CenterCrop(·) represents the image center cropping operation, γ×η represents the image pixels after center cropping, 0<γ<α, 0<η<β;

[0123] Specifically, the pixels after center cropping can be 224×224, that is,

[0124] S303: Normalize the center-cropped image using the following formula:

[0125]

[0126] in, represents the normalized image, μ and σ represent the normal distribution parameters after normalization, (x, y, ) represents the pixel coordinates, ( , , c) represents the channel, R represents the red channel, G represents the green channel, and B represents the blue channel;

[0127] At this time, we can calculate μ = [0.485, 0.456, 0.406], σ = [0.229, 0.224, 0.225]; the processed image The pixel values ​​obey the standard normal distribution, which is convenient for subsequent feature extraction.

[0128] Furthermore, the process of feature extraction for the standardized image is as follows:

[0129] The normalized image is input into the convolutional neural network, and features are extracted through multi-layer convolution operations. The output of the lth layer of convolution is expressed as:

[0130] Z l =ReLU(W l *Z l-1 +b l ),

[0131] Among them, W l represents the convolution kernel weight, Z l-1 represents the feature map of the l-1 layer, b l represents the bias of layer l;

[0132] After multiple layers of convolution operations, a feature map is obtained, which is compressed by global average pooling to generate the feature vector of the image:

[0133]

[0134] Among them, f i represents the feature vector of the image, (x', y') represents the pixel coordinates in the feature map, and ρ and υ both represent the size of the feature map;

[0135] Specifically, the size of the feature map is 7×7×512, that is, the feature map is compressed through the global average pooling (GAP) operation to generate a feature vector of the image with a length of 512, which represents the spatial global features of the image:

[0136]

[0137] Example 3

[0138] Combine Figure 1-Figure 2 As shown, based on Example 2, in this embodiment, step S4 specifically includes:

[0139] Assume that the feature vector set of each frame image in a certain video image set is {f1,f2,...,f j}, sort the feature vector set in chronological order to obtain the temporal feature matrix F of the image set video as follows:

[0140]

[0141] Among them, R j×χ represents a matrix with j rows and x columns;

[0142] For a dataset containing M videos, the temporal feature matrices of the M videos are vertically concatenated to obtain the global feature matrix X of the M videos as follows:

[0143]

[0144] Among them, R (M×j)×χ represents a matrix of M×j rows and x columns;

[0145] The density label vector Y of the pharmaceutical process concentrate generated for different video segments is as follows:

[0146]

[0147] Among them, 1 j It is a vector of all 1s, indicating that the j frames of each video share a pharmaceutical process concentrate density label y k (k=1,2,...,M);

[0148] Specifically, suppose there are M videos in total, each video contains 30 frames of images, retain the temporal independence, do not perform mean aggregation, and arrange the 30 feature vectors of each video in chronological order to form the temporal feature matrix of the entire video (i.e., M videos). Suppose the feature vector set of a certain video is {f1,f2,...,f 30}, the size of the feature map is 7×7×512, then the temporal feature matrix of this video is:

[0149]

[0150] For a dataset containing M videos, the temporal feature matrices of all videos are vertically spliced ​​to construct a global feature matrix:

[0151]

[0152] And correspondingly generate the density label vector of pharmaceutical process concentrate:

[0153]

[0154] Among them, 1 30 It is a full 1 vector, indicating that the 30 frames of each video share a pharmaceutical process concentrate density label y k (k=1,2,...,M);

[0155] Furthermore, in step S5, the process of normalizing the global feature matrix is ​​specifically as follows:

[0156] For the global feature matrix X containing M videos, normalize it by column. For the d-th column feature, calculate the mean μ of all samples d and standard deviation σ d , the calculation formula is as follows:

[0157]

[0158] Among them, F p,d Represents the d-th dimension eigenvalue of the p-th sample;

[0159] The standardized eigenvalues ​​are:

[0160]

[0161] Generate standardized feature matrix X norm ∈R (M×j)×χ ;

[0162] Specifically, assuming that each of the M videos contains 30 frames of images, then the feature matrix X∈R for M×30 samples (M×30)×512 Normalize by column;

[0163] For the dth column feature (a total of 512 columns), calculate the mean μ of all samples d and standard deviation σ d :

[0164]

[0165] Among them, F p,d represents the d-th dimension eigenvalue of the p-th sample (i.e., the p / 30th frame of the p / 30th video), and the generated standardized feature matrix is ​​X norm ∈R (M×30)×512 ;

[0166] This step can ensure that the weights of each feature component are balanced when training the multivariate correction model later, avoiding deviations caused by dimensional differences. Specifically, in this embodiment, the multivariate correction model is constructed using the LightGBM algorithm.

[0167] Furthermore, step S6 specifically includes:

[0168] Based on the standardized feature matrix X norm ∈R (M×j)×χ The multivariate calibration model is trained based on the density label vector Y of the pharmaceutical process concentrate. The nonlinear relationship between frame-level features and the density of the pharmaceutical process concentrate is fitted by a gradient boosting decision tree. Specifically, the multivariate calibration model is constructed using the LightGBM algorithm.

[0169] For the input feature z q , the output correction value is as follows:

[0170]

[0171] Where f(·) represents the multivariate calibration model established through training, Represents the density correction value of the pharmaceutical process concentrate for a single frame image;

[0172] Collect the current concentration process liquid surface video and extract the feature matrix X of the θ frame image in the video test ∈R θ×ω , using the trained multivariate calibration model to estimate the θ pharmaceutical process material density calibration values ​​frame by frame The density measurement result of the concentrated liquid corresponding to the current concentrated liquid surface video is the average of the θ pharmaceutical process concentrated liquid density correction values, as follows:

[0173]

[0174] in, represents the density measurement result of the concentrated liquid corresponding to the video of the liquid surface during the current concentration process, and τ represents the τth frame;

[0175] Specifically, assuming that based on multiple standardized feature matrices X norm ∈R (M×30)×512 and multiple pharmaceutical process concentrate density label vectors Y∈R (M×30)×1 (30 frames of each video share the same label y k ), train a multivariate correction model based on LightGBM;

[0176] For the newly collected video of the surface of the concentrated liquid, suppose the feature matrix X of its 30 frames is extracted test ∈R 30 ×512, using the trained multivariate correction model to estimate 30 concentrate density correction values ​​frame by frame The final video-level densitometric result is the mean of all frame-corrected values:

[0177]

[0178] Example 4

[0179] An embodiment of the present invention further provides a cross-modal robotic measurement and control system for pharmaceutical process density, which is applicable to the cross-modal measurement method for pharmaceutical process density as described above, and measures and monitors pharmaceutical process density, including an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit, and a remote monitoring unit;

[0180] The industrial robot unit is used to carry the machine vision unit;

[0181] The machine vision unit is used to collect multiple segments of liquid surface video in the pharmaceutical and chemical concentration process in real time;

[0182] The edge computing unit is used to process multiple segments of concentrated liquid surface video collected in real time by the machine vision unit, and continuously learn, calculate, and update the multivariate correction model for pharmaceutical process material density to determine the relative density of the pharmaceutical and chemical concentrated liquid;

[0183] The feedback control unit is used to receive the relative density signal of the pharmaceutical and chemical concentration process liquid output by the edge computing unit and output a control signal to the system;

[0184] The remote monitoring unit is used to analyze sampled image data in real time and monitor the dynamic changes in the density of the drug solution during the pharmaceutical and chemical concentration process. When the relative density of the drug solution reaches the preset judgment condition, an alarm signal is issued to remind the operator to perform control intervention;

[0185] Specifically, the system is mainly composed of industrial robot unit, machine vision unit, edge computing unit, feedback control unit, remote monitoring unit, etc. Figure 3 As shown, the workflow is as follows:

[0186] The industrial robot unit carries a machine vision unit, and the machine vision unit carried at the end of the industrial robot unit moves to the observation window of the concentration tank involved in the pharmaceutical process; the industrial camera in the machine vision unit can collect image data non-contactly by automatically photographing the surface of the concentrated liquid; through the intelligent AI automatic calculation of the edge computing unit, the relative density of the concentrated liquid in the pharmaceutical process can be obtained in real time; the remote monitoring unit analyzes and monitors the relative density of the concentrated liquid in the pharmaceutical process in real time, and judges the end point of the pharmaceutical and chemical concentration process; if the density measurement value does not meet the judgment condition of the concentration end point, the industrial robot unit carrying the machine vision unit will re-shoot. When the density measurement value reaches the judgment condition of the concentration end point, an alarm signal will be issued to remind the operator to complete subsequent on-site operations, or the feedback control unit will automatically output a control signal to the control system of the concentration device, thereby terminating the concentration process and power supply in time.

[0187] The specific method for relative density measurement involves extracting multi-scale image features, including texture complexity and bubble distribution, through a convolutional neural network or ResNet residual network. This is combined with the LightGBM algorithm to construct a nonlinear multivariate correction model, enabling cross-modal measurement of the relative density of a concentrated liquid in a pharmaceutical process from visual images of its boiling surface. (This algorithm is the core algorithm of the edge computing unit.)

[0188] Example 5

[0189] Different from the fourth embodiment, in this embodiment, the industrial robot unit is one of a fixed robot and a composite mobile robot;

[0190] The machine vision unit is an industrial camera installed on the industrial robot unit;

[0191] Specifically, the automatic photography industrial robot unit can adopt a variety of solutions, for example:

[0192] (1) The camera shooting action is realized by a fixed robot. The robot is fixed next to the concentration tank observation window. When a photo is needed, the robot drives the camera to the appropriate position to take the photo. After the photo is taken, it leaves to prevent the window from being damaged by high temperature. The robot can be a single-axis robot, a six-axis robot, or other types of industrial robots;

[0193] (2) The camera shooting action is realized by a composite mobile robot. When it is necessary to take a photo, the composite robot moves to the vicinity of the concentration tank, and the manipulator on the composite robot automatically completes the photo shooting. Among them, the mobile chassis can be a wheeled robot, a four-legged robot dog, a humanoid robot, etc.

[0194] In terms of density detection, it can achieve:

[0195] 1. High-precision detection: The mean absolute error (MAE) between the measured density value and the laboratory measurement value is less than 1.5%, which is better than traditional sensors;

[0196] 2. Strong real-time performance: The response time from video acquisition to density measurement is less than 10 seconds, supporting online monitoring and control;

[0197] 3. Eliminates the problems of easy blockage and contamination risks of contact sensing detection equipment, and avoids tedious operations such as sensor cleaning;

[0198] 4. Anti-interference ability: Standardized light source and video noise reduction technology effectively eliminate ambient light fluctuations and steam interference.

[0199] In terms of intelligent pharmaceutical process, it can achieve:

[0200] 1. Automatically detect the density of the pharmaceutical process, avoiding the tedious process of manual density detection operation;

[0201] 2. Real-time density detection in pharmaceutical and chemical concentration processes, and automatic feedback control based on intelligent judgment of the concentration endpoint;

[0202] 3. Accurately control the concentration endpoint, reduce the deviation of concentrate density between different batches, and improve the consistency of process quality;

[0203] 4. Terminate the energy supply of the concentration process in time to avoid energy waste.

[0204] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the scope of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A cross-modal density determination method for pharmaceutical processes based on AI, characterized in that: The following steps are involved: S1, real-time acquisition of multiple segments of pharmaceutical process material surface video; S2. Frame processing is performed on each collected video segment of the material surface of the pharmaceutical process to obtain multiple image sets of the multiple video segments; S3, preprocessing and feature extraction are performed on each frame of the image in the plurality of image sets to obtain a feature vector of each frame of the image; S4. Sort and concatenate the feature vectors of each frame of each video in chronological order to obtain a global feature matrix of multiple videos, and generate density label vectors of pharmaceutical process materials corresponding to different video segments; S5, normalizing the global feature matrix obtained in step S4 to obtain a standardized feature matrix; S6. Use the standardized feature matrix and the corresponding pharmaceutical process material density label vector as the training data set, fit the nonlinear relationship between image features and pharmaceutical process material density, and construct a cross-modal human-like visual depth multivariate correction model that spans from image values ​​to density measurement values, so as to determine the pharmaceutical process material density.

2. The AI-based cross-modal density measurement method for pharmaceutical processes according to claim 1, characterized in that: In step S1, the resolution of the collected pharmaceutical process material surface video parameters is set to W×H, the frame rate is set to F, W×H represents pixels, F represents the number of frames, and the output format is .mp4 format.

3. The AI-based cross-modal density measurement method for pharmaceutical processes according to claim 1, characterized in that: In step S2, the frame interval time Δt is calculated according to the following formula: Where N represents the total number of frames in the video; T represents the total duration of the video of the surface of a pharmaceutical process material collected; According to the frame interval time Δt, the time point for frame division is determined using the following calculation formula: t i =i×Δt(i=0,1,...,N-1), Where i represents the i-th frame of the video; Extract the video frames according to the time point of the frame division and obtain the framed image set {I1,I2,...,I N }, where {I1,I2,...,I N } indicates that the image set contains N frames of images, and I1 indicates the i-th frame of image.

4. The AI-based cross-modal density measurement method for pharmaceutical processes according to claim 1, characterized in that: In step S3, the process of preprocessing each frame of the plurality of image sets specifically includes: S301, assuming that a certain frame image is I i , image I i Scaling to pixels α×β, the formula is as follows: in, Represents the scaled image, Resized(·) represents the image scaling operation, and α×β represents the scaled image pixels; S302: scaling the image Perform center cropping, the formula is as follows: in, represents the image after center cropping, CenterCrop(·) represents the image center cropping operation, γ×η represents the image pixels after center cropping, 0<γ<α, 0<η<β; S303: Normalize the center-cropped image using the following formula: in, represents the normalized image, μ and σ represent the normal distribution parameters after normalization, (x, y, ) represents the pixel coordinates, ( , , c) represents the channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

5. The AI-based cross-modal density determination method for pharmaceutical processes according to claim 4, characterized in that: In step S3, the process of extracting features from the normalized image is as follows: The normalized image is input into the convolutional neural network, and features are extracted through multi-layer convolution operations. The output Z of the lth layer of convolution is l Expressed as: WITH l =ReLU(W l *WITH l-1 +b l ), Among them, W l represents the convolution kernel weight, Z l-1 represents the feature map of the l-1 layer, b l represents the bias of layer l; After multiple layers of convolution operations, a feature map is obtained, which is compressed by global average pooling to generate the feature vector of the image: Among them, f i represents the feature vector of the image, (x',y') represents the pixel coordinates in the feature map, and ρ and υ both represent the size of the feature map.

6. The AI-based cross-modal density measurement method for pharmaceutical processes according to claim 1, characterized in that: Step S4 specifically includes: Assume that the feature vector set of each frame image in a certain video image set is {f1,f2,...,f j }, sort the feature vector set in chronological order to obtain the temporal feature matrix F of the image set video as follows: Among them, R j×χ represents a matrix with j rows and x columns; For a dataset containing M videos, the temporal feature matrices of the M videos are vertically concatenated to obtain the global feature matrix X of the M videos as follows: Among them, R (M×j)×χ represents a matrix of M×j rows and x columns; The density label vector Y of the pharmaceutical process materials generated for different video segments is as follows: Among them, 1 j is a vector of all 1s, indicating that the j frames of each video share a pharmaceutical process material density label y k (k=1,2,...,M).

7. The AI-based cross-modal density determination method for pharmaceutical processes according to claim 6, characterized in that: In step S5, the process of normalizing the global feature matrix is ​​specifically as follows: For the global feature matrix X containing M videos, normalize it by column. For the d-th column feature, calculate the mean μ of all samples d and standard deviation σ d , the calculation formula is as follows: Among them, F p,d Represents the d-th dimension eigenvalue of the p-th sample; Normalized eigenvalues for: Generate standardized feature matrix X norm ∈R (M×j)×χ .

8. The AI-based cross-modal density determination method for pharmaceutical processes according to claim 7, characterized in that: Step S6 specifically includes: Based on the standardized feature matrix X norm ∈R (M×j)×χ The LightGBM-based multivariate calibration model is trained with the pharmaceutical process material density label vector Y, and the nonlinear relationship between frame-level features and pharmaceutical process material density is fitted by the gradient boosting decision tree; For the input feature z q , the output correction value is as follows: Where f(·) represents the multivariate calibration model established through training, Represents the density correction value of pharmaceutical process materials for a single frame image; Collect the surface video of the material in the current pharmaceutical process and extract the feature matrix X of the θ frame image in the video test ∈R θ×ω , using the trained multivariate calibration model to estimate the θ pharmaceutical process material density calibration values ​​frame by frame The material density measurement result corresponding to the current pharmaceutical process material surface video is the average of the θ pharmaceutical process material density correction values, as follows: in, It represents the material density measurement result corresponding to the material surface video of the current pharmaceutical process, and τ represents the τth frame.

9. An AI-based pharmaceutical process density cross-modal measurement and control robot system, which uses the AI-based pharmaceutical process density cross-modal measurement method according to any one of claims 1 to 8 to measure and monitor pharmaceutical process density, characterized in that: Including industrial robot unit, machine vision unit, edge computing unit, feedback control unit and remote monitoring unit; The industrial robot unit is used to carry the machine vision unit; The machine vision unit is used to collect multiple segments of material surface videos during the pharmaceutical process in real time; The edge computing unit is used to process multiple segments of pharmaceutical process material surface videos collected in real time by the machine vision unit, and continuously learn, calculate, and update the multivariate correction model for pharmaceutical process material density to measure the density of pharmaceutical process materials. The feedback control unit is used to receive the pharmaceutical process material density signal output by the edge computing unit and output a control signal to the system; The remote monitoring unit is used to analyze the sampled image data in real time and monitor the dynamic changes in the density of the pharmaceutical process materials. When the density of the pharmaceutical process materials reaches the preset judgment conditions, an alarm signal is issued to remind the operator to perform control intervention.

10. The AI-based pharmaceutical process density cross-modal measurement and control robot system according to claim 9, characterized in that: The industrial robot unit is one of a fixed robot and a composite mobile robot; The machine vision unit is an industrial camera installed on the industrial robot unit.

Images