AI-based pharmaceutical process density cross-modal determination method and control robot system

Abstract

The application belongs to the technical field of pharmaceutical process quality measurement and control, and provides an AI-based pharmaceutical process density cross-modal measurement method and a measurement and control robot system; comprising the following steps: S1, real-time collection of video data on the surface of material in a plurality of pharmaceutical processes; S2, frame processing of the video to obtain a plurality of image sets; S3, preprocessing of each frame of image in the image set to extract cross-modal image features of density and obtain a feature vector; S4, sorting and splicing of the feature vector to obtain a global feature matrix of the plurality of videos and generate a density label vector; S5, standardization processing of the global feature matrix to obtain a standardized feature matrix; S6, taking the standardized feature matrix and the corresponding density label vector as a data set, fitting a nonlinear relationship between image features and density measurement values, and constructing a human-like visual depth multivariate correction model to cross-modally measure the density of the pharmaceutical process. The application can realize non-contact dynamic monitoring of the density of the pharmaceutical process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pharmaceutical process quality measurement and control, and particularly relates to a density cross-modal measurement method for a pharmaceutical process based on AI and a measurement and control robot system. BACKGROUND

[0002] In the production and manufacturing process of medical products, real-time monitoring of material density is a key technology to ensure product quality and batch consistency. Traditional detection methods often rely on visual inspection based on human experience or offline sensing detection equipment (such as handheld density detectors, tuning fork concentration meters, differential pressure density meters, etc.). There are significant bottlenecks: manual measurement takes a long time of 5-10 minutes, resulting in a 2-3 hour lag in process parameter adjustment, which seriously hinders production rhythm; the experience of operators varies, which can lead to subjective judgment deviation, causing the concentration endpoint determination result to deviate from the actual value. In addition, for high-viscosity and high-solid-content concentrated liquids, the density meter is prone to sensor or sample liquid flow channel blockage due to drug liquid adhesion, crystallization or impurity deposition, causing measurement interruption or deviation. The superimposed problems of low efficiency, uncontrollable error and insufficient equipment reliability not only make it difficult to meet the hard requirements of GMP specifications for real-time measurement and control and data traceability in the pharmaceutical process, but also form a quality control blind area in complex scenarios such as cream and polysaccharide medical product concentration processes, highlighting the fundamental contradiction between traditional monitoring methods and the demand for modern pharmaceutical standardization. Specifically, the density detection of conventional medical and chemical concentration processes mainly relies on contact-type sensing detection equipment, which measures the density by directly contacting the drug liquid to measure physical parameters. However, there are the following problems:

[0003] 1. Contact detection method is not ideal: the sensor needs to invade the drug liquid, which can easily introduce impurity pollution and cannot meet the requirements of sterile production, and it is difficult to monitor dynamic changes in real time;

[0004] 2. Single information sensing mode in the pharmaceutical process: relying on a single physical parameter (such as temperature, pressure), which cannot simultaneously capture dynamic visual features of the material surface (such as boiling state, foam shape, and other key density image features);

[0005] 3. Response lag: poor real-time performance, manual sampling cannot capture the rapid changes in the concentration state, and the detection period is long (usually tens of minutes), making it difficult to track rapid changes in density;

[0006] 4. Unclear process quality change rule: conventional operations only detect the material density at the concentration endpoint, which makes it difficult to understand the process quality change rule, leading to endpoint density deviation and inaccurate control of the concentration endpoint;

[0007] 5. Energy waste: the energy supply equipment in the pharmaceutical process does not stop in time, which can easily cause energy waste;

[0008] 6. Environmental interference sensitive: environmental factors such as steam and temperature can easily affect the accuracy of the sensing detection equipment;

[0009] 7. Insufficient reliability of detection equipment: high viscosity materials often contain solid particles or foam, which can easily block the measurement channel or probe of the densimeter; regular disassembly and cleaning, shutdown cleaning or replacement of parts are required, increasing equipment maintenance cost and downtime, affecting production efficiency.

[0010] In summary, the existing difficulties in density measurement in the pharmaceutical process are:

[0011] 1. Gap between manual experience and measurement value: it is difficult to convert the manual experience of evaluating the density of the pharmaceutical process into quantitative indicators;

[0012] 2. Split between visual features and physical parameters: traditional methods lack the ability to analyze the surface cross-modal features (visual / physical) of the material;

[0013] 3. Insufficient dynamic process modeling: manual sampling and single-point measurement cannot construct a continuous density change model;

[0014] 4. Lack of intelligent sensing technology: there is no non-contact density cross-modal measurement method based on deep learning and human vision.

[0015] These problems seriously restrict the precision of the density measurement and control of the pharmaceutical process, and an innovative solution that integrates multi-modal information, dynamic feature extraction, and AI sensing is urgently needed. SUMMARY

[0016] The present application aims to solve the technical problems existing in the prior art, and provides an AI-based pharmaceutical process density cross-modal measurement method and control robot system combining machine automatic operation, computer vision and multi-dimensional information fusion, which realizes non-contact dynamic monitoring and real-time feedback control of the pharmaceutical process density.

[0017] To achieve the above technical purposes, the application adopts the following technical solutions:

[0018] An AI-based pharmaceutical process density cross-modal measurement method, comprising the following steps:

[0019] S1, real-time acquisition of multiple segments of pharmaceutical process material surface video;

[0020] S2, frame processing of each segment of the acquired pharmaceutical process material surface video to obtain multiple image sets of multiple segments of video;

[0021] S3, pre-processing and feature extraction of each frame of image in the multiple image sets to obtain a feature vector of each frame of image;

[0022] S4. Sort and concatenate the feature vectors of each frame of each video in chronological order to obtain the global feature matrix of multiple video segments, and generate material density label vectors for the pharmaceutical process corresponding to different video segments.

[0023] S5. Standardize the global feature matrix obtained in step S4 to obtain the standardized feature matrix.

[0024] S6. Using the standardized feature matrix and the corresponding pharmaceutical process material density label vector as the training dataset, fit the nonlinear relationship between image features and pharmaceutical process material density, and construct a cross-modal humanoid visual depth multivariate correction model that spans from image values ​​to density measurement values, thereby determining the pharmaceutical process material density.

[0025] Optionally, in step S1, the video parameters of the material surface collected in the pharmaceutical process are set to resolution W×H and frame rate F, where W×H represents pixels and F represents the number of frames, and the output format is .mp4.

[0026] Optionally, in step S2, the frame interval time Δt is calculated according to the following formula:

[0027]

[0028] Where N represents the total number of frames in the video, and T represents the total duration of the video of the material surface during a certain segment of the pharmaceutical process.

[0029] Based on the framing interval Δt, the time point for framing is determined using the following formula:

[0030] t i = i × Δt (i = 0, 1, ..., N-1),

[0031] Where i represents the i-th frame of the video;

[0032] Video frames are extracted according to the time points of the frame segmentation, resulting in a set of segmented images {I1,I2,...,I...} N}, where {I1,I2,...,I N} indicates that the image set contains N frames, and I1 represents the i-th frame.

[0033] Optionally, step S3, the preprocessing of each frame in the multiple image sets, specifically includes:

[0034] S301, Assume a certain frame image is I i Image I i To scale to pixels α×β, the formula is as follows:

[0035]

[0036] wherein, Resized(·) represents an image resizing operation, and α×β represents the pixels of the resized image;

[0037] S302, center cropping is performed on the resized image The formula is as follows:

[0038]

[0039] wherein, CenterCrop(·) represents an image center cropping operation, and γ×η represents the pixels of the center cropped image, 0<γ<α, 0<η<β;

[0040] S303, standardization is performed on the center cropped image, and the formula is as follows:

[0041]

[0042] wherein, μ,σ represent the normal distribution parameters after standardization processing, (x,y,·) represents the pixel point coordinates, (·,·,c) represents the channel, R represents the red channel, G represents the green channel, and B represents the blue channel.

[0043] Optionally, in step S3, the process of feature extraction on the standardized image is specifically as follows:

[0044] The standardized image is input into a convolutional neural network, and features are extracted through multi-layer convolution operation. The output of the lth layer of convolution is represented as:

[0045] Z l =ReLU(W l *Z l-1 +b l ),

[0046] wherein, W l represents a convolution kernel weight, Z l-1 represents the feature map of the (l-1)th layer, and b l represents the bias of the lth layer.

[0047] After multi-layer convolution operation, a feature map is obtained, and the feature map is compressed through a global average pooling operation to generate a feature vector of the image:

[0048]

[0049] wherein, f i represents the feature vector of the image, (x',y') represents the pixel coordinates in the feature map, and both ρ and υ represent the size of the feature map.

[0050] Optionally, step S4 specifically comprises:

[0051] Suppose the feature vector set of each frame image in the image set of a certain video segment is {f1, f2,..., f j , the feature vector set is sorted in time sequence to obtain the time sequence feature matrix F video of the image set as follows:

[0052]

[0053] wherein, R j×χ represents a matrix of j rows and χ columns;

[0054] For a data set containing M video segments, the time sequence feature matrices of the M video segments are vertically spliced in sequence to obtain the global feature matrix X of the M video segments as follows:

[0055]

[0056] wherein, R (M×j)×χ represents a matrix of Mxj rows and χ columns;

[0057] The pharmaceutical process material density label vector Y corresponding to different video segments is generated as follows:

[0058]

[0059] wherein, 1 j is a full 1 vector, indicating that j frames of each video segment share a pharmaceutical process material density label y k (k=1, 2,..., M).

[0060] Optionally, in step S5, the process of normalizing the global feature matrix is specifically:

[0061] For the global feature matrix X containing M video segments, the column is normalized. For the dth column feature, the mean μ d and the standard deviation σ d of all samples are calculated, and the calculation formula is as follows:

[0062]

[0063] wherein, F p,d represents the dth dimensional feature value of the pth sample;

[0064] The normalized feature value is:

[0065]

[0066] The normalized feature matrix X norm ∈R(M×j)×χ .

[0067] Optionally, step S6 specifically comprises:

[0068] training a LightGBM-based multivariate correction model based on the standardized feature matrix X norm ∈R (M×j)×χ and the pharmaceutical process material density label vector Y, fitting the non-linear relationship between the frame-level features and the pharmaceutical process material density through gradient boosting decision trees;

[0069] For the input feature z q , the output correction value is as follows:

[0070]

[0071] wherein f(·) represents the trained multivariate correction model, represents the pharmaceutical process material density correction value of a single frame image;

[0072] Collecting the current pharmaceutical process material surface video, extracting the feature matrix X test ∈R θ×ω of θ frame images in the video, using the trained multivariate correction model to estimate θ pharmaceutical process material density correction values frame by frame The material density measurement result corresponding to the current pharmaceutical process material surface video is the average of the θ pharmaceutical process material density correction values, as follows:

[0073]

[0074] wherein, represents the material density measurement result corresponding to the current pharmaceutical process material surface video, and τ represents the τth frame.

[0075] The application also provides an AI-based pharmaceutical process density cross-modal measurement and control robot system, which uses the AI-based pharmaceutical process density cross-modal measurement method as described above to measure and monitor the pharmaceutical process density, and comprises an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit and a remote monitoring unit.

[0076] The industrial robot unit is used to carry the machine vision unit.

[0077] The machine vision unit is used to collect multiple segments of pharmaceutical process material surface videos in real time.

[0078] The edge computing unit is used to process the multiple segments of pharmaceutical process material surface videos collected in real time by the machine vision unit, and continuously learn, calculate and update the pharmaceutical process material density multivariate correction model for measuring the pharmaceutical process material density.

[0079] The feedback control unit is used for receiving the pharmaceutical process material density signal output by the edge computing unit and outputting a control signal to the system.

[0080] The remote monitoring unit is used for analyzing the sampling image data in real time and monitoring the dynamic change of the pharmaceutical process material density.

[0081] Optionally, the industrial robot unit is one of a fixed robot and a compound mobile robot.

[0082] The machine vision unit is an industrial camera arranged on the industrial robot unit.

[0083] Compared with the prior art, the present application has the following beneficial effects:

[0084] (1) Non-contact dynamic detection: the present application realizes non-contact dynamic detection through the industrial robot unit and the machine vision unit, and realizes non-invasive density detection by combining the real-time acquisition of the pharmaceutical process material surface video through the industrial camera on the machine vision unit and the visual feature analysis of the edge computing unit.

[0085] (2) Multi-dimensional information fusion: the present application realizes multi-dimensional information fusion of dynamic and static by collecting the dynamic video of the pharmaceutical process material surface and extracting the features of each frame in the video to comprehensively represent the dynamic of the pharmaceutical process.

[0086] (3) Time sequence feature reservation: the present application reserves the independence of time sequence by constructing a video frame level feature matrix, and realizes density determination based on the dynamic video of the pharmaceutical process material surface by constructing a multi-element correction model based on LightGBM through video frame multi-dimensional feature extraction and time sequence feature matrix construction technology, fitting the nonlinear mapping relationship between the video frame level features and the density of the pharmaceutical process material, and then realizing the density determination based on the dynamic video of the pharmaceutical process material surface.

[0087] (4) Density determination and real-time feedback: the present application obtains image signals and extracts image features based on the current collected pharmaceutical process material surface video, inputs the image features into the multi-element correction model based on LightGBM, and outputs the density determination value of the pharmaceutical process material, and transmits the data in real time, and outputs the control signal to the system through the feedback control unit.

[0088] (5) High precision detection: the application reduces the average absolute error between the material density measurement value in the pharmaceutical process and the laboratory measurement value through video frame multi-dimensional feature engineering, is better than the traditional sensor, and has high real-time detection, realizes automatic feedback control based on the intelligent judgment of the concentrated preset condition, accurately controls the end point of concentration, timely terminates the energy supply equipment of the concentration system, avoids energy waste, reduces the deviation of the concentration liquid density between different batches, and improves the consistency of process quality;

[0089] (6) Strong real-time performance: the application has short response time from video acquisition to density measurement, strong real-time performance, and supports online monitoring and control;

[0090] (7) Anti-interference ability: the light source system of the machine vision unit is supplemented by the ring-shaped LED fill light, which ensures uniform illumination of the material surface, and the standard light source and video noise reduction technology effectively eliminate environmental light fluctuation and steam interference;

[0091] (8) Simplify the process: the application eliminates the problems of easy clogging and pollution risk of contact type sensing detection equipment, avoids the tedious operation of sensor cleaning, realizes automatic density detection, and avoids the tedious process of manual density detection operation. BRIEF DESCRIPTION OF DRAWINGS

[0092] Figure 1 The application provides an AI-based pharmaceutical process density cross-modal measurement method flowchart;

[0093] Figure 2 The application provides an AI-based pharmaceutical process density cross-modal measurement method flowchart;

[0094] Figure 3 The application provides an AI-based pharmaceutical process density cross-modal measurement method flowchart; DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0096] Embodiment 1

[0097] In combination with Figures 1-2 The application provides an AI-based pharmaceutical process density cross-modal measurement method, which comprises the following steps:

[0098] S1, real-time acquisition of multiple pharmaceutical process material surface videos;

[0099] S2. Perform frame-by-frame processing on each segment of the video of the material surface during the pharmaceutical process to obtain multiple image sets from multiple video segments.

[0100] S3. Perform preprocessing and feature extraction on each frame of the image in the multiple image sets to obtain the feature vector of each frame;

[0101] S4. Sort and concatenate the feature vectors of each frame of each video in chronological order to obtain the global feature matrix of multiple video segments, and generate material density label vectors for the pharmaceutical process corresponding to different video segments.

[0102] S5. Standardize the global feature matrix obtained in step S4 to obtain the standardized feature matrix.

[0103] S6. Using the standardized feature matrix and the corresponding pharmaceutical process material density label vector as the training dataset, fit the nonlinear relationship between image features and pharmaceutical process material density, and construct a cross-modal humanoid visual depth multivariate correction model that spans from image values ​​to density measurement values, thereby determining the pharmaceutical process material density.

[0104] Example 2

[0105] Combination Figures 1-2 As shown, based on Example 1, in this example, in step S1, the video parameters of the pharmaceutical process liquid surface are set to resolution W×H, frame rate F, and output format .mp4; the pharmaceutical process is specifically a pharmaceutical chemical concentration process.

[0106] Specifically, an industrial camera is used to capture real-time dynamic video of the pharmaceutical solution surface during the concentration process. The camera is positioned directly above the transparent observation window of the concentration tank, with the lens focal length covering the boiling area of ​​the solution. Video parameters are set to resolution W×H, frame rate F, and output format .mp4. A lighting system using a ring-shaped LED supplemental light is installed on the industrial camera to ensure uniform illumination of the solution surface and avoid glare interference. This non-contact video acquisition completely eliminates the risk of contamination caused by contact between traditional sensors and the pharmaceutical solution.

[0107] Further, in step S2, let the total duration of the video of the surface of the pharmaceutical liquid during a certain pharmaceutical concentration process be T (.mp4 format, resolution W×H, frame rate F, total duration T seconds, where W×H represents pixels and F represents the number of frames), and calculate the frame interval time Δt. The calculation formula is as follows:

[0108]

[0109] Where N represents the total number of frames in the video;

[0110] According to the frame interval time Δt, a time point for frame division is determined, and the calculation formula is as follows:

[0111] t i =i×Δt(i=0,1,...,N-1),

[0112] wherein i represents the i-th frame of the video;

[0113] According to the time point for frame division, the video frames are extracted to obtain a set of images {I1, I2,..., I N} after frame division, wherein {I1, I2,..., I N} represents that a total of N frames of images are contained in the image set, and I1 represents the i-th frame of image;

[0114] Specifically, assuming that N=30, each time point t i =i×Δt(i=0,1,...,N-1) for frame division is determined, and the video frames are extracted at this interval to obtain a set of images {I1, I2,..., I 30} after frame division, and this step can ensure that the dynamic changes of the concentration process are uniformly captured from the video.

[0115] Further, in step S3, the process of pre-processing each frame of image in the plurality of image sets specifically includes:

[0116] S301, assuming that a certain frame of image is I i , the image I i is scaled to pixels α×β, and the formula is as follows:

[0117]

[0118] wherein, Resized(·) represents the image scaling operation, and α×β represents the pixels of the scaled image;

[0119] Specifically, the pixels α×β can be 256×256, that is,

[0120] S302, the scaled image I is center cropped, and the formula is as follows:

[0121]

[0122] wherein, CenterCrop(·) represents the image center cropping operation, and γ×η represents the pixels of the center cropped image, 0<γ<α, and 0<η<β;

[0123] Specifically, the center cropped pixels can be 224x224, i.e.

[0124] S303, the center cropped image is standardized, and the formula is as follows:

[0125]

[0126] wherein, denotes the standardized image, μ and σ denote the normal distribution parameters after standardization processing, (x, y, ·) denotes the pixel point coordinates, (·, ·, c) denotes the channel, R denotes the red channel, G denotes the green channel, and B denotes the blue channel;

[0127] At this time, μ can be calculated as [0.485, 0.456, 0.406], and σ can be calculated as [0.229, 0.224, 0.225]; the pixel value of the processed image obeys the standard normal distribution, which is convenient for subsequent feature extraction.

[0128] Further, the process of feature extraction on the standardized image is specifically as follows:

[0129] The standardized image is input into a convolutional neural network, and features are extracted through multi-layer convolution operation. The output of the lth layer of convolution is denoted as:

[0130] Z l = ReLU(W l *Z l-1 +b l ),

[0131] wherein, W l denotes the convolution kernel weight, Z l-1 denotes the feature map of the (l-1)th layer, and b l denotes the bias of the lth layer.

[0132] After multi-layer convolution operation, a feature map is obtained, and the feature map is compressed through a global average pooling operation to generate a feature vector of the image:

[0133]

[0134] wherein, f i denotes the feature vector of the image, (x', y') denotes the pixel coordinates in the feature map, and both ρ and υ denote the size of the feature map.

[0135] Specifically, the size of the feature map is 7x7x512, i.e. the feature map is compressed through a global average pooling (GAP) operation to generate a feature vector of the image with a length of 512, and the vector represents the spatial global features of the image:

[0136]

[0137] Embodiment 3

[0138] In combination Figures 1-2 As shown in the embodiment 2, on the basis of the embodiment 2, in the embodiment, the step S4 specifically comprises:

[0139] Suppose the feature vector set of each frame image in the image set of a certain video is {f1, f2,...,f j , sort the feature vector set according to the time sequence, and obtain the time sequence feature matrix F video of the image set as follows:

[0140]

[0141] Wherein, R j×χ represents a matrix of j rows and χ columns;

[0142] For a data set containing M videos, the time sequence feature matrices of the M videos are vertically spliced in sequence to obtain the global feature matrix X of the M videos as follows:

[0143]

[0144] Wherein, R (M×j)×χ represents a matrix of Mxj rows and χ columns;

[0145] The pharmaceutical process concentrate density label vector Y corresponding to different videos is as follows:

[0146]

[0147] Wherein, 1 j is a full 1 vector, indicating that j frames of each video share a pharmaceutical process concentrate density label y k (k=1, 2,..., M);

[0148] Specifically, suppose that a total of M videos are included, each video contains 30 frames of images, the time sequence independence is retained, and no mean aggregation is performed, 30 feature vectors of each video are arranged in time sequence to form a time sequence feature matrix of the entire video (i.e. M videos), wherein suppose that the feature vector set of a certain video is {f1, f2,...,f 30}, and the size of the feature map is 7x7x512, then the time sequence feature matrix of the video is:

[0149]

[0150] For a data set containing M videos, the time sequence feature matrices of the M videos are vertically spliced in sequence to obtain the global feature matrix X of the M videos as follows:

[0151]

[0152] And the corresponding pharmaceutical process concentrate density label vector is generated:

[0153]

[0154] Wherein, 1 30 is a full 1 vector, indicating that 30 frames of each video share a pharmaceutical process concentrate density label y k (k = 1, 2,..., M);

[0155] Further, in step S5, the process of standardizing the global feature matrix is specifically:

[0156] For the global feature matrix X containing M videos, the column is standardized. For the dth column feature, the mean μ d and standard deviation σ d of all samples are calculated, and the calculation formula is as follows:

[0157]

[0158] Where F p,d represents the dth dimensional feature value of the pth sample;

[0159] The standardized feature value is:

[0160]

[0161] The standardized feature matrix X norm ∈ R (M×j)×χ ;

[0162] Specifically, assuming that each of the M videos contains 30 frames of images, then for the M × 30 sample feature matrix X ∈ R (M×30)×512 , the column is standardized;

[0163] For the dth column feature (a total of 512 columns), the mean μ d and standard deviation σ d of all samples are calculated:

[0164]

[0165] Where F p,d represents the dth dimensional feature value of the pth sample (i.e. the p / 30th frame of the p / 30th video), and the generated standardized feature matrix is X norm ∈ R (M×30)×512 ;

[0166] This step can ensure the weight balance of each feature component in subsequent training of the multivariate correction model, and avoid the deviation caused by the dimension difference; specifically, in the embodiment, the multivariate correction model is constructed by using the LightGBM algorithm.

[0167] Further, step S6 specifically includes:

[0168] Based on the standardized feature matrix X norm ∈R (M×j)×χ and the pharmaceutical process concentrate density label vector Y, the multivariate correction model is trained, and the non-linear relationship between the frame-level features and the pharmaceutical process concentrate density is fitted by gradient boosting decision tree; specifically, the multivariate correction model is constructed by using the LightGBM algorithm.

[0169] For the input feature z q , the correction value is output as follows:

[0170]

[0171] Wherein, f(·) represents the trained and established multivariate correction model, represents the pharmaceutical process concentrate density correction value of a single frame image;

[0172] The video of the current concentrate process liquid surface is collected, the feature matrix X test ∈R θ×ω of θ frames of images in the video is extracted, and the trained and established multivariate correction model is used to estimate the pharmaceutical process material density correction value The concentrate liquid density measurement result corresponding to the current concentrate process liquid surface video is the average of the θ pharmaceutical process concentrate density correction values, as follows:

[0173]

[0174] Wherein, represents the concentrate liquid density measurement result corresponding to the current concentrate process liquid surface video, and τ represents the τth frame.

[0175] Specifically, it is assumed that based on a plurality of standardized feature matrices X norm ∈R (M×30)×512 and a plurality of pharmaceutical process concentrate density label vectors Y (M×30)×1 (30 frames of each video share the same label y k ), a multivariate correction model based on LightGBM is trained.

[0176] For the newly collected concentrate process liquid surface video, it is assumed that the feature matrix X test ∈R 30 ×512, using the trained multivariate correction model to estimate 30 concentrated liquid density correction values frame by frame The final video-level density measurement result is the average of all frame correction values:

[0177]

[0178] Example 4

[0179] The embodiments of the present application also provide a pharmaceutical process density cross-modal robot measurement and control system, which is suitable for the pharmaceutical process density cross-modal measurement method as described above, and is used for measuring and monitoring the pharmaceutical process density, and includes an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit and a remote monitoring unit.

[0180] The industrial robot unit is used to carry the machine vision unit.

[0181] The machine vision unit is used to collect the surface video of the pharmaceutical chemical concentrated liquid in real time.

[0182] The edge computing unit is used to process the surface video of the concentrated liquid collected by the machine vision unit in real time, and continuously learns, calculates and updates the pharmaceutical process material density multivariate correction model, which is used to measure the relative density of the pharmaceutical chemical concentrated liquid.

[0183] The feedback control unit is used to receive the relative density signal of the pharmaceutical chemical concentrated liquid output by the edge computing unit, and output a control signal to the system.

[0184] The remote monitoring unit is used to analyze the sampling image data in real time and monitor the dynamic change of the density of the pharmaceutical chemical concentrated liquid, and when the relative density of the liquid reaches the preset judgment condition, an alarm signal is sent to remind the operator to control intervention.

[0185] Specifically, the system mainly includes an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit, a remote monitoring unit, etc., combined with the multivariate correction model of the pharmaceutical process density. Figure 3 As shown in the figure, the workflow is as follows:

[0186] The industrial robot unit carries a machine vision unit, and the machine vision unit carried by the end of the industrial robot unit moves to the observation window of the condensation tank involved in the pharmaceutical process; the industrial camera in the machine vision unit can non-contactly collect image data by automatically photographing the surface of the condensed liquid; through the intelligent AI automatic operation of the edge computing unit, the relative density of the condensed liquid in the pharmaceutical process can be obtained in real time; the remote monitoring unit analyzes and monitors the relative density of the condensed liquid in the pharmaceutical process in real time, and judges the end point of the pharmaceutical and chemical condensation process; if the density measurement value does not reach the judgment condition of the condensation end point, the machine vision unit carried by the industrial robot unit is used to re-photograph, and when the density measurement value reaches the judgment condition of the condensation end point, an alarm signal is sent to remind the operator to complete the subsequent on-site operation, or an automatic output control signal is sent to the control system of the condensation device through the feedback control unit, so that the condensation process and energy supply are timely terminated.

[0187] The specific method of relative density measurement is: multi-scale image features including texture complexity, bubble distribution, etc. are extracted through convolutional neural network or ResNet residual network, and a nonlinear multivariate correction model is constructed by combining LightGBM algorithm, so as to realize the cross-modal measurement from the visual image of the boiling surface of the condensed liquid in the pharmaceutical process to the relative density value of the condensed liquid. (This algorithm is the core algorithm of the edge computing unit.)

[0188] Embodiment 5

[0189] Different from embodiment 4, in this embodiment, the industrial robot unit is one of a fixed robot and a composite mobile robot.

[0190] The machine vision unit is an industrial camera arranged on the industrial robot unit.

[0191] Specifically, the industrial robot unit for automatic photographing can adopt various schemes, for example:

[0192] (1) The camera photographing action is realized by a fixed robot, the robot is fixed beside the observation window of the condensation tank, and when photographing is needed, the robot drives the camera to the appropriate position for photographing, and then leaves after photographing is completed to prevent the camera from being damaged by high temperature at the window. The robot can adopt various types of industrial robots such as single-axis robot, six-axis robot, etc.

[0193] (2) The camera photographing action is realized by a composite mobile robot, and when photographing is needed, the composite robot moves to the vicinity of the condensation tank, and the mechanical hand on the composite robot automatically completes the photographing. Among them, the moving chassis can adopt various forms such as wheeled robot, four-legged mechanical dog, humanoid robot, etc.

[0194] In the density detection aspect, the following can be achieved:

[0195] 1. High precision detection: the average absolute error (MAE) between the measured value of density and the laboratory measurement value is less than 1.5%, which is better than traditional sensors;

[0196] 2. Strong real-time performance: the response time from video acquisition to density measurement is less than 10 seconds, supporting online monitoring and control;

[0197] 3. Eliminates the risk of clogging and pollution of contact type sensor detection equipment, avoiding the cumbersome operation of sensor cleaning;

[0198] 4. Anti-interference ability: standardized light source and video noise reduction technology effectively eliminate environmental light fluctuation and steam interference.

[0199] In the intelligentization of the pharmaceutical process, it can achieve:

[0200] 1. Automatic detection of pharmaceutical process density, avoiding the cumbersome process of manual density detection operation;

[0201] 2. Real-time detection of pharmaceutical and chemical concentration process density, and automatic feedback control based on intelligent judgment of the concentration endpoint can be realized;

[0202] 3. Precise control of the concentration endpoint, reducing the deviation of the concentration liquid density between different batches, and improving the consistency of process quality;

[0203] 4. Timely termination of the concentration process energy supply to avoid energy waste.

[0204] The above only describes the embodiments of the present application and does not limit the present application. Any modification, equivalent replacement and improvement made within the scope of the application shall be included in the protection scope of the present application.

Claims

1. An AI-based pharmaceutical process density cross-modal determination method, characterized by, The method comprises the following steps: S1, collecting multiple segments of pharmaceutical process material surface videos in real time; S2, performing frame processing on each segment of the collected pharmaceutical process material surface videos to obtain multiple image sets of the multiple segments of videos; S3, performing preprocessing and feature extraction on each frame of image in the multiple image sets to obtain a feature vector of each frame of image; S4, sorting and splicing the feature vectors of each frame of image of each segment of video in chronological order to obtain a global feature matrix of the multiple segments of videos, and generating a pharmaceutical process material density label vector corresponding to different segments of video; S5, performing standardization processing on the global feature matrix obtained in step S4 to obtain a standardized feature matrix; S6, taking the standardized feature matrix and the corresponding pharmaceutical process material density label vector as a training data set, fitting a nonlinear relationship between image features and pharmaceutical process material density, constructing a cross-modal human visual depth multivariate correction model spanning from image values to density measurement values, and thus determining the pharmaceutical process material density; Step S6 specifically comprises: Standardized feature matrix And pharmaceutical process material density label vector Train a LightGBM-based multivariate correction model to fit the nonlinear relationship between frame-level features and pharmaceutical process material density through gradient boosting decision trees. For input features , the output correction value is as follows: ， wherein, represents a multivariate correction model established by training, represents a pharmaceutical process material density correction value of a single frame image; Collect video of the material surface during the current pharmaceutical process and extract the data from the video. Feature matrix of frame image The trained multivariate correction model is used to estimate the values ​​frame by frame. Material density correction value for pharmaceutical processes The material density measurement results corresponding to the current pharmaceutical process material surface video are as follows: The mean values ​​of material density correction values ​​for each pharmaceutical process are as follows: ， wherein, represents the material density measurement result corresponding to the current pharmaceutical process material surface video, represents the material density measurement result corresponding to the first frame. 2.The AI-based pharmaceutical process density cross-modality measurement method of claim 1, wherein, In step S1, the video parameter resolution of the collected surface of the pharmaceutical process material is set as , , represents a pixel, represents the number of frames, and the output format is.mp4 format. 3.The AI-based pharmaceutical process density cross-modality measurement method of claim 1, wherein, In step S2, the inter-framing interval time is calculated according to the following equation : ， wherein, represents the total number of frames of the video; represents the total time length of the video of the surface of the material of the pharmaceutical process collected in a certain period. According to the frame division interval time The time point for frame division is determined by the following calculation formula: ， wherein, represents the first frame of the video; According to the time point of the frame division, video frames are extracted to obtain a set of images after frame division wherein indicates that the set of images collectively contains frame images, indicates the first frame image. 4.The AI-based pharmaceutical process density cross-modality measurement method of claim 1, wherein, In step S3, the process of preprocessing each frame of image in the multiple image sets specifically comprises: S301、Assume a certain frame of image as , the image is scaled to pixels , the formula is as follows: ， wherein, represents the scaled image, represents the image scaling operation, represents a pixel of the scaled image; S302、to the scaled image Center crop is performed, and the formula is as follows: ， wherein, represents the center cropped image, represents the image center cropping operation, represents the center cropped image pixel, , ; S303, performing standardization on the center-cropped image, and the formula is as follows: ， wherein, denotes the normalized image, , denotes the normal distribution parameter after the normalization process, denotes the pixel point coordinate, denotes the channel, denotes the red channel, denotes the green channel, denotes the blue channel. 5.The AI-based pharmaceutical process density cross-modality measurement method of claim 4, wherein, In step S3, the process of performing feature extraction on the standardized image specifically comprises: The standardized image is input into a convolutional neural network, and features are extracted through multi-layer convolution operations, and the output of the last layer of convolution is represented as: the output of the last layer of convolution is represented as: ， wherein, denotes a convolution kernel weight, denotes a bias of the layer, denotes a bias of the layer. After a plurality of convolution operations, a feature map is obtained, and the feature map is compressed through a global average pooling operation to generate a feature vector of the image: ， wherein, represents a feature vector of an image, represents a pixel coordinate in a feature map, and both represent a size of a feature map. 6.The AI-based pharmaceutical process density cross-modality measurement method of claim 1, wherein, Step S4 specifically comprises: Suppose that the feature vector set of each frame image in the image set of a certain video is The feature vector set is sorted in time sequence to obtain the time sequence feature matrix of the image set as follows: ​ ， wherein represents row column matrix; For the data set of the video clip The time sequence feature matrix of the video clip is longitudinally spliced in sequence to obtain a global feature matrix of the video clip The time sequence feature matrix of the video clip is longitudinally spliced in sequence to obtain a global feature matrix of the video clip The time sequence feature matrix of the video clip is longitudinally spliced in sequence to obtain a global feature matrix of the video clip As follows: ， wherein represents row column matrix; Generating a pharmaceutical process material density label vector for a different segment of the video As follows: ， wherein, is the total vector, representing the frames share one pharmaceutical process material density label . 7.The AI-based pharmaceutical process density cross-modality measurement method of claim 6, wherein, In step S5, the process of performing standardization processing on the global feature matrix specifically comprises: For each segment video, the global feature matrix is computed as follows: The global feature matrix of a segment video Standardization is performed column by column, and for the i-th column feature, the mean and standard deviation of all samples are computed as follows: The mean and standard deviation of all samples are computed for the i-th column feature The mean and standard deviation of all samples are computed for the i-th column feature The mean and standard deviation of all samples are computed for the i-th column feature ， ， wherein, represents the first dimensional feature value of the dimensional feature value of the i-th sample. standardized feature values is: ， Generating a standardized feature matrix .

8. An AI-based pharmaceutical process density cross-modal measurement and control robot system for measurement and monitoring of pharmaceutical process density using the AI-based pharmaceutical process density cross-modal measurement method according to any one of claims 1-7, characterized by, The system comprises an industrial robot unit, a machine vision unit, an edge computing unit, a feedback control unit, and a remote monitoring unit; The industrial robot unit is used to carry the machine vision unit; The machine vision unit is used to collect multiple segments of pharmaceutical process material surface videos in real time; The edge computing unit is used to process the multiple segments of pharmaceutical process material surface videos collected in real time by the machine vision unit, continuously learn, calculate, and update a pharmaceutical process material density multivariate correction model, and determine the pharmaceutical process material density; The feedback control unit is used to receive the pharmaceutical process material density signal output by the edge computing unit and output a control signal to the system; The remote monitoring unit is used to analyze the sampled image data in real time and monitor the dynamic change of the pharmaceutical process material density, and when the pharmaceutical process material density reaches a preset judgment condition, an alarm signal is sent to remind the operator to intervene in control. 9.The AI-based manufacturing process density cross-modal supervisory and control robot system according to claim 8, wherein, The industrial robot unit is one of a fixed robot and a composite mobile robot; The machine vision unit is an industrial camera arranged on the industrial robot unit.

Images