Vaccine quality detection method based on visual language model

Abstract

The invention relates to a vaccine quality detection method based on a visual language model, and the method comprises the steps: S1, obtaining an image of a to-be-detected vaccine, and carrying out the vaccine subject detection to obtain a subject image of the to-be-detected vaccine; s2, inputting the main body image of the vaccine to be detected and a designed prompt word for auxiliary detection into a visual language model; s3, performing visual coding on the main body image by the visual language model to obtain vaccine appearance image features; semantic reasoning is guided through cue words to enable the model to fuse image features and semantic information, then large language model reasoning is carried out, and the quality of the vaccine to be detected is detected. According to the method, the pre-training advantage of the visual language model is fully played to realize high-precision vaccine quality evaluation, and especially in a long-tail scene, unseen vaccine abnormity can be effectively identified, so that the accuracy and reliability of detection are improved, and safe use of vaccines is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of vaccine detection and the technical field of data processing, and relates to a vaccine quality detection method based on a visual language model. Background Art

[0002] In recent years, with the widespread vaccination, the role of vaccines in preventing infectious diseases and improving public health has become more prominent. However, during the process from production to use, there may be problems such as impurities mixed in, seal failure, and damage to vaccine bottles, which may lead to vaccine deterioration, color change, agglomeration, precipitation, stratification, etc., which not only directly affect the vaccination effect, but also may pose certain safety risks.

[0003] Traditional vaccine quality testing methods usually rely on manual testing, where doctors perform visual inspections of vaccines before vaccination based on their experience. This method is subject to the doctor's experience and subjective judgment, and there is a risk of missing potential defects. Therefore, there is an urgent need for an automated method that can quickly evaluate vaccine quality at the vaccination site to help doctors make scientific and accurate decisions and ensure the safety and effectiveness of vaccines.

[0004] Vaccine quality testing based on machine vision and neural network image recognition algorithms has shown significant advantages in vaccine quality testing and can achieve efficient and real-time automated testing. However, these methods also have certain limitations. Traditional machine vision methods rely on preset rules for judgment and have difficulty dealing with complex abnormal situations. Although convolutional neural networks perform well in target recognition, their effectiveness depends on a large amount of training data. However, in reality, vaccine abnormality data is relatively scarce, and it is difficult to collect enough samples for comprehensive training. This leads to poor generalization of the model, and when encountering unseen anomalies in actual scenarios, the recognition accuracy will drop significantly.

[0005] The Vision-Language Model (VLM) combines computer vision and natural language processing, and can process image and text information simultaneously, and perform reasoning through the association between the two. Unlike traditional models, VLM shows stronger generalization in long-tail scenarios and scarce samples through large-scale cross-modal pre-training, can capture subtle features in complex scenarios, and improve the detection performance of unseen samples. In vaccine quality testing, VLM effectively makes up for the shortcomings of traditional machine vision and neural networks with its reasoning ability. However, the reasoning process of VLM consumes a lot of computing resources and has high computational complexity, resulting in slow detection speed, and each detection requires repeated calculations, which is difficult to accelerate. In end-side devices or embedded systems, it is difficult to meet real-time detection requirements. Therefore, optimizing reasoning efficiency and reducing repeated calculations are key issues to enhance the application value of VLM. Summary of the invention

[0006] In view of the above analysis, the present invention aims to disclose a vaccine quality detection method based on a visual language model to solve the problems in the prior art of reliance on manual judgment, insufficient generalization ability of traditional machine learning models, and slow reasoning speed of visual language models.

[0007] The present invention discloses a vaccine quality detection method based on a visual language model, comprising:

[0008] Step S1, obtaining an image of the vaccine to be tested, and obtaining a main body image of the vaccine to be tested after performing vaccine main body detection;

[0009] Step S2, inputting the main image of the vaccine to be tested and the designed prompt words for auxiliary detection into the visual language model;

[0010] Step S3: The visual language model visually encodes the subject image to obtain the vaccine appearance image features; guides semantic reasoning through prompt words to enable the model to integrate the image features with semantic information, and then performs large language model reasoning to detect the quality of the vaccine to be tested.

[0011] Furthermore, in step S1, the process of obtaining the main body image of the vaccine to be tested by vaccine main body detection includes:

[0012] Step S101, vaccine bottle positioning: the input image is processed by the target detection model to automatically identify and locate the vaccine bottle position in the image;

[0013] Step S102, border generation and cropping: Generate an accurate border for the vaccine bottle through the target detection model, isolate the area of ​​the vaccine bottle in the image, crop the area, and remove irrelevant background in the image;

[0014] Step S103: noise removal: using methods including filtering to remove noise, so that the input image is clear and clean, which helps the recognition of the visual language model.

[0015] Furthermore, the target detection model adopts YOLOv5 network or Faster R-CNN.

[0016] Furthermore, the C3RS module is used to replace the C3 module in the YOLOv5 network;

[0017] The C3RS module includes a first CBL layer, a second CBL layer, a RepResBlock layer, a concat layer, a SE layer and a third CBL layer; wherein the first CBL layer is the input layer of the C3RS module, and after the first CBL layer and the RepResBlock layer are connected in series, they are connected to the input end of the concat layer together with the second CBL layer, and the output end of the concat layer is sequentially connected to the SE layer and the third CBL layer; the third CBL layer serves as the output layer of the C3RS module.

[0018] Furthermore, the visual language model processing process in step S3 includes:

[0019] Step S301, performing image feature encoding on the main image of the vaccine to be tested input into the visual language model, and extracting visual features including color, texture, and transparency in the image;

[0020] Step S302: input the designed prompt words into the visual language model to help the model combine image features with semantic information;

[0021] The prompt words are a set of textual expressions describing vaccine characteristics or abnormal phenomena, which are used to help the model combine image features with semantic information, so that the visual language model can accurately understand abnormalities in vaccine images;

[0022] Step S303: fusing the visual features and the prompt words using a cross-modal attention mechanism to align the semantic information of the visual features and the prompt words;

[0023] The cross-modal attention mechanism focuses on the areas in the image related to the prompt word through weight allocation, and automatically filters out irrelevant background noise;

[0024] Step S304: performing large language model inference based on the aligned visual features and semantic information of the prompt word;

[0025] The large language model is a language model including GPT that has been pre-trained on a large number of image-text pairs;

[0026] Step S305: The large language model outputs the result of reasoning based on the fusion features;

[0027] The reasoning results include vaccine abnormalities including color change, agglomeration, precipitation, and stratification, and identify unprecedented abnormalities based on common sense reasoning ability, as well as the reasons for the judgment.

[0028] The present invention also discloses a vaccine quality detection method based on a visual language model, comprising:

[0029] Step 1, establishing a vaccine quality vector library containing sample vectors of vaccines of different qualities;

[0030] Step 2: Obtain an image of the vaccine to be tested, and obtain a main body image of the vaccine to be tested after performing a vaccine main body test;

[0031] Step 3, extracting features from the vaccine main body image to obtain a vaccine feature vector;

[0032] Step 4: Perform vector retrieval on the vaccine feature vector and match the closest sample vector from the vaccine quality vector library; if successful, the vaccine quality of the matched sample vector is used as the quality of the vaccine to be tested; if unsuccessful, the visual language model is used to perform semantic understanding and reasoning on the vaccine main image to detect the quality of the vaccine to be tested;

[0033] The visual language model adopts the visual language model described in the vaccine quality detection method based on the visual language model as described above, visually encodes the main image to obtain the vaccine appearance image features; guides semantic reasoning through prompt words to enable the model to integrate image features with semantic information, and then performs large language model reasoning to detect the quality of the vaccine to be tested; and adds the feature vector corresponding to the vaccine quality to the vaccine quality vector library.

[0034] Furthermore, in the construction of the vaccine quality vector library in step 1, historical vaccine images including a variety of determined quality conditions are used, and feature vectors are extracted through a feature extraction model constructed based on the ResNet network. Each vaccine quality condition corresponds to a sample vector; each sample vector corresponds to a vaccine quality label.

[0035] Furthermore, the feature extraction model uses an MS-Block module to replace the standard residual module in the ResNet model; and is used to extract features of vaccine images.

[0036] Furthermore, the MS-Block module includes N convolution modules of the same scale, a feature stacking module, a 1×1 convolution module, an SE module and a residual connection module connected in series;

[0037] Among the N convolution modules of the same scale connected in series, the input end of the first convolution module is used as the input end of the MS-Block module to access the original input features, and the output end of each convolution module is respectively connected to the feature stacking module;

[0038] The feature stacking module stacks the features output by each convolution module in descending order of scale to form a pyramid-shaped multi-layer feature set;

[0039] The 1×1 convolution module is connected to the feature stacking module, and a 1×1 convolution kernel is used to perform feature fusion on the multi-layer feature set output by the feature stacking module;

[0040] The SE module is connected to the 1×1 convolution module; it is used to adjust the fused features and sort the importance of feature channels by the learned weights;

[0041] The residual connection module is connected to the input end and the output end of the SE module respectively, and the original input features are merged with the multi-processed features output by the SE module, and output as the output features of the MS-Block module.

[0042] Furthermore, in step 4, a vector search is performed on the feature vector of the vaccine to be tested, and the closest sample vector is matched from the vaccine quality vector library; the Faiss vector search method is used; after the vector search, a similarity score list with each sample vector in the vaccine quality vector library is obtained; and a threshold judgment is performed on the highest score in the similarity score list;

[0043] When the highest score in the similarity score list is greater than the threshold, the match is successful, and the vaccine quality in the vaccine quality label corresponding to the sample vector with the highest score is output as the quality test result of the vaccine to be tested;

[0044] When the highest score in the similarity score list is not greater than the threshold, the match is unsuccessful. The visual language model is used to perform semantic understanding and reasoning on the vaccine main image to detect the quality of the vaccine to be tested; and the feature vector corresponding to the vaccine quality is added to the vaccine quality vector library.

[0045] The present invention can achieve one of the following beneficial effects:

[0046] The vaccine quality detection method based on the visual language model of the present invention fully utilizes the pre-training advantages of the visual language model and can achieve high-precision vaccine quality assessment even when samples are scarce. Especially in long-tail scenarios, it can effectively identify unseen vaccine abnormalities, thereby improving the accuracy and reliability of detection and ensuring the safe use of vaccines.

[0047] To improve the efficiency of reasoning, the method of the present invention embeds images with low similarity into the vector database, calls VLM reasoning only when necessary to avoid repeated calculations, and stores the reasoning results in the database for subsequent queries. This optimization significantly improves the reasoning efficiency and detection speed, making the method more efficient in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0049] Figure 1 This is a flow chart of a vaccine quality detection method based on a visual language model in Embodiment 1 of the present invention;

[0050] Figure 2 It is a conventional YOLOv5 network structure diagram in an embodiment of the present invention;

[0051] Figure 3 This is a diagram of the optimized YOLOv5 network structure in Embodiment 1 of the present invention;

[0052] Figure 4 This is a schematic diagram of the connection of the C3RS module in the first embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of RepResBlock during training in Embodiment 1 of the present invention;

[0054] Figure 6 This is a schematic diagram of the structure of RepResBlock during reasoning in the first embodiment of the present invention;

[0055] Figure 7 This is a flow chart of another vaccine quality detection method based on a visual language model in Embodiment 2 of the present invention;

[0056] Figure 8 This is a conventional ResNet structure diagram in Embodiment 2 of the present invention;

[0057] Fig. 9 This is a diagram of the improved ResNet structure in Embodiment 2 of the present invention;

[0058] Fig.10 This is a schematic diagram of the connection of MS-Block modules with N=5 in the second embodiment of the present invention;

[0059] Fig.11 This is a schematic diagram of the implementation process of the vaccine quality detection method based on the visual language model in Example 2 of the present invention. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used to illustrate the principles of the present invention together with the embodiments of the present invention.

[0061] Embodiment 1

[0062] One embodiment of the present invention discloses a vaccine quality detection method based on a visual language model, such as Figure 1 As shown, including:

[0063] Step S1, obtaining an image of the vaccine to be tested, and obtaining a main body image of the vaccine to be tested after performing vaccine main body detection;

[0064] Step S2, inputting the main image of the vaccine to be tested and the designed prompt words for auxiliary detection into the visual language model;

[0065] Step S3: The visual language model visually encodes the subject image to obtain the vaccine appearance image features; guides semantic reasoning through prompt words to enable the model to integrate the image features with semantic information, and then performs large language model reasoning to detect the quality of the vaccine to be tested.

[0066] Specifically, in step S1, the image data of the vaccine is collected in real time from the vaccination site or the vaccine storage scene to obtain the image of the vaccine to be inspected; during the collection, the appearance image of the vaccine bottle is obtained through a high-definition camera, including the state, color, and sealing condition of the vaccine liquid, etc. The collected image data needs to ensure clarity and stability for subsequent quality inspection.

[0067] In step S1, the position of the vaccine bottle is automatically detected from the image of the vaccine to be tested through vaccine body detection, and the image is cropped to remove irrelevant background interference to obtain the main image of the vaccine to be tested, thereby allowing the visual language model to focus on the area related to the vaccine bottle and improve the recognition rate.

[0068] Preferably, a target detection model based on a convolutional neural network, such as YOLO (You Only Look Once) or Faster R-CNN, is used to detect the vaccine body to achieve better target positioning capabilities.

[0069] Specifically, the process of obtaining the main body image of the vaccine to be tested by the vaccine main body detection includes:

[0070] Step S101, vaccine bottle positioning: the input image is processed by the target detection model to automatically identify and locate the vaccine bottle position in the image;

[0071] The target detection model can be trained with a large number of labeled vaccine bottle images to ensure accurate detection of vaccine bottles in various environments.

[0072] Step S102, border generation and cropping: Generate an accurate border for the vaccine bottle through the target detection model, isolate the area of ​​the vaccine bottle in the image, crop the area, and remove irrelevant background in the image; such as a desktop, wall or other objects.

[0073] Step S103: noise removal: using methods including filtering to remove noise, so that the input image is clear and clean, which helps the recognition of the visual language model.

[0074] By introducing the target detection model, not only can the detection accuracy be improved, but also the background interference can be reduced, and the visual language model's attention to the details of the vaccine bottle can be enhanced, thereby improving the accuracy and efficiency of vaccine anomaly detection.

[0075] In one solution of this embodiment, a target detection model of the YOLOv5 network is adopted.

[0076] YOLOv5 is a target detection network, such as Figure 2 As shown, it includes 4 parts: 1. Input: Perform adaptive scaling, color adjustment and Mosaic data enhancement to enhance the diversity of data set samples. At the same time, an adaptive anchor frame is used to dynamically calculate the optimal value. 2. Backbone network: mainly CSPDarknet53, including Focus module (to ensure that feature information is not lost during downsampling), SPPF module (to extract features of different spatial sizes) and C3 module (to combine deep and shallow features to enhance geometric and semantic information). 3. Neck network: Combine FPN and PANet to ensure that the features transmitted from top to bottom and bottom to top can be effectively fused. 4. Head network: Perform location and classification of feature maps, and use NMS algorithm to optimize the prediction results.

[0077] Specifically, in this embodiment, when detecting the vaccine body, the YOLOv5 network is improved by replacing the C3 module with the C3RS module; the optimized YOLOv5 network is as follows Figure 3 As shown; Figure 3 All C3 modules in the YOLOv5 network are replaced by C3RS modules.

[0078] The C3RS module, such as Figure 4 As shown, it includes a first CBL layer, a second CBL layer, a RepResBlock layer, a concat layer, a SE layer and a third CBL layer;

[0079] Among them, after the first CBL layer and the RepResBlock layer are connected in series, they are connected to the input end of the concat layer together with the second CBL layer, and the output end of the concat layer is sequentially connected to the SE layer and the third CBL layer;

[0080] Among them, during the training and reasoning process of the model, the structure of RepResBlock is respectively Figure 5 and Figure 6 Given in.

[0081] like Figure 5 As shown in the figure, during the training phase, RepResBlock is based on two connected (3×3) convolutional layers CBL of Res-Unit; a standard convolutional branch (1×1) convolutional layer CBL is added in parallel with the latter (3×3) convolutional layer CBL, which expands the receptive field of the model and merges the two branches after calculation to achieve multi-scale fusion of features. This optimization enables the model to capture richer feature information.

[0082] However, the new branch will increase parameters, which may increase the amount of computation and slow down the inference speed. To solve this problem,

[0083] like Figure 6 As shown in Figure 1, during the inference phase, the RepResBlock module undergoes structural reparameterization by padding the standard convolution branch with a convolution and fusing it with the convolution weights of another branch.

[0084] In this way, the model improves its representation capabilities during the training phase without adding additional computational burden during deployment.

[0085] In the solution of this embodiment, the C3RS module, which is a combination of C3, RepResBlock and SE, is used to replace the C3 module in the original YOLOv5; the C3RS module enhances the feature extraction capability of the network by performing multi-scale feature fusion while maintaining the reasoning speed of the original network structure. These modifications enhance its application performance in the vaccine quality detection scenario and provide optimization for the recognition process.

[0086] The prompt words for the auxiliary detection designed in step S2 are a set of text expressions describing vaccine characteristics or abnormal phenomena, which help the model combine image features with semantic information. The prompt words can be "vaccine color change", "whether there is agglomeration", "bottle cap sealing", etc., which form a corresponding relationship with the visual information in the image. The prompt word design needs to be combined with domain knowledge to ensure that the prompt can accurately cover the key points of the vaccine detection task.

[0087] Specifically, the visual language model processing process in step S3 includes:

[0088] Step S301, performing image feature encoding on the main image of the vaccine to be tested input into the visual language model, and extracting visual features including color, texture, and transparency in the image;

[0089] Among them, the visual encoder in the visual language model is a model based on a convolutional neural network (CNN) or Transformer architecture, which is used to extract visual features of an image. Taking ViT or CLIP as an example, the model divides the image into small patches and then converts them into vectorized representations to capture the appearance details of the vaccine bottle, such as color, texture, transparency, etc. The visual encoder extracts high-level semantic features layer by layer through multi-layer convolution or Transformer attention mechanism, such as visual anomalies of sealing status, precipitation, or stratification.

[0090] Step S302: input the designed prompt words into the visual language model to help the model combine image features with semantic information. The prompt words are a set of text expressions describing vaccine features or abnormal phenomena, which are used to help the model combine image features with semantic information, so that the visual language model can accurately understand abnormalities in vaccine images.

[0091] Step S303: Perform a cross-attention fusion on the visual features and the prompt words to align the semantic information of the visual features and the prompt words;

[0092] The cross-modal attention mechanism focuses on the areas in the image related to the prompt word through weight allocation, and automatically filters out irrelevant background noise;

[0093] The fusion process allows the visual language model to process image features and language cues simultaneously, thereby aligning the visual content in the vaccine image with the semantic information of the cue word. The cross-modal attention mechanism focuses on the areas in the image related to the cue word through weight distribution, and automatically filters out irrelevant background noise. For example, the cue word "bottle cap tightness" will guide the model to pay attention to the bottle cap area in the image, while the cue word "color change" will focus on the color characteristics of the vaccine liquid.

[0094] Step S304: performing large language model inference based on the aligned visual features and semantic information of the prompt word;

[0095] The large language model is a language model including GPT that has been pre-trained on a large number of image-text pairs;

[0096] The large language model has been pre-trained on a large number of image-text pairs and has rich common sense and semantic reasoning capabilities. The model can infer whether there are abnormalities in the vaccine based on the input image features and prompt words. For example, the model can infer whether the vaccine has deteriorated by analyzing the "color change" prompt word and the color features in the image, or judge whether the vaccine bottle is leaking based on the "bottle cap tightness" prompt word.

[0097] Step S305: The large language model outputs the result of reasoning based on the fusion features;

[0098] The reasoning results include vaccine abnormalities including color change, agglomeration, precipitation, and stratification, and identify unprecedented abnormalities based on common sense reasoning ability, as well as the reasons for the judgment.

[0099] By outputting the reasons for the judgment, it is easier for doctors or operators to understand the test results.

[0100] Furthermore, the solution of this embodiment also includes fine-tuning and continuous optimization of the visual language model;

[0101] To ensure the efficiency and accuracy of the model in actual use, the system can fine-tune the model through data from real-world scenarios. During the fine-tuning process, the model is retrained using labeled vaccine abnormality images to enhance its ability to detect specific vaccine abnormalities. At the same time, the prompt words can be continuously optimized according to the application scenario to ensure that the model can make accurate judgments when faced with new vaccine bottles or rare abnormalities.

[0102] The entire processing of the visual language model enables the system to efficiently and accurately detect abnormalities in vaccine bottles through image encoding, cross-modal fusion, language model reasoning, and abnormal output. The model not only has excellent generalization capabilities, but can also handle complex long-tail abnormal scenarios through prompt word design and large language model reasoning, ensuring the reliability and coverage of vaccine quality testing.

[0103] In summary, the vaccine quality detection method based on the visual language model fully utilizes the pre-training advantages of the visual language model and can achieve high-precision vaccine quality assessment even when samples are scarce. Especially in long-tail scenarios, it can effectively identify unseen vaccine abnormalities, thereby improving the accuracy and reliability of detection and ensuring the safe use of vaccines.

[0104] Embodiment 2

[0105] One embodiment of the present invention discloses another vaccine quality detection method based on a visual language model, which combines image embedding, vector retrieval and a visual language model to improve reasoning efficiency;

[0106] like Figure 7 As shown, the method includes:

[0107] Step 1, establishing a vaccine quality vector library containing sample vectors of vaccines of different qualities;

[0108] Step 2: Obtain an image of the vaccine to be tested, and obtain a main body image of the vaccine to be tested after performing a vaccine main body test;

[0109] Step 3, extracting features from the vaccine main body image to obtain a vaccine feature vector;

[0110] Step 4: Perform vector retrieval on the vaccine feature vector and match the closest sample vector from the vaccine quality vector library; if successful, the vaccine quality of the matched sample vector is used as the quality of the vaccine to be tested; if unsuccessful, the visual language model is used to perform semantic understanding and reasoning on the vaccine main image to detect the quality of the vaccine to be tested;

[0111] The visual language model adopts the visual language model as described in Example 1 to visually encode the main image to obtain the image features of the vaccine appearance; guide semantic reasoning through prompt words to enable the model to integrate the image features with semantic information, and then perform large language model reasoning to detect the quality of the vaccine to be tested; and add the feature vector corresponding to the vaccine quality to the vaccine quality vector library.

[0112] Specifically, in step 1, in the construction of the vaccine quality vector library, a plurality of historical vaccine images with determined quality conditions are used;

[0113] The historical vaccine images with determined quality conditions are stored in an image database. When constructing a vaccine quality vector library, historical vaccine images are retrieved from the image database and feature vectors are extracted through a feature extraction model constructed based on a ResNet network. Each vaccine quality condition corresponds to a sample vector; and each sample vector corresponds to a vaccine quality condition label.

[0114] ResNet (Residual Network) is a deep convolutional neural network (CNN) architecture and an existing commonly used feature extraction network. ResNet solves the gradient vanishing and degradation problems in deep neural networks by introducing residual modules. These residual modules can be stacked according to task requirements to form ResNet models of various depths, such as ResNet-34, ResNet-50, etc. ResNet has achieved significant performance improvements in multiple computer vision tasks. Due to its excellent performance and easy-to-expand design, ResNet has become an important benchmark in the field of deep learning and is widely used in various computer vision tasks and other fields.

[0115] Furthermore, in order to improve the feature extraction capability of the ResNet model, the solution of this embodiment improves the ResNet model; that is, the feature extraction model in this embodiment uses the MS-Block module to replace the standard residual module in the ResNet model; and is used to extract features of vaccine images.

[0116] The MS-Block module adopts a hierarchical structure to form a multi-level feature extraction pyramid, performs multi-scale convolution on the input features, arranges the multi-scale convolution results in order of scale, and then performs feature fusion and output; when fusing features, an attention mechanism is used to extract the unique features of the vaccine image.

[0117] like Figure 8 The conventional ResNet structure diagram is given; Fig. 9The improved ResNet structure diagram of this embodiment is given; in the improved ResNe network in this embodiment, the MS-Block module is used to replace the ID Block module in each stage after the first stage stage0 in the ResNet network.

[0118] The improved ResNe network not only inherits the advantages of conventional ResNet in dealing with gradient problems in deep networks, but also achieves a qualitative leap in multi-scale expression of features and information integration through MS-Block. This innovative architecture can more accurately extract the unique features of vaccines and enhance the generalization ability of the model, achieving better performance than the traditional ResNet model.

[0119] More specifically, the MS-Block module includes N convolutional modules of the same scale, a feature stacking module, a 1×1 convolutional module, an SE module, and a residual connection module connected in series;

[0120] Among the N convolution modules of the same scale connected in series, the input end of the first convolution module is used as the input end of the MS-Block module to access the original input features, and the output end of each convolution module is respectively connected to the feature stacking module;

[0121] The feature stacking module stacks the features output by each convolution module in descending order of scale to form a pyramid-shaped multi-layer feature set;

[0122] The 1×1 convolution module is connected to the feature stacking module, and a 1×1 convolution kernel is used to perform feature fusion on the multi-layer feature set output by the feature stacking module;

[0123] The SE module is connected to the 1×1 convolution module to further adjust the fused features, sort the importance of feature channels through the learned weights, strengthen or suppress features with different effects, and enhance the network's ability to respond to key information;

[0124] The residual connection module is connected to the input end and the output end of the SE module respectively, and the original input features are merged with the multi-processed features output by the SE module, and output as the output features of the MS-Block module.

[0125] The residual connection module bypasses a series of complex processing steps and directly merges the original input features with the module output; such residual connections not only facilitate the rapid flow of information and alleviate the gradient vanishing problem in deep network training, but also provide a shortcut that enables the network to adapt to the learning of complex functions more quickly during deep learning.

[0126] In a more preferred solution, each of the N convolution modules of the same scale uses a 3×3 convolution kernel;

[0127] The number of convolution modules is determined according to the specific computing resources.

[0128] like Fig.10 A schematic diagram of the connection of the MS-Block module with N=5 is given.

[0129] By connecting five 3×3 convolution kernels in series, multi-scale features such as 3×3, 5×5, 7×7, 9×9, and 11×11 are formed. In the feature stacking module, they are stacked from large to small scales to form a pyramid-shaped multi-layer feature set.

[0130] The feature extraction experiment on the vaccine image proved that the MS-Block module of this embodiment achieved a balance between reasoning speed and accuracy. This balance is achieved by carefully designing the number of convolutional layers, the size of the convolution kernel, and the comprehensive configuration of the attention mechanism. Nevertheless, the design of the MS-Block module has a certain flexibility, allowing the number and type of convolutional layers to be adjusted according to specific application scenarios, as well as the replacement of SE modules to adapt to different requirements for performance and accuracy. The versatility and configurability of this design make MS-Block not only limited to vaccine feature extraction scenarios, but also widely applicable to other image recognition tasks, demonstrating its wide application potential.

[0131] Specifically, in step 2, the same method as in Example 1 is used to obtain an image of the vaccine to be tested, and the main body image of the vaccine to be tested is obtained after the vaccine main body is tested.

[0132] Specifically, when extracting features from the main image of the vaccine to be tested in step 3, the same feature extraction model as that used to establish the vaccine quality vector library in step 1 is used to obtain the feature vector of the vaccine to be tested.

[0133] Specifically, in step 4, a vector search is performed on the feature vector of the vaccine to be tested, and the closest sample vector is matched from the vaccine quality vector library; the Faiss (Fast Approximate Nearest Neighbor Search Library) vector search method is used; after the vector search, a similarity score list with each sample vector in the vaccine quality vector library is obtained; and a threshold judgment is performed on the highest score in the similarity score list;

[0134] When the highest score in the similarity score list is greater than the threshold, the match is successful, and the vaccine quality in the vaccine quality label corresponding to the sample vector with the highest score is output as the quality test result of the vaccine to be tested;

[0135] When the highest score in the similarity score list is not greater than the threshold, the match is unsuccessful, and the visual language model is used to perform semantic understanding and reasoning on the vaccine subject image to detect the quality of the vaccine to be tested;

[0136] The visual language model adopts the visual language model as described in Example 1 to visually encode the main image to obtain the image features of the vaccine appearance; guide semantic reasoning through prompt words to enable the model to integrate the image features with semantic information, and then perform large language model reasoning to detect the quality of the vaccine to be tested; and add the feature vector corresponding to the vaccine quality to the vaccine quality vector library.

[0137] More specifically, when performing vector retrieval, the Euclidean distance between the feature vector of the vaccine to be tested and each sample vector is calculated, and the similarity score is calculated according to the size of the Euclidean distance. The smaller the Euclidean distance, the higher the score.

[0138] The actual process diagram in this embodiment is as follows Fig.11 As shown, in the embodiment, by combining image embedding and vector retrieval technology, the reasoning process of the visual language model is optimized. By embedding images with higher similarity and storing them in the vector database, when images with lower similarity are detected, the VLM (visual language model) is called for reasoning, thereby reducing repeated calculations. The reasoning results of the VLM will also be stored in the database for subsequent query use. Through this mechanism, the system significantly reduces the occupancy of computing resources, improves reasoning efficiency, and speeds up detection, so that the technology can better adapt to real-time detection needs in practical applications.

[0139] In summary, the vaccine quality detection method based on the visual language model of the embodiment of the present invention has the following beneficial effects:

[0140] 1. Improve the accuracy and generalization of vaccine quality testing: Traditional vaccine quality testing mainly relies on manual judgment or rule-based machine learning methods, which are difficult to deal with complex or rare abnormal situations. Through the pre-trained visual language model, combined with visual features and common sense reasoning, not only the accuracy of detecting abnormalities in vaccine bottles is improved, but also the generalization ability of the model is enhanced. When dealing with long-tail scenarios and scarce samples, the system can identify rare anomalies through reasoning, provide more accurate detection results, reduce the probability of missed detection and misjudgment, and ensure reliable application in various scenarios.

[0141] 2. Efficient reasoning performance: While improving the accuracy and generalization ability of vaccine quality detection, the high computational complexity problem in the visual language model reasoning process has been optimized. By introducing image embedding and vector retrieval technology, the system can quickly compare the detected image with the image stored in the vector database, and only call the visual language model for reasoning when the similarity is low. The reasoning results will also be stored in the database for subsequent queries to avoid repeated calculations. This mechanism greatly improves the reasoning efficiency and reduces the consumption of computing resources, allowing the system to achieve faster response speeds while maintaining high detection accuracy, which is suitable for real-time detection needs.

[0142] 3. Improve detection efficiency and reduce labor costs: Traditional manual detection methods are inefficient and easily affected by subjective factors. The present invention greatly improves detection efficiency and reduces dependence on manual operators through an automated vaccine quality detection process. It not only improves work efficiency, but also saves a lot of labor costs, ensuring that the detection process is more stable and reliable.

[0143] 4. Improve vaccine safety and recipients’ trust: Through the efficient detection of the present invention, potential vaccine problems can be quickly screened to ensure vaccine safety during the vaccination process. This not only protects public health, but also helps to enhance recipients’ trust in the vaccination system and promote the widespread and safe use of vaccines.

[0144] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A vaccine quality detection method based on a visual language model, characterized in that: include: Step S1, obtaining an image of the vaccine to be tested, and obtaining a main body image of the vaccine to be tested after performing vaccine main body detection; Step S2, inputting the main image of the vaccine to be tested and the designed prompt words for auxiliary detection into the visual language model; Step S3, the visual language model visually encodes the subject image to obtain vaccine appearance image features; By guiding semantic reasoning through prompt words, the model can integrate image features with semantic information, and then perform large language model reasoning to detect the quality of the vaccine to be tested.

2. The vaccine quality detection method based on the visual language model according to claim 1 is characterized in that: In step S1, the process of obtaining the main body image of the vaccine to be tested by detecting the main body of the vaccine includes: Step S101, vaccine bottle positioning: the input image is processed by the target detection model to automatically identify and locate the vaccine bottle position in the image; Step S102, border generation and cropping: Generate an accurate border for the vaccine bottle through the target detection model, isolate the area of ​​the vaccine bottle in the image, crop the area, and remove irrelevant background in the image; Step S103: noise removal: using methods including filtering to remove noise, so that the input image is clear and clean, which helps the recognition of the visual language model.

3. The vaccine quality detection method based on visual language model according to claim 2 is characterized in that: The target detection model uses the YOLOv5 network or Faster R-CNN.

4. The vaccine quality detection method based on visual language model according to claim 3 is characterized in that: In the YOLOv5 network, the C3RS module is used to replace the C3 module; The C3RS module includes a first CBL layer, a second CBL layer, a RepResBlock layer, a concat layer, a SE layer and a third CBL layer; wherein the first CBL layer is the input layer of the C3RS module, and after the first CBL layer and the RepResBlock layer are connected in series, they are connected to the input end of the concat layer together with the second CBL layer, and the output end of the concat layer is sequentially connected to the SE layer and the third CBL layer; the third CBL layer serves as the output layer of the C3RS module.

5. The vaccine quality detection method based on visual language model according to claim 3 is characterized in that: The visual language model processing process in step S3 includes: Step S301, performing image feature encoding on the main image of the vaccine to be tested input into the visual language model, and extracting visual features including color, texture, and transparency in the image; Step S302: input the designed prompt words into the visual language model to help the model combine image features with semantic information; The prompt words are a set of textual expressions describing vaccine characteristics or abnormal phenomena, which are used to help the model combine image features with semantic information, so that the visual language model can accurately understand abnormalities in vaccine images; Step S303: fusing the visual features and the prompt words using a cross-modal attention mechanism to align the semantic information of the visual features and the prompt words; The cross-modal attention mechanism focuses on the areas in the image related to the prompt word through weight allocation, and automatically filters out irrelevant background noise; Step S304: performing large language model inference based on the aligned visual features and semantic information of the prompt word; The large language model is a language model including GPT that has been pre-trained on a large number of image-text pairs; Step S305: The large language model outputs the result of reasoning based on the fusion features; The reasoning results include vaccine abnormalities including color change, agglomeration, precipitation, and stratification, and identify unprecedented abnormalities based on common sense reasoning ability, as well as the reasons for the judgment.

6. A vaccine quality detection method based on a visual language model, characterized in that: include: Step 1, establishing a vaccine quality vector library containing sample vectors of vaccines of different qualities; Step 2: Obtain an image of the vaccine to be tested, and obtain a main body image of the vaccine to be tested after performing a vaccine main body test; Step 3, extracting features from the vaccine main body image to obtain a vaccine feature vector; Step 4: Perform vector retrieval on the vaccine feature vector and match the closest sample vector from the vaccine quality vector library; if successful, the vaccine quality of the matched sample vector is used as the quality of the vaccine to be tested; if unsuccessful, the visual language model is used to perform semantic understanding and reasoning on the vaccine main image to detect the quality of the vaccine to be tested; The visual language model adopts the visual language model described in the vaccine quality detection method based on the visual language model according to any one of claims 1 to 5, and visually encodes the subject image to obtain vaccine appearance image features; By guiding semantic reasoning through prompt words, the model can integrate image features with semantic information, and then perform large language model reasoning to detect the quality of the vaccine to be tested; and add the feature vector corresponding to the vaccine quality to the vaccine quality vector library.

7. The vaccine quality detection method based on visual language model according to claim 6 is characterized in that: In the construction of the vaccine quality vector library in step 1, historical vaccine images with various determined quality conditions are used, and feature vectors are extracted through a feature extraction model built based on the ResNet network. Each vaccine quality condition corresponds to a sample vector; each sample vector corresponds to a vaccine quality label.

8. The vaccine quality detection method based on visual language model according to claim 7 is characterized in that: The feature extraction model uses the MS-Block module to replace the standard residual module in the ResNet model; and is used to extract features of vaccine images.

9. The vaccine quality detection method based on visual language model according to claim 8 is characterized in that: The MS-Block module includes N convolution modules of the same scale, a feature stacking module, a 1×1 convolution module, an SE module and a residual connection module connected in series; Among the N convolution modules of the same scale connected in series, the input end of the first convolution module is used as the input end of the MS-Block module to access the original input features, and the output end of each convolution module is respectively connected to the feature stacking module; The feature stacking module stacks the features output by each convolution module in descending order of scale to form a pyramid-shaped multi-layer feature set; The 1×1 convolution module is connected to the feature stacking module, and a 1×1 convolution kernel is used to perform feature fusion on the multi-layer feature set output by the feature stacking module; The SE module is connected to the 1×1 convolution module; it is used to adjust the fused features and sort the importance of feature channels by the learned weights; The residual connection module is connected to the input end and the output end of the SE module respectively, and the original input features are merged with the multi-processed features output by the SE module, and output as the output features of the MS-Block module.

10. The vaccine quality detection method based on visual language model according to claim 9 is characterized in that: In step 4, a vector search is performed on the feature vector of the vaccine to be tested, and the closest sample vector is matched from the vaccine quality vector library; the Faiss vector search method is used; after the vector search, a similarity score list with each sample vector in the vaccine quality vector library is obtained; Perform threshold judgment on the highest score in the similarity score list; When the highest score in the similarity score list is greater than the threshold, the match is successful, and the vaccine quality in the vaccine quality label corresponding to the sample vector with the highest score is output as the quality test result of the vaccine to be tested; When the highest score in the similarity score list is not greater than the threshold, the match is unsuccessful, and the visual language model is used to perform semantic understanding and reasoning on the vaccine subject image to detect the quality of the vaccine to be tested; And the feature vector corresponding to the vaccine quality is added to the vaccine quality vector library.

Images