Medical marketing monitoring method and system based on image recognition
The system uses image recognition and optical character recognition to accurately identify and monitor medical staff and pharmaceuticals in hospitals, addressing inefficiencies in traditional management and enhancing decision-making.
Patent Information
- Application Number
- CN202510238537.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-03
- Publication Date
- 2025-07-15
AI Technical Summary
The traditional manual management model is difficult to meet the needs of efficient management and precise decision-making of large comprehensive hospitals with numerous departments, complex personnel, diverse equipment and numerous drugs. How to achieve accurate identification and monitoring of medical staff and drugs has become an important topic.
Using an image recognition method, we use the method of extracting the appearance and clothing characteristics of medical staff, combining Pap distance calculation to identify different medical staff and their departments and jobs, and through optical character recognition, we extract key information from the medical device identification nameplate to monitor the integrity, accuracy and consistency of drugs, and use convolutional neural networks to extract and classify features to achieve real-time monitoring of medical staff and drugs.
Accurate identification of medical staff is achieved, avoid identity confusion, reduce the risk of medical accidents, ensure orderly drug management, improve drug use accuracy, and reduce the problem of misuse of drugs.
Smart Images
Figure CN120319418A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and particularly relates to a method and system for pharmaceutical marketing monitoring based on image recognition. Background Art
[0002] With the rapid development of medical technology, hospital management is gradually moving towards informatization and intelligence. While improving the quality and efficiency of medical services, how to efficiently and accurately collect and analyze various types of data within the hospital has become an important issue in modern medical management. Especially in large general hospitals, there are numerous departments, complex personnel, diverse equipment, and a large number of drugs. The traditional manual management mode has been difficult to meet the needs of efficient management and precise decision-making. Therefore, it is particularly important to use advanced Internet of Things technology, edge computing, and deep learning algorithms to build an intelligent system that can obtain and analyze comprehensive data of each department in the hospital in real time. Summary of the Invention
[0003] In view of the technical problems existing in the prior art, the present invention provides a method and system for pharmaceutical marketing monitoring based on image recognition.
[0004] The technical solution of the present invention to solve the above technical problems is as follows: A method for pharmaceutical marketing monitoring based on image recognition, comprising the following steps:
[0005] S101. By identifying the image data of medical representatives and medical staff, extract the appearance features and clothing features of medical representatives and medical staff, to help identify different medical representatives, medical staff, and the departments and job positions of the pharmaceutical companies and medical staff to which their medical representatives belong;
[0006] S102. For the annotation information of drug images, extract the features of integrity, accuracy, and consistency of the annotation, and classify the drugs according to the annotation information;
[0007] S103. Use optical character recognition to extract key information from the medical device identification nameplate, obtain the image features of medical staff, combine them into feature vectors, and determine their affiliated departments in combination with the calculation of the Bhattacharyya distance. When medical staff pick up drugs, operate medical devices, or participate in departmental academic activities,
[0008] Check whether the department to which they belong matches the department of the medical staff. If not, an alarm is issued, so as to achieve accurate identification of medical staff and effective monitoring of medical devices and drugs;
[0009] Meanwhile, determine which pharmaceutical representative's marketed drugs and which pharmaceutical company they are according to the drugs taken by medical staff. In a preferred embodiment, in S01, the scenarios of each department in the hospital are monitored in real time to obtain comprehensive influence data, and the collected influence data is preprocessed, including image enhancement and denoising processing. Using a convolutional neural network, key information analysis and feature extraction are performed on the appearance features and clothing features of medical staff in the images. The specific steps for extracting the appearance features of medical staff are as follows:
[0010] S1. Gray-scale distribution feature: Convert the image containing the face of the medical staff collected into a grayscale image, traverse each pixel point of the image, count the frequencies of different gray-scale values, and thus obtain a gray-scale histogram. Let the image size be M×N, the gray-level be L, and the number of pixels with gray-scale value i be n i , then the frequency p i at which the gray-scale value i appears is calculated as follows:
[0011]
[0012] where i = 0, 1,..., L - 1. By calculating the mean and variance of p i to characterize the gray-scale distribution feature, the specific formula for calculating the mean is as follows:
[0013]
[0014] where μ represents the mean of the gray-scale distribution, which describes the overall average gray-scale level of the face. The specific formula for calculating the variance is as follows:
[0015]
[0016] where σ 2 represents the variance of the gray-scale distribution, which reflects the degree of dispersion of the gray-scale values relative to the mean, helps to identify the differences in the facial skin colors of different individuals, and the stability under different lighting conditions;
[0017] S2. Facial edge sharpness feature: Perform edge detection on the facial image of the medical staff by using an edge detection operator, calculate the gradient amplitude and direction of the edge pixels, and evaluate the facial edge sharpness based on the distribution of the gradient amplitude. Apply convolution kernels in the horizontal and vertical directions respectively for convolution operations to obtain the horizontal direction gradient G x and the vertical direction gradient G y . For each pixel (x, y) in the image, the specific formula for the horizontal direction gradient G x is as follows:
[0018]
[0019] Among them, x m，n represents the pixel value within the (x, y) neighborhood, and w x，m，n represents the Sobel convolution kernel coefficients in the horizontal direction. The specific calculation formula for the vertical direction gradient G y is as follows:
[0020]
[0021] Among them, w y，m，n represents the Sobel convolution kernel coefficients in the vertical direction. The gradient magnitude is calculated based on the horizontal direction gradient and the vertical direction gradient. The specific calculation formula is as follows:
[0022]
[0023] Among them, G represents the gradient magnitude, which synthesizes the gradient information in the horizontal and vertical directions and is used to represent the edge intensity at the pixel point. The mean and variance of the gradient magnitude G are statistically calculated as the quantization indicators for edge sharpness. The specific calculation formula for the mean of the gradient magnitude is as follows:
[0024]
[0025] Among them, μ G represents the mean of the gradient magnitude. The specific calculation formula for the variance of the gradient magnitude is as follows:
[0026]
[0027] Among them, represents the variance of the gradient magnitude;
[0028] S3. Clothing feature extraction: For the appearance features of medical staff and seasonal clothing, extract clothing features, including the color, style, and material features of the clothing. Based on the clothing features, help identify different medical staff and the departments and job positions to which the medical staff belong. Use the edge detection algorithm to extract its contour point set P = {p1, p2,..., p n}, where p i = (L xi , L yi ) is the point coordinate on the contour. For each contour point p i , construct a log-polar coordinate space centered on it. Divide the polar coordinate space into N r radial intervals and N θ angular intervals to form N = N r ×N θ bins. Calculate the distribution of all other contour points except p i in this log-polar coordinate space to obtain the shape context descriptor S i for each point p i, for any point p on the contour except p i other than j =(L xj , L yj ), a logarithmic polar coordinate space is constructed with p i =(L xi , L yi ) as the center. The specific calculation formulas for its logarithmic polar coordinates (r ij , θ ij ) are as follows:
[0029]
[0030] where p j represents any point on the contour except p i , r ij represents the logarithmic distance from point p j to point p i , θ ij represents the angle of point p j relative to point p i , L xi , L yi respectively represent the abscissa and ordinate of the selected center point p i in the Cartesian coordinate system, L xj , L yj respectively represent the abscissa and ordinate of any point p i other than p j on the contour in the Cartesian coordinate system. For each point p j , its corresponding bin is determined according to its logarithmic polar coordinates (r ij , θ ij ). Let the number of points falling into the k-th bin be n ik , then the shape context descriptor S i of point p i can be expressed as S i =[n i1 , n i2 ,..., n iN . The Bhattacharyya distance is used to compare the similarity of corresponding points on two contours. The specific calculation formula for the Bhattacharyya distance is as follows:
[0031]
[0032] where S a , S b respectively represent the shape context descriptors of corresponding points on two different contours for comparing their similarity, S ak represents the k-th element of the shape context descriptor S a , S bkDenote the k-th element of the shape context descriptor S b With a small Bhattacharyya distance, it indicates that the shapes of the two contours are similar, and the shapes of the clothing styles at the key parts are close. Based on the calculation of the Bhattacharyya distance, the extracted clothing style features can be associated with the department and job information of the medical staff, which is used to identify different medical staff and their affiliated departments and job positions.
[0033] In a preferred embodiment, in S102, according to the drug information list, which covers drug name, specification, manufacturer, approval number, and expiration date, check one by one whether the drug image annotation contains all items in the list to define the integrity index. If all key items are annotated, the integrity index is set to full marks. If there are J n key items not annotated, and the total number of key items is J, the specific calculation formula of the integrity index is as follows:
[0034]
[0035] Precisely compare the position of the annotated drug information with the position of the actual drug information in the image. Using the image masking technology, make a mask image for the annotated drug information area and the area of the actual drug information in the image. Let the mask of the annotated drug name area be M a , and the actual area mask be M r , then the specific calculation formula of the overlapping area is as follows:
[0036]
[0037] where A overlap represents the overlapping area, the area of the annotated area the area of the actual area The annotation accuracy index is obtained by calculating the ratio of the overlapping area to the maximum value of the annotated area and the actual area. The specific calculation formula of the accuracy index is as follows:
[0038]
[0039] I a value close to 1 indicates high accuracy of the annotation result, that is, the matching degree between the position of the annotation information in the image and the position of the actual drug information is good. For the case of multiple annotations for the same image, check the consistency of the description of the drug key information and the position annotation between different annotations. By comparing the key information of different annotations, judge whether there is a conflict in the annotation. Suppose there are B n annotations, and for the key information, the number of pairs of inconsistent annotations is counted as B m , then the specific calculation formula of the consistency index is as follows:
[0040]
[0041] Among them I co The value close to 1 indicates better annotation consistency and high reliability of annotation information;
[0042] According to the annotation information of drugs, corresponding uses and efficacy categories are labeled as tags for each drug, and the data set is divided into a training set, a validation set and a test set. Feature extraction is performed on the drug annotation information, including keywords in the drug name and the content of active ingredients in the specifications, and the extracted data is standardized. The drug annotation information after feature extraction and standardization is used as the input of the model, and the drug annotation information is associated with the corresponding drug uses and efficacy, and a training data set with labels is constructed. Then the prediction model for drug classification specifically includes the following steps:
[0043] S1. Input the training set data into the CNN model, and obtain the prediction results of drug uses and efficacy categories through forward propagation. In the forward propagation process, the input data passes through the convolutional layer, activation layer, pooling layer and fully connected layer in sequence, extracts and processes the features in the drug annotation information, and obtains the prediction probability values corresponding to each drug use and efficacy category at the output layer;
[0044] S2. Calculate the difference between the prediction result and the true label according to the loss function. The drug use and efficacy classification belongs to a multi-classification problem, and the cross-entropy loss function is used for calculation. The specific calculation formula is as follows:
[0045]
[0046] where, model i represents the probability that the model predicts to belong to category i, y i represents the true category label, and L represents the cross-entropy loss function;
[0047] S3. Through the backpropagation algorithm, calculate the gradient of the loss with respect to each model parameter;
[0048] S4. Update the model parameters, and continuously iterate to minimize the loss function value, so that the prediction result of the model is closer to the true label;
[0049] S5. After each training cycle ends, use the validation set to evaluate the model performance to prevent overfitting.
[0050] In a preferred embodiment, in S103, optical character recognition technology and image recognition algorithms are used to extract the identification and nameplate on the medical device, recognize the text, numbers, and symbols in the identification to obtain the model, brand, production date, and usage instruction information of the medical device. At the same time, the color, position, and shape features of the identification are extracted to assist in identifying different medical devices, and the image features of the medical staff are obtained. The appearance features and clothing features are extracted from the image and combined into a feature vector f p =[f1, f2,..., f n , where f i represents the appearance features and clothing features. Let the feature vector of drug Y be f Y =[Y1, Y2,..., Y k , where Y i represents the type and use of the drug. Let the feature vector of medical device Q be f Q =[Q1, Q2,..., Q l , where Q i represents the type and use of the device. According to the calculation of the Bhattacharyya distance, the department K i to which the medical staff H belongs is obtained. The drugs and medical devices are matched by department. Let the department matching function of drug Y be C Y (Y), which returns the department to which drug Y belongs. Let the department matching function of device Q be C Q (Q), which returns the department to which device Q belongs. When the medical staff H picks up drug Y and device Q, it is checked whether C Y (Y)≠K i and C Q (Q)≠K i . If the above conditions are met, an alarm is issued. Through the above calculations, the identification of medical staff, the monitoring of medical devices and drugs can be realized, and a warning can be issued in time when an abnormal situation is detected.
[0051] The embodiment of the present invention also provides an image recognition-based pharmaceutical marketing monitoring system, including:
[0052] Feature recognition module: By recognizing the image data of medical staff, the appearance features and clothing features of medical staff are extracted to help identify different medical staff and their affiliated departments and job positions;
[0053] Drug feature extraction module: For the annotation information of drug images, the integrity, accuracy, and consistency features of the annotation are extracted, and the drugs are classified according to the annotation information;
[0054] Abnormal monitoring module: Using optical character recognition, extract key information from the medical device identification nameplate, obtain the image features of medical staff, combine them into a feature vector, and determine their affiliated department in combination with the calculation of the Bhattacharyya distance. When medical staff pick up drugs or devices, check whether the affiliated department matches the department of the medical staff. If not, an alarm is issued, so as to achieve accurate identification of medical staff and effective monitoring of medical devices and drugs.
[0055] The beneficial effects of the present invention are as follows: Identify medical staff based on appearance and clothing features, can accurately distinguish different personnel, avoid identity confusion, facilitate management and work assignment, associate medical staff with their affiliated departments and positions, can monitor the behavior of medical staff picking up devices and drugs in real time, discover illegal operations and give warnings in time, reduce the risk of medical accidents, and ensure patient safety. Classify according to the departments of medical staff and the types of drugs, etc., to make drug management more orderly. By matching departments and drugs, it is possible to avoid medical staff from misappropriating drugs, improve the accuracy of drug use, and reduce problems caused by improper drug use. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 is the flowchart of the method of the present invention;
[0057] Figure 2 is the system block diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0058] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.
[0059] In the description of the present application, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.
[0060] In the description of the present application, the term "for example" is used to mean "serving as an example, illustration, or explanation". Any embodiment described as "for example" in the present application is not necessarily construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but rather to be in line with the broadest scope consistent with the principles and features disclosed in the present application.
[0061] As Figure 1 , this embodiment provides: A pharmaceutical marketing monitoring method based on image recognition, specifically including the following steps:
[0062] S101. By recognizing the image data of pharmaceutical representatives and medical staff, extract the appearance features and clothing features of pharmaceutical representatives and medical staff to help identify different pharmaceutical representatives, medical staff, as well as the departments and job positions of the pharmaceutical companies and medical staff to which the pharmaceutical representatives belong;
[0063] Furthermore, conduct real-time monitoring on the scenes of each department in the hospital, obtain comprehensive impact data, preprocess the collected impact data, including image enhancement and denoising processing, and use a convolutional neural network to analyze and extract key information on the appearance features and seasonal clothing wear of medical staff and pharmaceutical representatives in the images, and extract features related to their standardization based on the physical features of the images. The specific steps for extracting the appearance features of medical staff are as follows:
[0064] S1. Gray-scale distribution feature: Convert the collected images containing the faces of medical staff and pharmaceutical representatives into grayscale images, traverse each pixel point of the image, count the frequencies of different gray-scale values, and thus obtain a gray-scale histogram. Assume the image size is M×N, the gray-scale level is L, and the number of pixels with gray-scale value i is n i , then the frequency p i of the gray-scale value i appears is calculated as follows:
[0065]
[0066] where i = 0, 1,..., L - 1. By calculating the mean and variance of p i , the gray-scale distribution feature is characterized. The specific calculation formula for the mean is as follows:
[0067]
[0068] Among them, μ represents the mean of the grayscale distribution, which describes the overall average grayscale level of the face. The specific calculation formula for the variance is as follows:
[0069]
[0070] Among them, σ 2 represents the variance of the grayscale distribution, reflecting the degree of dispersion of the grayscale values relative to the mean, which helps to identify the differences in the facial skin colors of different individuals and the stability under different lighting conditions;
[0071] S2. Facial edge sharpness feature: By using an edge detection operator to perform edge detection on the facial images of medical staff and pharmaceutical representatives, calculate the gradient magnitude and direction of the edge pixels, and evaluate the facial edge sharpness based on the distribution of the gradient magnitude. Apply convolution kernels in the horizontal and vertical directions respectively for convolution operations to obtain the horizontal direction gradient G x and the vertical direction gradient G y . For each pixel (x, y) in the image, the specific calculation formula for the horizontal direction gradient G x is as follows:
[0072]
[0073] Among them, x m，n represents the pixel value within the neighborhood of (x, y), w x，m，n represents the Sobel convolution kernel coefficient in the horizontal direction. The specific calculation formula for the vertical direction gradient G y is as follows:
[0074]
[0075] Among them, w y，m，n represents the Sobel convolution kernel coefficient in the vertical direction. Calculate the gradient magnitude based on the horizontal direction gradient and the vertical direction gradient. The specific calculation formula is as follows:
[0076]
[0077] Among them, G represents the gradient magnitude, which combines the gradient information in the horizontal and vertical directions and is used to represent the edge intensity at the pixel point. The mean and variance of the gradient magnitude G are statistically used as the quantization indexes of the edge sharpness. The specific calculation formula for the mean of the gradient magnitude is as follows:
[0078]
[0079] Among them, μ G represents the mean of the gradient magnitude. The specific calculation formula for the variance of the gradient magnitude is as follows:
[0080]
[0081] Among them, represents the variance of the gradient magnitude;
[0082] It should be noted that by calculating the mean and variance of the gradient magnitude, we can quantitatively evaluate the edge sharpness of the medical staff's face. The mean can let us understand the average level of the edge strength, while the variance reflects the variation of the face edge strength in the image. If the variance is large, it means that the distribution of the face edge strength is relatively dispersed, which may indicate that the face contains edge features of different intensities, such as large differences in the face contour and facial feature contours. If the variance is small, it means that the face edge strength is relatively uniform, and the edges of the face may be relatively simple and clear.
[0083] S3. Clothing feature extraction: For the appearance features of medical staff and pharmaceutical representatives and their seasonal clothing, extract clothing features, including the color, style, and material features of the clothing. Based on the clothing features, help identify different medical staff and the departments and job positions to which the medical staff belong. Use the edge detection algorithm to extract its contour point set P = {p1, p2,..., p n}, where p i = (L xi , L yi ) is the coordinate of the point on the contour. For each contour point p i , construct a log-polar coordinate space centered on it, divide the polar coordinate space into N r radial intervals and N θ angular intervals to form N = N r ×N θ bins. Calculate the distribution of all other contour points except p i in this log-polar coordinate space to obtain the shape context descriptor S i of each point p i . For any point p i except p j = (L xj , L yj ) on the contour, construct a log-polar coordinate space centered on p i = (L xi , L yi ). The specific calculation formula of its log-polar coordinates (r ij , θ ij ) is as follows:
[0084]
[0085] Among them, p j represents any point on the contour except p i , and r ij represents the point pj The logarithmic distance to point p i , θ ij represents the angle of point p j relative to point p i , L xi , L yi respectively represent the abscissa and ordinate of the selected central point p on the contour i in the Cartesian coordinate system, L xj , L yj respectively represent the abscissa and ordinate of any point p other than p i on the contour j in the Cartesian coordinate system. For each point p j , according to its log-polar coordinates (r ij , θ ij ), determine the bin it falls into. Let the number of points falling into the k-th bin be n ik , then the shape context descriptor S i of point p i can be expressed as S i = [n i1 , n i2 ,..., n iN . Use the Bhattacharyya distance to compare the similarity of corresponding points on two contours. The specific calculation formula of the Bhattacharyya distance is as follows:
[0086]
[0087] where S a , S b respectively represent the shape context descriptors of corresponding points on two different contours, used to compare the similarity between them. S ak represents the k-th element of the shape context descriptor S a , S bk represents the k-th element of the shape context descriptor S b . The smaller the Bhattacharyya distance, the more similar the shapes of the two contours, and the closer the shapes of the clothing styles at the key parts. Based on the calculation of the Bhattacharyya distance, the extracted clothing style features can be associated with the department and job position information of medical staff, used to identify different medical staff and their affiliated departments and job positions.
[0088] S102. For the annotation information of drug images, extract the integrity, accuracy, and consistency features of the annotations, and classify the drugs according to the annotation information;
[0089] Further, according to the drug information list, which covers drug name, specification, manufacturer, approval number, and expiration date, check one by one whether the drug image annotation contains all items in the list to define the integrity index. If all key items are annotated, the integrity index is set to full marks. If there are J n unannotated key items and the total number of key items is J, the specific calculation formula for the integrity index is as follows:
[0090]
[0091] Precisely compare the position of the annotated drug information with the position of the actual drug information in the image. Using image masking technology, create a mask image for the annotated drug information area and the area of the actual drug information in the image. Let the mask of the annotated drug name area be M a , and the actual area mask be M r , then the specific calculation formula for the overlapping area is as follows:
[0092]
[0093] Among them, A overlap represents the overlapping area, the area of the annotated area the area of the actual area The annotation accuracy index is obtained by calculating the ratio of the overlapping area to the maximum value of the annotated area and the actual area. The specific calculation formula for the accuracy index is as follows:
[0094]
[0095] When the value of I a is close to 1, it indicates that the annotation result has high accuracy, that is, the matching degree between the position of the annotation information in the image and the position of the actual drug information is good. For the case where there are multiple annotations for the same image, check the consistency of the description of the drug key information and the position annotation between different annotations. By comparing the key information of different annotations, determine whether there are conflicts in the annotations. Suppose there are B n annotations, and for the key information, the number of pairs of inconsistent annotations is counted as B m , then the specific calculation formula for the consistency index is as follows:
[0096]
[0097] Among them When the value of I co is close to 1, it indicates that the annotation has good consistency and the reliability of the annotation information is high;
[0098] According to the labeling information of drugs, corresponding uses and efficacy categories are labeled as tags for each drug. The dataset is divided into a training set, a validation set, and a test set. Feature extraction is performed on the drug labeling information, including keywords in the drug name and the content of active ingredients in the specifications. The extracted data is standardized. The drug labeling information after feature extraction and standardization is used as the input of the model. The drug labeling information is associated with the corresponding drug uses and efficacy to construct a training dataset with labels. The prediction model for drug classification specifically includes the following steps:
[0099] S1. Input the training set data into the CNN model, and obtain the prediction results of drug uses and efficacy categories through forward propagation. During the forward propagation process, the input data passes through the convolutional layer, activation layer, pooling layer, and fully connected layer in sequence to extract and process the features in the drug labeling information, and obtain the prediction probability values corresponding to each drug use and efficacy category at the output layer;
[0100] S2. Calculate the difference between the prediction results and the true labels according to the loss function. The classification of drug uses and efficacy belongs to a multi-classification problem, and the cross-entropy loss function is used for calculation. The specific calculation formula is as follows:
[0101]
[0102] where, model i represents the probability that the model predicts to belong to category i, y i represents the true category label, and L represents the cross-entropy loss function;
[0103] S3. Calculate the gradient of the loss with respect to each model parameter through the backpropagation algorithm;
[0104] S4. Update the model parameters, and continuously iterate to minimize the loss function value, so that the prediction results of the model are closer to the true labels;
[0105] S5. After each training cycle, use the validation set to evaluate the model performance to prevent overfitting.
[0106] It should be noted that the annotation information of drug images mainly includes the annotation of drug basic information, the annotation of ingredient content, and the annotation of specification parameters. The annotation of drug basic information mainly marks and records the key basic information visible in the drug image. These annotations usually cover information such as drug name, manufacturer, approval number, etc., and clarify the location and clear range of the information. The annotation of ingredient content is the annotation of the ingredients contained in the drug and their content. The annotation generally indicates the specific name of each ingredient, the proportion it occupies, and the content of the active ingredient. The annotation of specification parameters mainly annotates parameters such as the dosage form, packaging specification, and single dose of the drug, and details and records relevant specification data such as the size, weight, and volume of the drug. Complete annotation can ensure the comprehensiveness of information in the drug image dataset and greatly enhance the value of the data. When constructing a drug database for drug research and development, quality control, or training of an intelligent drug recognition system, complete annotation can effectively avoid system deviation or inaccurate research results caused by data missing. For example, in the training of an intelligent drug recognition system, if the manufacturer annotation in the drug basic information is missing, it may cause confusion when the system identifies the same drug produced by different manufacturers and affect the recognition effect.
[0107] S103. Use optical character recognition to extract key information from the medical device identification nameplate, obtain the image features of medical staff, combine them into a feature vector, and determine their affiliated department by calculating the Bhattacharyya distance. When medical staff pick up drugs or devices, check whether the affiliated department matches the department of the medical staff. If not, an alarm is issued, so as to achieve the accurate identification of medical staff and the effective monitoring of medical devices and drugs.
[0108] Furthermore, use optical character recognition technology and image recognition algorithms to extract the identification and nameplate on the medical device, identify the text, numbers, and symbols in the identification to obtain the model, brand, production date, and usage instruction information of the medical device. At the same time, extract the color, position, and shape features of the identification to assist in identifying different medical devices, obtain the image features of medical staff and pharmaceutical representatives, extract the appearance features and clothing features from the image, and combine them into a feature vector f p =[f1, f2,..., f n , where f i represents the appearance features and clothing features. Let the feature vector of drug Y be f Y =[Y1, Y2,..., Y k , where Y i represents the type and use of the drug. Let the feature vector of medical device Q be f Q =[Q1, Q2,..., Q l , where Q iIndicates the type and use of the device. Based on the calculation of the Bhattacharyya distance, obtain the department K to which the medical staff and pharmaceutical representative H belong. i , match the drugs and medical devices with the departments. Let the department matching function of drug Y be C Y (Y), and return the department to which drug Y belongs. Let the department matching function of device Q be C Q (Q), and return the department to which device Q belongs. When the medical staff and pharmaceutical representative H pick up drug Y and device Q, check whether it satisfies C Y (Y) ≠ K i and C Q (Q) ≠ K i . If the above conditions are met, an alarm is issued. Through the calculation of the above content, it is possible to realize the identification of medical staff and pharmaceutical representatives, and the monitoring of medical devices and drugs, and issue a warning in a timely manner when abnormal situations are detected.
[0109] For example Figure 2 , this embodiment provides: A pharmaceutical marketing monitoring system based on image recognition, including:
[0110] Feature recognition module: By recognizing the image data of medical staff and pharmaceutical representatives, extract the appearance features and clothing features of medical staff and pharmaceutical representatives, help identify different medical staff and their affiliated departments and job positions, and determine which pharmaceutical representative is marketing the drug and which pharmaceutical company according to the drugs taken by the medical staff;
[0111] Drug feature extraction module: For the annotation information of drug images, extract the integrity, accuracy, and consistency features of the annotations, and classify the drugs according to the annotation information;
[0112] Abnormal monitoring module: Use optical character recognition to extract key information from the identification nameplate of medical devices, obtain the image features of medical staff, combine them into feature vectors, and determine their affiliated departments in combination with the calculation of the Bhattacharyya distance. When medical staff pick up drugs or devices, check whether the department to which they belong matches the department of the medical staff. If not, an alarm is issued, so as to realize the accurate identification of medical staff and the effective monitoring of medical devices and drugs.
[0113] It should be noted that in the above embodiments, the descriptions of each embodiment have their own emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0114] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.
[0115] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.
[0116] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realizes the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.
[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide steps for realizing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.
[0118] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.
[0119] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.
Claims
1. A pharmaceutical marketing monitoring method based on image recognition, characterized in that The following steps are involved: S101. By identifying the image data of medical representatives and medical staff, the appearance features and clothing features of the medical representatives and medical staff are extracted to help identify different medical representatives, medical staff, the pharmaceutical companies to which the medical representatives belong, and the departments and job positions described by the medical staff; S102, extracting the completeness, accuracy and consistency characteristics of the annotation information of the drug image, and classifying the drugs according to the annotation information; S103. Use optical character recognition to extract key information from the nameplate of medical equipment, obtain the image features of medical staff, combine them into feature vectors, and use Bhattacharyya distance calculation to determine the department to which they belong. When medical staff take medicines, operate equipment, or participate in academic activities of the department, Check whether the department to which the patient belongs matches the department of the medical staff. If not, an alarm is issued, thereby achieving accurate identification of medical staff and effective monitoring of medical devices and drugs; At the same time, based on the drugs taken by the medical staff, we can determine which medical representative is marketing the drugs and which pharmaceutical company they are from.
2. The pharmaceutical marketing monitoring method based on image recognition according to claim 1, wherein In the above S01, the scenes of each department of the hospital are monitored in real time to obtain comprehensive image data, and the collected image data is pre-processed, including image enhancement and denoising. A convolutional neural network is used to analyze and extract key information of the appearance and clothing features of the medical staff in the images.
3. The method for monitoring pharmaceutical marketing based on image recognition according to claim 2, wherein The extraction of medical staff appearance features specifically includes the following steps: S1. Gray distribution feature: Convert the captured image containing the faces of medical staff into a grayscale image, traverse each pixel point of the image, and count the frequencies of different gray values to obtain a gray histogram. Assume the image size is M×N, the gray level is L, and the number of pixels with gray value i is n i , then the frequency p of the gray value i occurring i The specific calculation formula is as follows: where \(i = 0, 1,\cdots, L - 1\), by calculating the mean value and variance of \(p\) i to characterize the gray - level distribution feature. The specific calculation formula for the mean value is as follows: Among them, μ represents the mean of the grayscale distribution, which describes the average grayscale level of the entire face. The specific calculation formula of the variance is as follows: Among them, σ 2 represents the variance of the gray-scale distribution, reflecting the degree of dispersion of the gray-scale values relative to the mean value, which helps to identify the differences in facial skin colors of different individuals and the stability under different lighting conditions; S2. Facial edge sharpness feature: Edge detection is performed on the facial image of medical staff by using an edge detection operator. The gradient magnitude and direction of edge pixels are calculated, and the facial edge sharpness is evaluated based on the distribution of the gradient magnitude. Convolution operations are respectively applied to the convolution kernels in the horizontal and vertical directions to obtain the horizontal direction gradient G x and the vertical direction gradient G y . For each pixel (x, y) in the image, the specific calculation formula of the horizontal direction gradient G x is as follows: Among them, x m,n represents the pixel value in the (x, y) neighborhood, and w x,m,n represents the Sobel convolution kernel coefficient in the horizontal direction. The specific calculation formula for the vertical direction gradient G y is as follows: where, w y,m,n represents the Sobel convolution kernel coefficient in the vertical direction. The gradient magnitude is calculated based on the horizontal direction gradient and the vertical direction gradient. The specific calculation formula is as follows: Among them, G represents the gradient amplitude, which integrates the gradient information in the horizontal and vertical directions and is used to represent the edge strength at the pixel point. The mean and variance of the statistical gradient amplitude G are used as quantitative indicators of edge clarity. The specific calculation formula of the gradient amplitude mean is as follows: Among them, μ G represents the mean of the gradient magnitude. The specific calculation formula for the variance of the gradient magnitude is as follows: Among them, represents the variance of the gradient magnitude.
4. A method for monitoring pharmaceutical marketing based on image recognition according to claim 2, wherein Clothing feature extraction helps identify different medical staff and their departments and job positions by extracting the color, style and material features of clothing. The edge detection algorithm is used to extract the contour point set P = {p1, p2, ..., p n }, where p i =(L xi ,L yi ) is the coordinate of the point on the contour. For each contour point p i , construct a logarithmic polar coordinate space with it as the center, and divide the polar coordinate space into N r radial intervals and N θ Angle intervals, forming N = N r ×N θ bins, calculate the p i The distribution of all other contour points in the logarithmic polar coordinate space is obtained for each point p i The shape context descriptor S i , for the contour except p i Any point p outside j =(L xj ,L yj ), with p i =(L xi ,L yi ) is used as the center to construct the logarithmic polar coordinate space, and its logarithmic polar coordinate (r ij ,θ ij ) is calculated as follows: Among them, p j represents any point on the contour other than p i , r ij represents the logarithmic distance from point p j to point p i , θ ij represents the angle of point p j relative to point p i , L xi , L yi respectively represent the abscissa and ordinate of the selected central point p i on the contour in the Cartesian coordinate system. L xj , L yj respectively represent the abscissa and ordinate of any point p i other than p j on the contour in the Cartesian coordinate system. For each point p j , according to its log-polar coordinates (r ij , θ ij ), determine the bin it falls into. Let the number of points falling into the k-th bin be n ik , then the shape context descriptor S i of point p i can be expressed as S i = [n i1 , n i2 ,..., n iN . The Bhattacharyya distance is used to compare the similarity of corresponding points on two contours. The specific calculation formula of the Bhattacharyya distance is as follows: Among them, S a , S b respectively represent the shape context descriptors of corresponding points on two different contours, used to compare the similarity between them. S ak represents the k-th element of the shape context descriptor S a , and S bk represents the k-th element of the shape context descriptor S b . A small Bhattacharyya distance indicates that the shapes of the two contours are similar, and the shapes of the clothing styles at the key parts are close. Based on the calculation of the Bhattacharyya distance, the extracted clothing style features can be associated with the department and job position information of the medical staff, used to identify different medical staff and their affiliated departments and job positions.
5. A method for monitoring pharmaceutical marketing based on image recognition according to claim 1, characterized in that, In S102, according to the drug information list, which covers drug name, specification, manufacturer, approval number, and expiration date, check one by one whether the drug image annotation contains all items in the list to define the integrity index. If all key items are annotated, the integrity index is set to full marks. If there are n J key items not annotated and the total number of key items is J, the specific calculation formula for the integrity index is as follows: Precisely compare the position of the labeled drug information with the position of the actual drug information in the image. Using the image masking technology, create a mask image for the labeled drug information area and the area of the actual drug information in the image. Let the mask of the labeled drug name area be M a , and the mask of the actual area be M r . Then the specific calculation formula for the overlapping area is as follows: Among them, A overlap represents the overlapping area, the area of the marked region the area of the actual region The marking accuracy index is obtained by calculating the ratio of the overlapping area to the maximum value of the area of the marked region and the area of the actual region. The specific calculation formula of the accuracy index is as follows: I a A value close to 1 indicates high accuracy of the annotation result, that is, a good match between the position of the annotation information in the image and the position of the actual drug information. For the case where there are multiple annotations for the same image, check the consistency of the description of the key drug information and the position annotation between different annotations. By comparing the key information of different annotations, determine whether there are conflicts in the annotations. Suppose there are B n annotations. For the key information, count the number of pairs of inconsistent annotations as B m , then the specific calculation formula for the consistency index is as follows: Among them I co The value close to 1 indicates better annotation consistency and high reliability of the annotation information.
6. The method for monitoring pharmaceutical marketing based on image recognition according to claim 5, wherein According to the drug labeling information, the corresponding use and efficacy category of each drug label is used as a label, and the data set is divided into a training set, a validation set, and a test set. The drug labeling information is feature extracted, including the keywords in the drug name and the active ingredient content in the specification. The extracted data is standardized, and the drug labeling information after feature extraction and standardization is used as the input of the model. The drug labeling information is associated with the corresponding drug use and efficacy, and a labeled training data set is constructed. The prediction model of drug classification specifically includes the following steps: S1. Input the training set data into the CNN model, and obtain the prediction results of drug use and efficacy category by forward propagation. In the forward propagation process, the input data passes through the convolution layer, activation layer, pooling layer and fully connected layer in sequence, extracts and processes the features in the drug annotation information, and obtains the prediction probability value corresponding to each drug use and efficacy category in the output layer; S2. Calculate the difference between the predicted result and the true label based on the loss function. The classification of drug use and efficacy belongs to multi-classification problem, and the cross entropy loss function is used for calculation. The specific calculation formula is as follows: Among them, model i represents the probability that the model predicts belonging to class i, and y i represents the true class label, and L represents the cross-entropy loss function; S3. Calculate the gradient of loss for each model parameter through the back propagation algorithm; S4. Update the model parameters and continuously iterate to minimize the loss function value, so that the prediction result of the model is closer to the true label; S5. After each training cycle, use the validation set to evaluate the model performance to prevent overfitting.
7. A method for monitoring pharmaceutical marketing based on image recognition according to claim 1, characterized in that, In S103, the optical character recognition technology and image recognition algorithms are used to extract the identification and nameplate on the medical device, recognize the text, numbers, and symbols in the identification to obtain the model, brand, production date, and usage instructions of the medical device. At the same time, the color, position, and shape features of the identification are extracted to assist in identifying different medical devices. The image features of the medical staff are obtained, and the appearance features and clothing features are extracted from the image and combined into a feature vector f p =[f1,f2,...,f n , where f i represents the appearance features and clothing features. Let the feature vector of drug Y be f Y =[Y1,Y2,...,Y k , where Y i represents the type and use of the drug. Let the feature vector of medical device Q be f Q =[Q1,Q2,...,Q l , where Q i represents the type and use of the device. According to the calculation of the Bhattacharyya distance, the department K i to which the medical staff H belongs is obtained. The drug and the medical device are matched by department. Let the department matching function of drug Y be C Y (Y), which returns the department to which drug Y belongs. Let the department matching function of device Q be C Q (Q), which returns the department to which device Q belongs. When the medical staff H picks up drug Y and device Q, check whether it satisfies C Y (Y)≠K i and C Q (Q)≠K i . If the above conditions are satisfied, an alarm is issued. Through the calculation of the above content, the identification of medical staff, the monitoring of medical devices and drugs can be realized, and a warning can be issued in time when an abnormal situation is detected.
8. A pharmaceutical marketing monitoring system based on image recognition, which is applied to a pharmaceutical marketing monitoring method based on image recognition as described in any one of claims 1-7, characterized in that, Including: Feature recognition module: By recognizing the image data of medical staff, extract the appearance features and clothing features of medical staff to help identify different medical staff and their affiliated departments and job positions; Drug feature extraction module: For the annotation information of drug images, extract the integrity, accuracy and consistency features of the annotation, and classify the drugs according to the annotation information; Abnormality monitoring module: Use optical character recognition to extract key information from the identification nameplate of medical devices, obtain the image features of medical staff, combine them into feature vectors, and determine their affiliated departments in combination with the calculation of the Bhattacharyya distance. When medical staff take drugs or devices, check whether the affiliated department matches the department of the medical staff. If not, an alarm will be issued, so as to achieve accurate identification of medical staff and effective monitoring of medical devices and drugs.