Spt wheal identification grading method, device, system, equipment, medium and product

Abstract

The application discloses a kind of SPT wheal identification grading method, device, system, equipment, medium and product, it is related to medical image processing technical field.The method is first collected with different skin types and / or different point prick results multiple SPT sample image sequences and corresponding multiple SPT wheal identification grading label information, then each SPT sample image sequence is respectively introduced into the image feature extraction model based on artificial intelligence algorithm, and the corresponding multiple image feature extraction results are output, then the extraction result is used as model input item and the label information is used as model output item, and is imported into decision tree model to be rated and verified modeling, obtain SPT wheal identification grading model, finally the model is applied to obtain the SPT wheal identification grading result of target patient and be shown, so it can not be artificially carried out SPT result interpretation, substantially reduce clinical operation, and improve interpretation efficiency and interpretation result accuracy, and then can reduce the error caused by human factor.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to an SPT wheal identification and grading method, device, system, equipment, medium and product. BACKGROUND

[0002] In recent years, the self-reported prevalence of allergic rhinitis has been increasing year by year, and allergic diseases can affect the quality of life of patients and increase the economic burden of society, so it has become a global health problem. The main diagnostic methods of allergic diseases include skin test and serum specific immunoglobulin E (SlgE) detection (which has high consistency with the results of skin test, but SIgE detection is an in vitro blood test, which is less tolerated by children, and the waiting time is as long as 2-3 days, and the cost is more expensive than skin test, so it is generally used as a supplementary examination method for skin test contraindications). And the like.

[0003] The types of skin test are divided into intradermal test and skin prick test (SPT), among which, the intradermal test is to inject allergen solution into the dermis layer of the skin, which has a higher risk of causing systemic reaction, and the dosage is larger; while the SPT is to prick the allergen extract into the epidermis of the skin (usually on the palmar surface of the forearm) through a fine needle, and the liquid amount entering the body is only one ten-thousandth of the intradermal test, and the possibility of causing systemic reaction is low, so the SPT-based allergen in vivo test method has been widely used worldwide.

[0004] At present, the existing SPT result interpretation method mainly depends on the presence or absence of SPT wheal and its size (there are different standards for judging the level of SPT reaction at home and abroad, mainly using three ways of area judgment method, average diameter judgment method and longest diameter judgment method), and the doctor needs to operate manually in the whole medical procedure. Considering that the SPT wheal usually has irregular shape and large or small size, the measurement of the longest diameter is easy to be different due to different subjective measurements, so the judgment results of the average diameter judgment method and the longest diameter judgment method will have certain errors; as shown in the table, the area judgment method is the most accurate SPT result interpretation method, but the operation is complex, which brings great challenges to clinical operation. Although artificial intelligence (AI, which is to use machines to simulate the working way of human brain neurons, so as to think and master learning like human beings) has made rapid application development in the medical field, but the scheme of using artificial intelligence technology to assist computer equipment to interpret SPT results is still rarely developed at home and abroad. Figure 1

[0005] ​In summary, with the rapid development of artificial intelligence in the medical field, the continuous emergence of new technologies will provide more possibilities for the diagnosis and treatment of allergic diseases. Therefore, how to accurately identify and grade SPT wheals based on artificial intelligence technology and images before and after skin prick test, in order to assist clinicians in interpreting wheal size information and disease grading prediction, and thus improve the efficiency and accuracy of diagnosis and reduce errors caused by human factors, is a topic that needs to be studied by those skilled in the art. SUMMARY

[0006] The purpose of the present application is to provide an SPT wheal identification and grading method, device, system, computer equipment, computer readable storage medium and computer program product, to solve the problems of complex operation, low efficiency and easy human error in the existing SPT result interpretation method.

[0007] In order to achieve the above-mentioned purpose, the present application adopts the following technical scheme:

[0008] In a first aspect, an SPT wheal identification and grading method is provided, comprising:

[0009] A plurality of SPT sample image sequences with different skin types and / or different prick results and a plurality of SPT wheal identification and grading annotation information corresponding one-to-one to the plurality of SPT sample image sequences are collected, wherein the SPT sample image sequence contains a single-frame pre-prick test sample image and a plurality of post-prick test sample images arranged in order from early to late according to the collection time, the plurality of post-prick test sample images are periodically collected after the skin prick test, and the SPT wheal identification and grading annotation information contains annotation results of the presence or absence of SPT wheal, the size and / or size grading of the SPT wheal;

[0010] Each SPT sample image sequence in the plurality of SPT sample image sequences is respectively introduced into an image feature extraction model based on an artificial intelligence algorithm, and a plurality of image feature extraction results corresponding one-to-one to the plurality of SPT sample image sequences are output;

[0011] The plurality of image feature extraction results are respectively used as model input items, and the plurality of SPT wheal identification and grading annotation information are respectively used as model output items, which are introduced into a decision tree model for rate verification modeling to obtain an SPT wheal identification and grading model;

[0012] An SPT sample image sequence of a target patient is obtained;

[0013] The SPT sample image sequence of the target patient is introduced into the image feature extraction model, and an image feature extraction result of the target patient is output;

[0014] The image feature extraction result of the target patient is input into the SPT wheal identification grading model, and an SPT wheal identification grading result of the target patient is output and displayed.

[0015] Based on the above invention content, a new scheme for SPT wheal identification and grading based on artificial intelligence technology and images before and after skin prick test is provided, that is, a plurality of SPT sample image sequences with different skin types and / or different prick results and corresponding SPT wheal identification grading annotation information are collected first, then each SPT sample image sequence is input into an image feature extraction model based on an artificial intelligence algorithm, and a plurality of image feature extraction results are output, then the extraction results are input as model input items and the annotation information is input as model output items, and a decision tree model is input for rate verification modeling to obtain an SPT wheal identification grading model, and finally the model is applied to obtain an SPT wheal identification grading result of a target patient and display it, so that SPT result interpretation can be performed without manual operation, the clinical operation can be greatly reduced, and the interpretation efficiency and accuracy can be improved, thereby assisting clinicians in interpreting wheal size information and predicting disease classification, etc., which is beneficial to improving the diagnosis efficiency and accuracy, reducing errors caused by human factors, and facilitating practical application and promotion.

[0016] In one possible design, each frame image in the SPT sample image sequence is collected by a depth camera.

[0017] In one possible design, the image feature extraction model includes an image preprocessor, a deep residual neural network, a long short-term memory recurrent neural network, a fully connected neural network, and a multilayer perceptron connected in sequence, wherein the deep residual neural network, the long short-term memory recurrent neural network, the fully connected neural network, and the multilayer perceptron are respectively used to identify image features of different dimensions, so that the image feature extraction result contains color information of a prick area, position information of an SPT wheal, three-dimensional data information of an SPT wheal, and / or surrounding erythema information of an SPT wheal.

[0018] In one possible design, the rate verification modeling includes a model training process using a model parameter adjustment method and / or an hyperparameter optimization algorithm, and a process of preventing model overfitting using a cross-validation technique and / or a regularization technique.

[0019] In a second aspect, an SPT wheal identification grading device is provided, which includes a sample data collection unit, an image feature extraction unit, a rate verification modeling unit, an image sequence acquisition unit, and a wheal model application unit.

[0020] The sample data collection unit is configured to collect a plurality of SPT sample image sequences with different skin types and / or different point stimulation results and a plurality of SPT wheal identification grading label information corresponding to the plurality of SPT sample image sequences one by one, wherein the SPT sample image sequence contains a single-frame pre-point stimulation test sample image and a plurality of post-point stimulation test sample images arranged in order from early to late according to the collection time, the plurality of post-point stimulation test sample images are periodically collected after the skin point stimulation test, and the SPT wheal identification grading label information contains a label result of the appearance, size and / or size grading of the SPT wheal;

[0021] The image feature extraction unit is in communication connection with the sample data collection unit and is configured to respectively import each SPT sample image sequence in the plurality of SPT sample image sequences into an image feature extraction model based on an artificial intelligence algorithm, and output a plurality of image feature extraction results corresponding to the plurality of SPT sample image sequences;

[0022] The rate verification modeling unit is in communication connection with the sample data collection unit and the image feature extraction unit respectively, configured to respectively import the plurality of image feature extraction results as model input items and the plurality of SPT wheal identification grading label information as model output items into a decision tree model for rate verification modeling, and obtain an SPT wheal identification grading model;

[0023] The image sequence acquisition unit is configured to acquire an SPT sample image sequence of a target patient;

[0024] The image feature extraction unit is further in communication connection with the image sequence acquisition unit and is further configured to import the SPT sample image sequence of the target patient into the image feature extraction model, and output an image feature extraction result of the target patient;

[0025] The wheal model application unit is in communication connection with the image feature extraction unit and the rate verification modeling unit respectively, configured to import the image feature extraction result of the target patient into the SPT wheal identification grading model, output an SPT wheal identification grading result of the target patient and display the result.

[0026] In a third aspect, the present application provides an SPT wheal identification grading system, comprising a database server, an SPT diagnosis and treatment table and an image analysis device, wherein the SPT diagnosis and treatment table comprises a hand to be examined carrying base, a vertical rod, a horizontal bar and a camera, the bottom end of the vertical rod is fixedly connected with the hand to be examined carrying base, one end of the horizontal bar is fixedly connected with the upper part of the vertical rod, and the camera is installed on the horizontal bar and the camera lens faces the hand to be examined carrying base;

[0027] The database server is in communication connection with the image analysis device, and is configured to store a plurality of SPT sample image sequences with different skin types and / or different point prick results, and a plurality of SPT wheal identification grading annotation information corresponding to the plurality of SPT sample image sequences, wherein the SPT sample image sequence contains single-frame pre-point prick test sample images and multi-frame post-point prick test sample images sorted in time sequence from early to late, the multi-frame post-point prick test sample images are periodically collected after the skin point prick test, and the SPT wheal identification grading annotation information contains annotation results of the presence or absence of SPT wheal, the size of the SPT wheal, and / or the size grading of the SPT wheal.

[0028] The camera is in communication connection with the image analysis device, and is configured to collect images of the patient's arm placed on the arm bearing base before and after the skin point prick test to obtain the SPT sample image sequence of the target patient, and transmit the image collection result to the image analysis device in real time.

[0029] The image analysis device is configured to execute the SPT wheal identification grading method as described in the first aspect or any possible design of the first aspect.

[0030] In one possible design, the SPT diagnosis and treatment platform further comprises an LED light source and a brightness sensor, wherein the LED light source is installed on the horizontal bar and has a light-emitting surface facing the arm bearing base, and the brightness sensor is installed on the lower part of the vertical rod and has a light-sensing surface also facing the arm bearing base.

[0031] The brightness sensor is in communication connection with the image analysis device, and is configured to synchronously collect brightness information when the camera collects images, and transmit the brightness collection result to the image analysis device in real time, so that the image analysis device uses the brightness information to correct the synchronously collected images when preprocessing the SPT sample image sequence of the target patient.

[0032] In a fourth aspect, the present application provides a computer device comprising a memory, a processor and a transceiver in communication connection, wherein the memory is configured to store a computer program, the transceiver is configured to receive and send messages, and the processor is configured to read the computer program and execute the SPT wheal identification grading method as described in the first aspect or any possible design of the first aspect.

[0033] In a fifth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions are executed on a computer, the SPT wheal identification grading method as described in the first aspect or any possible design of the first aspect is executed.

[0034] In a sixth aspect, the present application provides a computer program product comprising computer programs or instructions which, when executed by a computer, implement the SPT wheal identification and grading method as described in the first aspect or any possible design of the first aspect.

[0035] The above-mentioned scheme has the following beneficial effects:

[0036] (1) The present application creatively provides a new scheme for SPT wheal identification and grading based on artificial intelligence technology and images before and after skin prick test, that is, a plurality of SPT sample image sequences with different skin types and / or different prick results and corresponding SPT wheal identification and grading annotation information are collected first, then each SPT sample image sequence is introduced into an image feature extraction model based on artificial intelligence algorithm, and a plurality of image feature extraction results are output, then the extraction results are taken as model input items and the annotation information is taken as model output items, and a decision tree model is introduced for rate verification modeling to obtain an SPT wheal identification and grading model, finally the model is applied to obtain the SPT wheal identification and grading results of a target patient and display the results, so that SPT result interpretation can be performed without manual operation, the clinical operation is greatly reduced, the interpretation efficiency and accuracy are improved, and the clinician can be assisted in wheal size information interpretation and disease grading prediction, etc., which is beneficial to improve the diagnosis efficiency and accuracy and reduce errors caused by human factors;

[0037] (2) The hardware entity system can also have the advantages of low cost, good heat dissipation and simple manufacturing, which is beneficial to the direct transformation of research and development achievements;

[0038] (3) Not only the accuracy of SPT wheal interpretation is solved, but also the accuracy and feasibility of the whole scheme are verified through clinical control study in the project development process, which provides data basis for medical device registration;

[0039] (4) A simple hardware design scheme is proposed and made, which adopts the latest sensor technology and hardware manufacturing process, optimizes the selection and layout of hardware components, reduces unnecessary components, reduces the complexity and manufacturing cost of the whole scheme, adopts a high-efficiency heat dissipation structure to ensure that the whole scheme can maintain stable working temperature during long-time operation, thereby improving its reliability and durability;

[0040] (5) For the difficulty of a large number of parameter optimization in the detection and signal processing stage, a deep residual neural network, a long short-term memory recurrent neural network and a fully connected neural network are introduced to construct a model, and a machine learning algorithm is used to automatically adjust parameters to adapt to the wheal detection requirements under different conditions, and the research and optimization of the signal processing algorithm are also strengthened to improve its robustness and accuracy in complex environments;

[0041] (6) In order to solve the problem of poor repeatability of the recognition result, the decision tree model can obtain expert feature estimation from the deep residual neural network, calculate the importance of these features to the final decision, and combine the data manually labeled by professional doctors to train the model. By continuously adjusting the model parameters and optimizing the algorithm, the recognition accuracy and generalization ability of the model are improved, and cross-validation and regularization techniques are used to prevent overfitting, thereby improving the reliability and consistency of the recognition result, facilitating practical application and promotion. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The annotation before and after of the two groups of point puncture surface sample images with irregular SPT wheal morphology provided by the prior art are shown in the example graphs, wherein, Figure 1 (a) in the example graphs shows a group of point puncture surface sample images with irregular SPT wheal morphology before and after annotation, Figure 1 (b) in the example graphs shows another group of point puncture surface sample images with irregular SPT wheal morphology before and after annotation.

[0044] Figure 2 The flowchart of the SPT wheal recognition grading method provided by the embodiment of the present application.

[0045] Figure 3 The structural diagram of the image feature extraction model provided by the embodiment of the present application.

[0046] Figure 4 The basic flowchart of the decision tree provided by the embodiment of the present application.

[0047] Figure 5 The structural diagram of the SPT wheal recognition grading device provided by the embodiment of the present application.

[0048] Figure 6 The structural diagram of the SPT wheal recognition grading system provided by the embodiment of the present application.

[0049] Figure 7 The structural diagram of the SPT diagnosis and treatment station provided by the embodiment of the present application.

[0050] Figure 8A front view structure schematic diagram of the arm carrying base to be inspected provided by the embodiment of the present application.

[0051] Figure 9 A structure schematic diagram of the computer device provided by the embodiment of the present application.

[0052] In the above-mentioned drawings: 201-arm carrying base to be inspected; 2011-arm limiting groove; 202-vertical rod; 203-crossbar; 204-camera; 2041-camera head; 2042-TOF sensor; 205-LED light source; 206-brightness sensor. DETAILED DESCRIPTION

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the present application will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the drawings structure is only some embodiments of the present application, and for those skilled in the art, other embodiments can be obtained without creative labor on the basis of these embodiments. It should be noted that the description of these embodiment modes is used to help understand the present application, but does not constitute a limitation on the present application.

[0054] It should be understood that although the terms first and second, etc. may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, the first object can be called the second object, and similarly, the second object can be called the first object, without departing from the scope of the example embodiments of the present application.

[0055] It should be understood that for the term "and / or" which may appear in the present text, it is only a description of the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can mean that there are three cases of A alone, B alone, or A and B at the same time; For example, A, B and / or C, which means that any one of A, B and C or any combination thereof exists; For the term " / and" which may appear in the present text, it is another description of the relationship of another associated object, which means that there can be two kinds of relationships, for example, A / and B, which can mean that there are two cases of A alone or A and B at the same time; In addition, for the character " / " which may appear in the present text, it generally means that the associated objects before and after are an "or" relationship.

[0056] Embodiment:

[0057] As Figures 2 to 4As shown, the SPT wheal identification and grading method provided in the first aspect of the present embodiment can be executed by, but not limited to, a computer device with certain computing resources, such as a cloud server, a personal computer (PC, a multi-purpose computer with size, price and performance suitable for personal use), a smart phone, a personal digital assistant (PDA) or a wearable device, etc. Figure 2 As shown, the SPT wheal identification and grading method can include, but is not limited to, the following steps S1-S6.

[0058] S1. Collecting a plurality of SPT sample image sequences with different skin types and / or different point sting results and a plurality of SPT wheal identification and grading annotation information corresponding to the plurality of SPT sample image sequences one by one, wherein the SPT sample image sequence contains, but is not limited to, a single-frame pre-point sting interview sample image and a plurality of post-point sting interview sample images arranged in order from early to late according to the collection time, the plurality of post-point sting interview sample images are periodically collected after the skin point sting test, and the SPT wheal identification and grading annotation information contains, but is not limited to, the annotation results of the presence or absence of SPT wheal, the size and / or size grading, etc.

[0059] In the step S1, the plurality of SPT sample image sequences and the plurality of SPT wheal identification and grading annotation information constitute the sample data required for subsequent construction of the SPT wheal identification and grading model, wherein each frame of image in the SPT sample image sequence can be specifically, but not limited to, collected by the camera 204 as shown: Figure 7 As shown, after the patient's arm is placed on the arm bearing base 201 to be examined, the single-frame pre-point sting interview sample image is collected by the camera 204; then the skin point sting is performed, and the post-point sting interview sample images are collected by the camera 204. Figure 1The "+" and "-" shown, and the point prick position numbers such as "1", "2", "3", "4", and "5"; the first frame of the post-prick sample image is collected by the camera 204 at 10 minutes after the skin prick; the second frame of the post-prick sample image is collected by the camera 204 at 15 minutes after the skin prick; the third frame of the post-prick sample image is collected by the camera 204 at 20 minutes after the skin prick; and so on (i.e., the collection period of the post-prick sample image is 5 minutes as an example). Considering that the SPT wheal has a three-dimensional feature, in order to facilitate the extraction of more abundant and accurate image features in the subsequent process, preferably, each frame of image in the SPT sample image sequence contains depth information, i.e., each frame of image in the SPT sample image sequence is collected by a depth camera, which is also called a 3D camera, which is an existing device capable of obtaining object depth information. The SPT wheal recognition and grading annotation information can be manually annotated by professional doctors according to their experience in SPT wheal recognition and grading. In addition, the plurality of SPT sample image sequences and the plurality of SPT wheal recognition and grading annotation information can be stored in a database for local device reading and collection; the different skin types specifically include, but are not limited to, yellow skin, white skin, black skin, and brown skin; and the different prick results specifically include, but are not limited to, positive, negative, and weak positive.

[0060] S2. Each SPT sample image sequence in the plurality of SPT sample image sequences is introduced into an image feature extraction model based on an artificial intelligence algorithm, and a plurality of image feature extraction results corresponding one-to-one to the plurality of SPT sample image sequences are output.

[0061] In the step S2, the artificial intelligence algorithm is a special type of computing program designed to simulate, extend, or surpass certain aspects of human intelligence. Specifically, through machine learning, deep learning, or reinforcement learning, etc., the computer system can perform complex tasks such as learning, reasoning, self-correction, perception, and understanding of natural language, so that the image feature extraction model can be similar to a doctor's visual recognition organ, and can recognize the multi-dimensional image feature extraction results. In order to extract rich and accurate image features, so as to accurately perform SPT wheal recognition and grading in the subsequent process, preferably, as shown in FIG. 2, the image feature extraction model includes a convolutional neural network (CNN) and a recurrent neural network (RNN), and the CNN and the RNN are connected in series. Figure 3As shown, the image feature extraction model includes but is not limited to an image preprocessor, a deep residual neural network, a long short-term memory recurrent neural network, a fully connected neural network, and a multi-layer perceptron, which are connected in sequence, wherein the deep residual neural network, the long short-term memory recurrent neural network, the fully connected neural network, and the multi-layer perceptron are respectively used to identify image features of different dimensions, so that the image feature extraction result includes but is not limited to color information of the pruritic area, position information of the SPT wheal, three-dimensional data information of the SPT wheal, and / or surrounding erythema information of the SPT wheal, etc.

[0062] In the image feature extraction model, specifically, the image preprocessor is used to perform conventional processing such as color difference correction, image scaling, and / or grayscale on each frame of image imported in the SPT sample image sequence, so that the pre-processing result is suitable for input into the deep residual neural network, etc.

[0063] In the image feature extraction model, specifically, the deep residual neural network (ResNet) is a deep convolutional neural network (Convolutional Neural Networks, CNN) architecture, which solves the degradation problem in deep network training by introducing "residual learning". In ResNet, residual learning is achieved by introducing "shortcut connections" or "skip connections", which allow direct transmission of input from earlier layers to later layers. The essence of ResNet: the residual network does not fit the initial mapping but fits a residual mapping such as ​ResNet alleviates vanishing gradient through residual blocks and skip connections, extracts key features with downsampling and global pooling, classifies or regresses through fully connected layers, and accelerates training and generalization with batch normalization and ReLU. The deep residual neural network exemplarily but not limitedly adopts a ResNet34 network, and its workflow can be briefly summarized as follows: (1) input layer: first, image data is sent as input into the first convolutional layer of ResNet34, which is usually a 7x7 convolutional layer for preliminary feature extraction; (2) residual block: then, image data passes through a series of residual blocks, each containing two convolutional layers for further feature extraction, and each residual block is implemented through a skip connection, which means that the input of each residual block is directly added to its output; (3) downsampling: between residual blocks, downsampling operations can be achieved by changing the convolutional stride or using additional convolutional layers to reduce the size of the feature map and preserve the spatial hierarchy; (4) fully connected layer: after all residual blocks, a global average pooling layer reduces the spatial dimensions of the feature map to 1x1, generating a feature vector, which is then passed to one or more fully connected layers (also known as dense connection layers or linear layers) for classification or regression tasks; (5) output: finally, the ResNet34 network outputs the prediction result, for classification tasks, the output is usually a probability distribution vector representing the probability of the image belonging to each class; for regression tasks, the output may be specific numerical values.

[0064] In the image feature extraction model, specifically, the long short-term memory recurrent neural network LSTM (Long Short-Term Memory) is a time recurrent neural network suitable for processing and predicting important events with relatively long intervals and delays in time series. The first stage of the LSTM is the forget gate, which determines which information needs to be forgotten from the cell state, the next stage is the input gate, which determines which new information can be stored in the cell state, and the last stage is the output gate, which determines what value to output. The forget gate is the output of the previous layer and the sequence data to be input into the current layer as input, through an activation function , the output is . The output value of the forget gate is in the interval [0, 1], representing the probability of forgetting the cell state of the previous layer, 1 is "completely retained", and 0 is "completely discarded". The input gate includes two parts, the first part uses an activation function , the output is , and the second part uses a tanh activation function, the output is , i.e. the new information that the cell is to retain. Cell state update: is the output of the forget gate, controlling how much of the previous cell state is forgotten, is the multiplication of the two outputs of the input gate, representing the new information that the cell is to retain, based on which the cell state of this layer can be updated using the new information . The output gate is used to control how much of the cell state of this layer is filtered: first, an activation function is applied to obtain a value in the interval [0, 1], then the cell state is multiplied by , which is the output of this layer.

[0065] In the image feature extraction model, specifically, the Fully Connected Neural Network (FCNN) belongs to a kind of feedforward neural network, mainly composed of an input layer, a hidden layer and an output layer, and there can be multiple neurons in each hidden layer, which is the most basic type of neural network. FCNN is composed of multiple fully connected layers, and each layer of neurons is connected to each neuron of the previous layer, which can learn the complex relationship between different features.

[0066] In the image feature extraction model, specifically, the Multilayer Perceptron (MLP) is a kind of feedforward artificial neural network model, composed of an input layer, one or more hidden layers and an output layer. Each node (or artificial neuron) is connected to other nodes with corresponding weights and biases. MLP can approximate any continuous function (universal approximation theorem) through the non-linear transformation of the hidden layer.

[0067] S3. The image feature extraction results are input into the decision tree model for rate verification modeling, and the SPT nodule identification grading model is obtained.

[0068] In step S3, the working principle of the decision tree model is to make a series of binary decisions according to the input features, and finally complete the identification and grading of SPT nodule. Specifically, the image feature extraction results can mainly be the feature extraction results obtained by the deep residual neural network, so that the decision tree can calculate the importance of these features to the final decision, and then combine the professional doctor's artificial marking conclusion (i.e. the SPT nodule identification grading annotation information) to establish a decision rule with the maximum accuracy. The specific basic process is as followsFigure 4 The construction of the decision tree is firstly selecting the features with strong classification ability, and secondly generating the decision tree, typical algorithms are ID3 algorithm and C4.5 algorithm, the processes of generating the decision tree by the two algorithms are similar, but the ID3 algorithm uses information gain as the feature selection measure, and the C4.5 algorithm uses information gain ratio. Decision tree pruning: the reason for pruning is that the tree generated by the decision tree generation algorithm is very accurate in predicting the training data, but very poor in classifying unknown data, which produces the phenomenon of overfitting, the involved algorithm is CART algorithm (Classification and Regression Trees, a supervised learning algorithm based on binary tree). The ID3 algorithm uses the division method of calculating information entropy, assuming that there is a training data set D and a feature A, the information gain is calculated according to the following steps: first, calculate the empirical entropy of the data set D:

[0069]

[0070] wherein, is the number of the kth sample, is the number of the data set D; then calculate the empirical conditional entropy of the feature A to the data set D ; then calculate the information gain: ; the greater the information gain, the greater the "purity improvement" obtained by using attribute A to divide. Therefore, the information gain can be used to select the division attribute of the decision tree. The division criterion used by the C4.5 algorithm is information gain ratio, the information gain ratio of the feature A to the data set D , is called the entropy of the data set D about A: ; in the classification problem, assuming that there are K classes, the probability that the sample point belongs to k, then the Gini index of the probability distribution is:

[0071]

[0072] For a given sample set D, the Gini index is:

[0073]

[0074] The CART decision tree uses "Gini index" to select the division attribute. The purity of the data set D can be measured by the Gini value, the smaller the Gini(D), the higher the purity of the data set.

[0075] In the step S3, the specific process of the rate verification modeling is also a model training and optimization process; specifically, the rate verification modeling includes but is not limited to a model training process using a model parameter adjustment method and / or an hyperparameter optimization algorithm and a process using a cross-validation technique and / or a regularization technique to prevent model overfitting. The aforementioned model parameter adjustment method is a conventional method; the aforementioned hyperparameter optimization algorithm is exemplified but not limited to a Bayesian optimization algorithm; and the aforementioned cross-validation technique and regularization technique are also existing technologies.

[0076] S4. Obtain an SPT sample image sequence of a target patient.

[0077] In the step S4, the SPT sample image sequence of the target patient can also be but is not limited to acquired by a camera 204 as shown in the figure. Figure 7

[0078] S5. Import the SPT sample image sequence of the target patient into the image feature extraction model, and output an image feature extraction result of the target patient.

[0079] S6. Import the image feature extraction result of the target patient into the SPT wheal identification and grading model, output an SPT wheal identification and grading result of the target patient, and display the result.

[0080] After the step S6, the SPT wheal identification and grading model can also be continuously improved and optimized according to the feedback opinions / data of professional doctors on the SPT wheal identification and grading result, so as to adapt to different use scenarios and requirements. In addition, considering the privacy protection ethics, it is also necessary to properly protect the personal information and image data of patients in the whole scheme, so as to avoid data leakage and abuse, and comply with relevant laws, regulations and ethical standards, and ensure the legality of the whole scheme.

[0081] ​Thus, based on the SPT wheal identification grading method described in the foregoing steps S1-S6, a new scheme for SPT wheal identification and grading based on artificial intelligence technology and images before and after skin prick test is provided, that is, a plurality of SPT sample image sequences with different skin types and / or different prick results and corresponding SPT wheal identification grading annotation information are collected first, then each SPT sample image sequence is introduced into an image feature extraction model based on an artificial intelligence algorithm, and a plurality of image feature extraction results corresponding to the plurality of SPT sample image sequences are output, then the extraction results are taken as model input items and the annotation information is taken as model output items, and a decision tree model is introduced for rate verification modeling to obtain an SPT wheal identification grading model, finally the model is applied to obtain SPT wheal identification grading results of a target patient and to display the results, so that SPT result interpretation can be performed without manual operation, clinical operation can be greatly reduced, interpretation efficiency and interpretation result accuracy can be improved, and the clinician can be assisted in interpreting wheal size information and predicting disease grading, which is beneficial to improving diagnosis efficiency and accuracy, reducing errors caused by human factors, and facilitating actual application and promotion.

[0082] As shown in Figure 5 The second aspect of the present embodiment provides a virtual device for implementing the SPT wheal identification grading method of the first aspect, comprising a sample data collection unit, an image feature extraction unit, a rate verification modeling unit, an image sequence acquisition unit, and a wheal model application unit.

[0083] The sample data collection unit is configured to collect a plurality of SPT sample image sequences with different skin types and / or different prick results and a plurality of SPT wheal identification grading annotation information corresponding one-to-one to the plurality of SPT sample image sequences, wherein the SPT sample image sequence contains a single-frame pre-prick test sample image and a plurality of post-prick test sample images arranged in order from early to late according to the collection time, the plurality of post-prick test sample images are periodically collected after the skin prick test, and the SPT wheal identification grading annotation information contains annotation results of the presence or absence of SPT wheal, the size of the SPT wheal, and / or the size grading of the SPT wheal.

[0084] The image feature extraction unit is in communication connection with the sample data collection unit and is configured to introduce each SPT sample image sequence in the plurality of SPT sample image sequences into an image feature extraction model based on an artificial intelligence algorithm, and output a plurality of image feature extraction results corresponding one-to-one to the plurality of SPT sample image sequences.

[0085] The rate verification modeling unit is respectively communicatively connected with the sample data collection unit and the image feature extraction unit, is configured to input the image feature extraction results as model input items, and is configured to input the SPT wheal identification grading information as model output items, and is configured to import the decision tree model to perform rate verification modeling, and obtain an SPT wheal identification grading model.

[0086] The image sequence acquisition unit is configured to acquire an SPT sample image sequence of a target patient.

[0087] The image feature extraction unit is further communicatively connected with the image sequence acquisition unit, and is configured to input the SPT sample image sequence of the target patient into the image feature extraction model, and output the image feature extraction result of the target patient.

[0088] The wheal model application unit is respectively communicatively connected with the image feature extraction unit and the rate verification modeling unit, is configured to input the image feature extraction result of the target patient into the SPT wheal identification grading model, and is configured to output and display the SPT wheal identification grading result of the target patient.

[0089] The working process, working details and technical effects of the foregoing device provided by the second aspect of the embodiment can be referred to the SPT wheal identification grading method described in the first aspect, and will not be described here again.

[0090] As Figures 6 to 8As shown, the third aspect of the present embodiment provides an entity system for implementing the SPT wheal identification and grading method of the first aspect, including but not limited to a database server, an SPT diagnosis and treatment table, an image analysis device, and the like. The SPT diagnosis and treatment table includes but is not limited to a to-be-examined arm carrying base 201, a vertical rod 202, a horizontal bar 203, a camera 204, and the like. The bottom end of the vertical rod 202 is fixedly connected to the to-be-examined arm carrying base 201. One end of the horizontal bar 203 is fixedly connected to the upper part of the vertical rod 202. The camera 204 is installed on the horizontal bar 203 with the camera lens facing the to-be-examined arm carrying base 201. The database server is communicatively connected to the image analysis device and is used to store a plurality of SPT sample image sequences with different skin types and / or different point sting results and a plurality of SPT wheal identification and grading annotation information corresponding to the plurality of SPT sample image sequences. The SPT sample image sequence contains single-frame pre-point sting test sample images and multi-frame post-point sting test sample images sorted in chronological order from early to late. The multi-frame post-point sting test sample images are periodically collected after the skin point sting test. The SPT wheal identification and grading annotation information contains annotation results of the presence or absence of SPT wheal, the size of the SPT wheal, and / or the size grading of the SPT wheal. The camera 204 is communicatively connected to the image analysis device and is used to collect images of the patient's arm placed on the to-be-examined arm carrying base 201 before and after the skin point sting test to obtain the SPT sample image sequence of the target patient, and to transmit the image collection results to the image analysis device in real time. The image analysis device is used to execute the SPT wheal identification and grading method of the first aspect. In order to achieve the purpose of limiting the patient's arm (i.e., to avoid unnecessary differences in each frame of image in the SPT sample image sequence of the target patient due to arm movement), as shown, an arm limiting groove 2011 is formed on the top surface of the to-be-examined arm carrying base 201. In addition, as shown, the camera 204 can be composed of a camera 2041 and a TOF (Time of Flight) sensor 2042. Figure 8 Figure 7

[0091] ​​In a possible design, the SPT diagnosis and treatment platform further includes, but is not limited to, an LED (Light-emitting Diode) light source 205 and a brightness sensor 206, etc. The LED light source 205 is installed on the horizontal rod 203 and has a light-emitting surface facing the arm carrying base 201 to be examined. The brightness sensor 206 is installed at a lower part of the vertical rod 202 and has a light-sensing surface also facing the arm carrying base 201 to be examined. The brightness sensor 206 is communicatively connected to the image analysis device, and is configured to synchronously collect brightness information when the camera 204 collects images, and transmit the brightness collection result to the image analysis device in real time, so that the image analysis device uses the brightness information to correct the synchronously collected images when preprocessing the SPT sample image sequence of the target patient. The aforementioned correction processing is one of the image preprocessing, so as to avoid the image feature extraction difference caused by the inconsistent brightness, and further ensure the accuracy of the recognition and grading results.

[0092] The working process, working details and technical effects of the aforementioned system provided by the third aspect of the embodiment can be referred to the SPT wheal recognition and grading method described in the first aspect, and will not be described here again. In addition, the aforementioned system can have the following technical effects: the hardware entity system can also be low in cost, good in heat dissipation, and simple to manufacture, and the like, and is beneficial to the direct transformation of research and development achievements.

[0093] As shown in Figure 9 The fourth aspect of the embodiment provides a computer device for executing the SPT wheal recognition and grading method described in the first aspect, which includes a memory, a processor and a transceiver that are communicatively connected in sequence. The memory is configured to store a computer program, the transceiver is configured to receive and send messages, and the processor is configured to read the computer program and execute the SPT wheal recognition and grading method described in the first aspect. Specifically, the memory can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a first input first output (FIFO) memory and / or a first input last output (FILO) memory, etc. The processor can be, but is not limited to, a microprocessor with a model number of STM32F105 series. In addition, the computer device can further include, but is not limited to, a power module, a display screen and other necessary components.

[0094] The working process, working details and technical effects of the aforementioned computer device provided by the fourth aspect of the embodiment can be referred to the SPT wheal identification grading method described in the first aspect, and will not be repeated here.

[0095] The fifth aspect of the embodiment provides a computer readable storage medium storing instructions of the SPT wheal identification grading method described in the first aspect, that is, the computer readable storage medium stores instructions, and when the instructions are run on a computer, the SPT wheal identification grading method described in the first aspect is executed. Wherein, the computer readable storage medium refers to a carrier for storing data, which can include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, USB flash disks and / or Memory Stick and other computer readable storage media, and the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0096] The working process, working details and technical effects of the aforementioned computer readable storage medium provided by the fifth aspect of the embodiment can be referred to the SPT wheal identification grading method described in the first aspect, and will not be repeated here.

[0097] The sixth aspect of the embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements the SPT wheal identification grading method described in the first aspect. Wherein, the computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices.

[0098] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of SPT wheal identification grading, characterized in that, The method comprises the following steps: Collecting a plurality of SPT sample image sequences with different skin types and / or different point prick results and a plurality of SPT wheal identification grading annotation information corresponding to the plurality of SPT sample image sequences, wherein the SPT sample image sequence contains a single-frame pre-point prick interview sample image and a plurality of post-point prick interview sample images arranged in order from early to late according to the collection time, the plurality of post-point prick interview sample images are periodically collected after the skin prick test, and the SPT wheal identification grading annotation information contains the annotation results of the presence or absence, size and / or size grading of SPT wheal; Each SPT sample image sequence in the plurality of SPT sample image sequences is respectively introduced into an image feature extraction model based on an artificial intelligence algorithm, and a plurality of image feature extraction results corresponding to the plurality of SPT sample image sequences are output. The plurality of image feature extraction results are respectively used as model input items of a decision tree model, and the plurality of SPT wheal identification grading annotation information is respectively used as model output items of the decision tree model, which are introduced into the decision tree model for rate verification modeling to obtain an SPT wheal identification grading model. Obtaining an SPT sample image sequence of a target patient; The SPT sample image sequence of the target patient is introduced into the image feature extraction model, and an image feature extraction result of the target patient is output. The image feature extraction result of the target patient is introduced into the SPT wheal identification grading model, and an SPT wheal identification grading result of the target patient is output and displayed.

2. The SPT wheal identification grading method according to claim 1, characterized in that, Each frame of image in the SPT sample image sequence is collected by a depth camera.

3. The SPT wheal identification grading method of claim 1, wherein, The image feature extraction model comprises an image preprocessor, a deep residual neural network, a long short-term memory recurrent neural network, a fully connected neural network and a multilayer perceptron connected in sequence, wherein the deep residual neural network, the long short-term memory recurrent neural network, the fully connected neural network and the multilayer perceptron are respectively used to identify image features of different dimensions, so that the image feature extraction result contains color information of a point prick area, position information of SPT wheal, three-dimensional data information of SPT wheal and / or surrounding erythema information of SPT wheal.

4. The SPT wheal identification grading method of claim 1, wherein, The rate verification modeling includes a model training process using a model parameter adjustment method and / or a hyperparameter optimization algorithm and a process of preventing model overfitting using a cross-validation technique and / or a regularization technique.

5. A SPT wheal identification grading apparatus characterized by, The method comprises a sample data collection unit, an image feature extraction unit, a rate verification modeling unit, an image sequence acquisition unit and a wheal model application unit. The sample data collection unit is configured to collect a plurality of SPT sample image sequences with different skin types and / or different point stimulation results and a plurality of SPT wheal identification grading annotation information corresponding to the plurality of SPT sample image sequences one by one, wherein the SPT sample image sequence contains a single-frame pre-point stimulation test sample image and a plurality of post-point stimulation test sample images arranged in order from early to late according to the collection time, the plurality of post-point stimulation test sample images are periodically collected after the skin point stimulation test, and the SPT wheal identification grading annotation information contains annotation results of the presence or absence, size and / or size grading of SPT wheal; The image feature extraction unit is in communication connection with the sample data collection unit and is configured to import each SPT sample image sequence in the plurality of SPT sample image sequences into an image feature extraction model based on an artificial intelligence algorithm respectively, and output a plurality of image feature extraction results corresponding to the plurality of SPT sample image sequences one by one; The rate verification modeling unit is in communication connection with the sample data collection unit and the image feature extraction unit respectively, and is configured to import the plurality of image feature extraction results as model input items of a decision tree model and the plurality of SPT wheal identification grading annotation information as model output items of the decision tree model respectively, import the decision tree model for rate verification modeling, and obtain an SPT wheal identification grading model; The image sequence acquisition unit is configured to acquire an SPT sample image sequence of a target patient; The image feature extraction unit is also in communication connection with the image sequence acquisition unit and is also configured to import the SPT sample image sequence of the target patient into the image feature extraction model, and output an image feature extraction result of the target patient; The wheal model application unit is in communication connection with the image feature extraction unit and the rate verification modeling unit respectively, and is configured to import the image feature extraction result of the target patient into the SPT wheal identification grading model, output an SPT wheal identification grading result of the target patient and display the result.

6. A SPT wheal identification grading system characterized in that, The system comprises a database server, an SPT diagnosis and treatment table, and an image analysis device. The SPT diagnosis and treatment table comprises a hand to be examined bearing base (201), a vertical rod (202), a horizontal bar (203), and a camera (204). The bottom end of the vertical rod (202) is fixedly connected to the hand to be examined bearing base (201). One end of the horizontal bar (203) is fixedly connected to the upper part of the vertical rod (202). The camera (204) is installed on the horizontal bar (203) and the camera lens faces the hand to be examined bearing base (201). The database server is in communication connection with the image analysis device, and is configured to store a plurality of SPT sample image sequences with different skin types and / or different point prick results and a plurality of SPT wheal identification grading annotation information corresponding to the plurality of SPT sample image sequences, wherein the SPT sample image sequence comprises a single-frame pre-point prick test sample image and a plurality of post-point prick test sample images sorted in time sequence from early to late, the post-point prick test sample images are periodically collected after the skin point prick test, and the SPT wheal identification grading annotation information comprises annotation results of the presence or absence of SPT wheal, the size and / or size grading of the SPT wheal; The camera (204) is in communication connection with the image analysis device, and is configured to collect images of the patient's arm placed on the arm-to-be-inspected bearing base (201) before and after the skin point prick test to obtain the SPT sample image sequence of the target patient, and transmit the image collection result to the image analysis device in real time; The image analysis device is configured to execute the SPT wheal identification grading method according to any one of claims 1-4.

7. The SPT wheal identification grading system of claim 6, wherein, The SPT diagnosis and treatment table further comprises an LED light source (205) and a brightness sensor (206), wherein the LED light source (205) is installed on the horizontal bar (203) and the light-emitting surface faces the arm-to-be-inspected bearing base (201), and the brightness sensor (206) is installed at the lower part of the vertical rod (202) and the light-sensitive surface also faces the arm-to-be-inspected bearing base (201); The brightness sensor (206) is in communication connection with the image analysis device, and is configured to synchronously collect brightness information when the camera (204) collects images, and transmit the brightness collection result to the image analysis device in real time, so that the image analysis device uses the brightness information to correct the synchronously collected images when preprocessing the SPT sample image sequence of the target patient.

8. A computer device, comprising: The SPT wheal identification grading method according to any one of claims 1-4 is executed by a computer including a memory, a processor and a transceiver connected in sequence, wherein the memory is configured to store a computer program, the transceiver is configured to transceive messages, and the processor is configured to read the computer program.

9. A computer-readable storage medium, characterized in that The computer readable storage medium stores instructions, and when the instructions are executed on the computer, the SPT wheal identification grading method according to any one of claims 1-4 is executed.

10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or the instructions realize the SPT wheal identification grading method according to any one of claims 1-4 when executed by a computer.

Images