Patient home subcutaneous injection auxiliary guidance system based on smart phone

Abstract

The invention relates to the field of telemedicine, and particularly discloses a patient home subcutaneous injection auxiliary guidance system based on a smart phone, comprising a smart phone terminal, the smart phone terminal comprising: a storage module for storing a preset standardized operation process for patient home subcutaneous injection; the demonstration module is used for displaying a complete demonstration video of the standardized operation process to a patient when a first injection request is received; the acquisition module is used for acquiring a skin image of the patient and an operation video of the whole process of home injection after the complete demonstration video is played; the processing module is used for preprocessing the skin image, calling a pre-trained skin lesion recognition deep learning model, extracting features of the area, the erythema degree and the edema level of the skin lesion, inputting the features into a preset follow-up visit demand prediction model, and outputting a follow-up visit priority and suggested follow-up visit time; by adopting the technical scheme of the invention, standardized operation guidance, real-time interactive evaluation, dynamic feedback and intelligent follow-up visit can be integrated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of telemedicine, in particular to a patient home subcutaneous injection auxiliary guidance system based on a smart phone. BACKGROUND

[0002] With the wide application of biological agents in the treatment of chronic diseases such as psoriasis, home subcutaneous injection has become the preferred administration method for patients due to its convenience. However, at present, patients mainly rely on paper instructions, one-time oral training or general videos for injection operation learning, which cannot provide dynamic guidance for individual operation habits, resulting in insufficient understanding of patients on key links such as injection site selection, aseptic operation and drug preservation. For example, psoriasis patients often cause injection site reaction (ISR) due to insufficient pre-injection assessment and non-standard operation, with an incidence rate of 0.5-40%. ISR is the abbreviation of "Injection Site Reaction", which refers to the local adverse reactions of the injection site and the surrounding skin of the patient after subcutaneous injection of drugs (such as biological agents for treating psoriasis), and is a common clinical phenomenon in subcutaneous injection administration.

[0003] The lack of real-time evaluation mechanism for the standardization of patient home injection operation makes it difficult for medical staff to find details such as non-standard handling of sharp instruments after not staying for 5-10 seconds after injection, resulting in missed reporting of adverse reactions and fluctuation of treatment effect.

[0004] The standardization operation system without integrating expert consensus has great differences in guidance content among different medical institutions, and has not combined with disease specificity, such as long-term maintenance treatment for psoriasis patients and individualized program design for injection site selection according to the state of skin lesions. The patient compliance significantly decreases with the extension of treatment cycle, and the compliance reduction rate reaches 60% after half a year.

[0005] Traditional follow-up relies on manual telephone reminders, which cannot combine with patient injection operation data such as injection time, site and adverse reaction symptoms for precise intervention, resulting in low follow-up efficiency, delayed adverse reaction treatment and increased serious risk.

[0006] Therefore, an integrated system of standardized operation guidance, real-time interactive evaluation, dynamic feedback and intelligent follow-up is needed to solve the problems of non-standard operation, lagging evaluation and fragmented management in home subcutaneous injection, and to improve the safety of patient medication and treatment compliance. SUMMARY

[0007] The present application provides a patient home subcutaneous injection auxiliary guidance system based on a smart phone, which can integrate standardized operation guidance, real-time interactive evaluation, dynamic feedback and intelligent follow-up.

[0008] To solve the above technical problems, the present application provides the following technical solutions: The patient home subcutaneous injection auxiliary guidance system based on a smart phone comprises a smart phone terminal, the smart phone terminal comprises: A storage module is configured to store a preset standardized operation process for patient home subcutaneous injection; A demonstration module is configured to show a complete demonstration video of the standardized operation process to the patient when receiving a first injection request; A collection module is configured to collect a skin image and a home injection whole-process operation video of the patient after playing the complete demonstration video; A processing module is configured to pre-process the skin image, call a pre-trained skin lesion recognition deep learning model, extract area, erythema degree, and edema grade features of the skin lesion, input the features into a preset follow-up demand prediction model, and output a follow-up priority and a recommended follow-up time; Then, the operation video is analyzed in multiple dimensions, the video is segmented into a preset number of key time sequence segments according to steps of the standardized operation process, operation features are extracted for each time sequence segment, the operation features include an area coverage rate of skin disinfection obtained by calculating a contact area ratio of a disinfection cotton piece and the skin through a semantic segmentation algorithm, a needle insertion angle obtained by calculating an included angle between a needle and a skin plane, a push injection speed obtained by calculating a ratio of a moving distance of a syringe push rod and time through a frame difference method, and a needle removal dwell time obtained by identifying a time difference between complete insertion of the needle and needle removal through a target detection algorithm; The features of each time sequence segment are compared with threshold ranges of the standardized operation process in the storage module, and compliance of a single step is output; a non-compliant step is weighted and scored in combination with an error frequency of the same step in historical operations of the patient, and a step risk value is obtained; A feedback module is configured to generate feedback information according to the follow-up priority and the step risk value.

[0009] The basic scheme principle and beneficial effects are as follows: the storage module in the present application presets a subcutaneous injection standard process, the demonstration module visualizes operation steps through a complete video, ensures that the patient obtains uniform and standardized operation instructions, and solves the problem of lack of professional on-site guidance for home injection. The collection module synchronously obtains a skin image and an operation video, the processing module performs time sequence segmentation and feature extraction on the video, such as disinfection coverage rate, needle insertion angle, etc., and compares the features with standardized threshold values in real time, so that dynamic monitoring and quantitative evaluation of the operation process are realized.

[0010] The feedback module generates personalized feedback according to operation compliance, step risk value and skin state, follow-up priority, corrects the current operation deviation, and predicts potential risks such as infection risk, and forms a closed loop. Through the follow-up priority and time suggestion output by the lesion identification model, the follow-up plan is automatically planned, so that the medical resource allocation is more accurate, such as high-risk patients being preferentially followed up, and the overall management efficiency is improved.

[0011] The demonstration module supports automatic marking of high-risk steps according to patient historical error records, such as incomplete disinfection, and preferentially displays the segment and superimposes voice warnings next time.

[0012] More than 90% of data processing is completed on a smart phone, including lesion identification, operation video analysis, and only the anonymized risk assessment result is uploaded to the cloud, so that the risk of patient privacy data leakage is reduced.

[0013] The present application breaks through the limitation of traditional home guidance relying on written instructions or one-way video through the innovation of multi-modal data acquisition, edge intelligent analysis and closed-loop feedback intervention, realizes real-time quantitative evaluation and dynamic guidance of the operation process. Especially in the lesion identification and operation feature extraction link, multi-dimensional algorithm fusion is adopted, that is, semantic segmentation, frame difference method and target detection, so that the evaluation accuracy reaches the clinical usable level, and the privacy safety is ensured through edge computing, providing a scalable and replicable intelligent management scheme for chronic disease patients at home.

[0014] In summary, the present application realizes the effects of integrating standardized operation guidance, real-time interactive evaluation, dynamic feedback and intelligent follow-up.

[0015] Further, the demonstration module is also used to retrieve patient historical operation error records when receiving a non-first injection request, and only display the demonstration video segments corresponding to the operation steps with error rates exceeding a preset value in the records; the feedback information includes follow-up reminders and recommended follow-up times, labels of non-compliant steps and operation optimization suggestions.

[0016] Further, the preprocessing of the skin image by the processing module includes: performing illumination equalization processing on the skin image to eliminate environmental light interference, and then preliminarily positioning the suspected lesion area through an adaptive threshold segmentation algorithm; the pre-trained lesion identification deep learning model is an improved ResNet model optimized for psoriasis lesions. When extracting the area, erythema degree and edema grade features of the lesion, the skin image corresponding to the body part is first identified through image feature matching, and then the area feature is weighted and calculated in combination with the weight coefficient of the part in the psoriasis area and severity index score. At the same time, when extracting the erythema degree and edema grade features, the lesion identification deep learning model synchronously identifies whether there are features related to injection site reactions in the image.

[0017] Furthermore, the processing module is also used to extract the weighted value of the skin lesion area, the quantitative value of the degree of erythema, the quantitative value of the degree of edema, and the average of the step risk values ​​of the last three injection operations recorded by the acquisition module; The follow-up demand prediction model outputs the follow-up priority and recommended follow-up time in the following manner: the input parameters are standardized and weighted using a dynamic weight allocation algorithm, with the weighted value of the lesion area accounting for 40%, the quantitative value of the erythema degree accounting for 20%, the quantitative value of the edema grade accounting for 15%, and the average step risk value accounting for 25%. The weight of the step risk value of the most recent injection is adjusted according to the time decay coefficient; then the comprehensive risk score is calculated, and when the score exceeds the preset maximum value, a high priority is output, and the recommended follow-up time is within 3 days. At the same time, the demonstration module is triggered to automatically add a slow-motion demonstration clip of the high-risk step when the next injection request is made; when the score is within the preset interval, a medium priority is output, and the recommended follow-up time is 7-14 days; when the score is lower than the preset minimum value, a low priority is output, and the recommended follow-up time is 1-3 months; when it is recognized that the weighted value of the lesion area increases by more than a preset threshold within two consecutive follow-up intervals on the same body part, the priority is automatically increased by one level, and the recommended follow-up time is shortened.

[0018] Furthermore, the video is divided into a preset number of key timing segments according to the steps of the standardized operation process: divided into 7 key timing segments, corresponding to pre-injection preparation, skin cleaning, skin disinfection, injection site selection, needle insertion, injection of liquid medicine, needle removal and waste disposal; during the segmentation process, the processing module monitors the continuity of the operation actions in the video in real time through the inter-frame difference method. If the operation interruption time of a certain timing segment exceeds the preset time, the demonstration module is automatically triggered to play the standardized operation video segment corresponding to the segment and superimpose text prompts, and the reason for the interruption is recorded in the patient's operation file.

[0019] Furthermore, when the semantic segmentation algorithm is used to calculate the proportion of the contact area between the disinfection cotton pad and the skin, the processing module synchronously calls the infrared thermal imaging analysis function based on the RGB-IR dual-channel imaging of the smartphone camera to identify the temperature changes in the contact area of ​​the disinfection cotton pad. If the contact area coverage rate reaches the preset value but the temperature change does not reach the threshold, it is determined to be a formal disinfection, and the demonstration module immediately plays a video of the correct disinfection technique and provides a voice prompt; at the same time, the result is associated with the sterile operation compliance score.

[0020] Furthermore, when the processing module calculates the angle between the needle tip and the skin plane, it detects the smartphone's camera viewing angle offset angle θ in real time; and uses the skin texture deformation compensation model to correct the needle insertion angle: Correction angle = original angle × cosθ + skin elasticity coefficient × |90° - original angle|; If the corrected angle deviates from the standard range for a duration exceeding the preset time, an automatic vibration alarm is triggered and a standard angle trajectory line is projected on the screen. The processing module constructs a three-dimensional coordinate system through a binocular vision algorithm, and displays the needle-skin plane angle value in real time on the operation interface of the smartphone screen. If the needle insertion angle deviates from the preset standard range, in addition to outputting the angle deviation prompt, the deviation direction is analyzed synchronously, the collection module is triggered to focus on shooting the real-time image of the needle insertion site, which is used for subsequent judgment of whether there is a risk of drug liquid exosmosis; at the same time, the angle deviation data is associated with the injection site reaction risk model, and the frequency of monitoring the site next time is automatically increased when the angle is abnormal.

[0021] Further, the processing module, before comparing the characteristics of each time sequence segment with the threshold range of the standardized operation process in the storage module, first constructs a high-risk operation prediction model based on the patient's historical operation data, including the error frequency, step risk value and corresponding PASI score change of each time sequence segment. The high-risk operation prediction model outputs the high-risk time sequence segment in which the patient may appear non-compliant operation in this injection through a machine learning algorithm, and sorts them according to the risk probability; The processing module, before outputting the compliance of a single step, pushes personalized guidance information in advance through the demonstration module for the predicted high-risk time sequence segment; After obtaining the compliance result of a single step, if the predicted high-risk time sequence segment is still determined to be non-compliant, the processing module automatically analyzes whether the operation video obtained by the collection module has quality problems: including detecting whether the video is blurred due to hand shaking through an image clarity evaluation algorithm, and determining whether the feature extraction is incomplete due to shooting angle deviation through a key operation area detection algorithm; If there is a video quality problem, the processing module immediately switches the guidance mode, calls the pre-stored 3D animation demonstration to replace the video comparison for the step corresponding to the blurred video, generates shooting angle correction instructions for the step with angle deviation, and synchronously pushes the decomposition diagram of the step; if the video quality problem is excluded, the processing module determines that there is an operation cognitive bias, triggers the demonstration module to play the preset deep analysis video of the step, and records the bias type to the patient's cognitive archive for optimizing the guidance strategy next time.

[0022] Furthermore, the processing module is also used to obtain the risk value of the current step in real time. If the step risk value reaches a preset high-risk threshold; the follow-up priority output by the skin image evaluation unit is high; the acquisition module synchronously collects the patient's voice information during the operation, and the processing module performs semantic analysis on the voice information to extract the help keywords or confused tone features contained therein; when any of the above conditions is met, the processing module automatically triggers the remote guidance warning and pushes the pre-processed help information to the doctor terminal. The help information includes: the patient's skin lesion characteristics of this injection, the operation video clip of the high-risk step, the historical correction record of similar errors and the current follow-up priority.

[0023] Furthermore, the processing module is further configured to capture a real-time image of the patient using the acquisition module. The processing module analyzes the patient's emotional state using a facial expression recognition algorithm to identify whether there are positive facial features indicating acceptance of guidance. The processing module also performs background analysis on the image to determine whether the operating environment meets the basic conditions for remote guidance. Combined with the results of the patient's speech and semantic analysis, if no rejection-related words appear in the speech and the image analysis meets the conditions, it is determined that remote guidance can be connected. Otherwise, the remote guidance request is temporarily stored and a delayed confirmation option is pushed through the feedback module. After the remote guidance is connected, the processing module automatically starts the recording function and performs structured marking on the remote guidance process of the doctor's terminal; after the recording is completed, the processing module associates and stores the video with the error type of the patient's operation and the characteristics of the skin lesion site; When the same patient subsequently triggers the same remote guidance needs, the processing module will give priority to retrieving the matching historical recorded video and playing it in picture-in-picture format through the demonstration module; if the patient indicates through voice or operation feedback that real-time guidance is still needed, the processing module will transfer the call to the doctor's terminal and simultaneously push the historical video to the doctor's terminal as a reference; at the same time, the processing module regularly performs standardized verification on the recorded video to ensure that the content is consistent with the latest operating specifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a logical block diagram of an embodiment of a smartphone-based patient home subcutaneous injection auxiliary guidance system. DETAILED DESCRIPTION

[0025] The following is further described in detail through specific implementation methods: Smartphone-based home subcutaneous injection guidance system for patients (such as Figure 1 As shown), including a smart phone terminal, the smart phone terminal includes: A storage module, used for storing preset standardized operating procedures for subcutaneous injections at home; A demonstration module, configured to show a complete demonstration video of the standardized operating procedure to the patient upon receiving a first injection request; The collection module is used for collecting the skin image and the whole process of home injection operation video of the patient after playing the complete demonstration video; The processing module is used for pre-processing the skin image, calling a pre-trained skin lesion recognition deep learning model, extracting area, erythema degree, and edema grade features of the skin lesion, inputting the features into a preset follow-up demand prediction model, and outputting a follow-up priority and a recommended follow-up time. Then, the operation video is analyzed in multiple dimensions, the video is segmented into a preset number of key time segments according to the steps of the standardized operation process, operation features are extracted for each time segment, the operation features include the area coverage rate of skin disinfection obtained by calculating the contact area ratio of the disinfectant cotton piece and the skin through a semantic segmentation algorithm, the needle insertion angle is calculated by calculating the angle between the needle and the skin plane, the injection speed is obtained by calculating the ratio of the moving distance of the syringe plunger to the time through the frame difference method, and the needle removal dwell time is obtained by identifying the time difference between the complete insertion of the needle and the needle removal through the target detection algorithm. The features of each time segment are compared with the threshold range of the standardized operation process in the storage module, and the compliance of a single step is output; the non-compliant steps are weighted and scored according to the error frequency of the same step in the patient's historical operation, and a step risk value is obtained. The feedback module is used for generating feedback information according to the follow-up priority and the step risk value.

[0026] Of course, a medical terminal can also be included, which is used to receive feedback information and realize automatic follow-up reminders in combination with a calendar.

[0027] In specific use, the following will be described in detail in combination with the scene of home subcutaneous injection of biological agents for psoriasis patients.

[0028] It is run on a smart phone with Android 12 or iOS 16 and above system, such as Huawei Mate60, iPhone15, hardware supports rear 4800 million pixel camera + infrared camera (additional purchase and communication with smart phone), gyroscope and loudspeaker, software is integrated into a special medical APP, such as psoriasis injection assistant.

[0029] The storage module is preloaded with a standardized operation process for home subcutaneous injection of biological agents for psoriasis patients (previously shot and marked and split for easy understanding by patients), including 7 core steps and quantitative standards, such as skin disinfection requiring circular wiping with a diameter of ≥5cm, needle insertion angle of 15-30° (recommended standard), needle removal dwell time of 5-10 seconds, etc., stored in the local encrypted partition of the phone to prevent data leakage.

[0030] When the demonstration module processes a first injection request, the first injection request is initiated through the user interface of the smartphone terminal when the patient uses the system for the first time to perform a home subcutaneous injection. For example, after the patient opens the system application on the smartphone, the patient clicks the first injection button.

[0031] After the demonstration module receives the first injection request, the demonstration module retrieves a complete standardized operation process demonstration video from the storage module. The demonstration video details the entire process from preparation before injection to disposal of waste, including key steps such as skin cleaning, disinfection, injection site selection, needle insertion, drug injection, and needle removal.

[0032] The demonstration module displays the complete demonstration video to the patient. For example, the video clearly shows the operation details of each step, such as the correct use of disinfectant cotton, the optimal needle insertion angle with the skin plane, etc., through a combination of animation and live demonstration, helping the patient to familiarize themselves with the entire injection process.

[0033] For patients who are not injecting for the first time, when they initiate an injection request on the smartphone, the demonstration module retrieves the patient's historical operation error records. For example, the system records errors such as insufficient coverage of the skin disinfection area and inaccurate needle insertion angle during previous injection processes.

[0034] The demonstration module only displays the demonstration video segments corresponding to the operation steps with error rates exceeding the preset value in the records. Assuming that the system's preset error rate threshold is 20%, if the patient's error rate for the skin disinfection step in the historical operation is 25%, the demonstration module only displays the demonstration video segment for the skin disinfection step when the patient initiates the injection request.

[0035] The demonstration video segments highlight the error points and provide operation optimization suggestions. For example, in the skin disinfection demonstration segment, the patient is prompted through arrows and text to cover the skin area range with disinfectant cotton and the correct wiping technique, helping the patient to correct the error operation.

[0036] After the demonstration module displays the complete demonstration video, the acquisition module starts the skin image acquisition function. The patient aims the smartphone camera at the skin of the injection site and prompts the patient to adjust the camera position to ensure that the skin of the injection site is clearly and completely displayed in the camera view.

[0037] The acquisition module acquires the patient's skin image. For example, before preparing for injection, the patient aims the smartphone camera at the abdominal skin, and the system acquires an image containing information such as the texture and color of the abdominal skin for subsequent skin lesion identification analysis.

[0038] The acquisition module simultaneously starts the home injection operation video acquisition function. The patient performs the injection operation according to the standardized operation process in the demonstration video, and the smartphone camera records the entire operation process in real time.

[0039] For example, from the beginning of the preparation of the syringe from the patient, to the disposal after the completion of the injection (of course, when the waste disposal is qualified, a prompt message can also be generated), the collection module shoots the operation video throughout the process. The video records the operation details of the patient at each step of skin cleaning, disinfection, needle insertion, drug injection, and needle removal, providing a data basis for subsequent multi-dimensional analysis.

[0040] The processing module pre-processes the collected skin image. First, the skin image is subjected to illumination equalization processing to eliminate environmental light interference. For example, in the case of indoor light being relatively dark, the system adjusts the brightness and contrast of the image through the illumination equalization algorithm, making the details of the skin image clearer.

[0041] Then, the suspected skin lesion area is preliminarily located through an adaptive threshold segmentation algorithm. For example, the system automatically adjusts the threshold value based on the gray scale distribution of the skin image, preliminarily identifying the area with a larger color and texture difference from the surrounding normal skin as a suspected skin lesion area.

[0042] The processing module calls a pre-trained improved ResNet model optimized for psoriasis lesions. The improved ResNet model (lesion recognition deep learning model) first identifies the body part corresponding to the skin image through image feature matching when extracting the area, erythema degree, and edema grade features of the lesion.

[0043] Specifically, the improved ResNet model optimized for psoriasis lesion recognition is a deep learning model that has been improved in four dimensions: input layer, feature extraction layer, output layer, and training strategy, based on the original ResNet (such as ResNet-50), combined with the clinical features of psoriasis lesions (such as blurred erythema boundary, scale coverage, and multi-site distribution difference) and mobile deployment requirements. The original ResNet only accepts RGB single-channel images, and the improved model is expanded to RGB-IR dual-channel input. Among them, the RGB channel retains the color features of the lesion (such as the gradual change from light red to deep red of the erythema), and the near-infrared channel enhances the gray difference between the scale and the normal skin (the scale has higher reflectivity in the near-infrared, making it easier to distinguish from the erythema area).

[0044] The input layer is embedded with a Retinex-based illumination equalization subnetwork that automatically detects the backlight and shadow areas in the skin image (such as the color distortion of the erythema caused by uneven light at the patient's injection site). By decomposing the illumination component and the reflection component, the image contrast is corrected to ensure that the quantification of the erythema degree and edema grade is not affected by environmental light interference.

[0045] A lightweight semantic segmentation branch (based on the simplified structure of U-Net) is added to quickly locate the suspected lesion area (such as a red spot with a diameter of >0.5 cm on the torso) at the early stage of model forward propagation, and to mask the background (such as clothes and bed sheets) through masking operation to reduce the interference of invalid information on feature extraction.

[0046] The residual block of the original ResNet adopts the structure of convolution + BN + ReLU. The improved model adds a dilated convolution branch after the convolution layer, sets different dilation rates (1, 2, 4) for the 3 × 3 convolution kernel, and captures multi-scale features of psoriasis lesions (such as small area of scaly spots and large area of confluent erythema) through multi-scale receptive field. For example, the convolution layer with a dilation rate of 4 can cover a confluent erythema with a diameter of >2 cm, and the convolution layer with a dilation rate of 1 focuses on a scaly spot with a diameter of <0.5 cm.

[0047] The skip connection of the residual block introduces feature weighting fusion, giving dynamic weights to low-level features (edge, texture) and high-level features (semantic) (such as enhancing the texture feature weight of the scaly area), solving the boundary blur problem caused by equal fusion of different level features in the original ResNet.

[0048] A channel attention layer is added after the output of each residual block, which strengthens the feature channels related to psoriasis (such as RGB channels reflecting the color of erythema and near-infrared channels reflecting the thickness of scales) through the squeeze-excitation (SE) mechanism and suppresses irrelevant channels (such as background noise channels). This can improve the IoU value of erythema area recognition.

[0049] A spatial attention layer is added after the channel attention for the spatial distribution characteristics of the center erythema + edge scales of psoriasis lesions, which generates a spatial weight map (high weight for erythema center and low weight for normal skin) to guide the model to focus on the core area of the lesion and reduce the interference of normal skin texture (such as skin wrinkles).

[0050] The original ResNet is a single classification output, and the improved model designs 3 parallel output heads. The regression head 1 outputs the lesion area, adopts the L1 loss function, and adds the Dice loss to optimize the edge prediction accuracy according to the irregularity of the psoriasis lesion boundary. The classification head 2 outputs the erythema degree (5-level classification: none, mild, moderate, severe, and extremely severe), combining the cross-entropy loss and the Focal Loss (to solve the imbalance problem of mild / severe samples). The classification head 3 outputs the edema grade (3-level classification: none, mild, and moderate), and simultaneously identifies the injection site reaction (ISR) features (such as local redness within 24 hours after injection), through the multi-task shared feature extraction layer, and uses the correlation between tasks (such as severe erythema often accompanied by edema) to improve the overall prediction accuracy.

[0051] To address the different weightings of psoriasis scores on different body parts (e.g., 10% for scalp, 30% for trunk in PASI score), a part feature embedding is added before the output layer, i.e., first determine the part of the lesion through image feature matching (e.g., hair follicle density identifies scalp, skin texture identifies limbs); then weight the extracted area features according to the part weight (e.g., trunk lesion area x 30%, limbs x 20%), so that the output area features directly adapt to the PASI score calculation requirements, avoiding secondary weighting errors in subsequent processing modules.

[0052] To address the scarcity of psoriasis annotation data (especially the fine annotation of lesions at different stages), generative data augmentation is used to generate synthetic lesion images under different lighting and angles based on StyleGAN (while preserving the erythema-scales distribution features); then perform "elastic deformation" (simulate the effect of skin wrinkles on lesion morphology) and "scale superposition" (superimpose scales textures of different thickness on the erythema area through texture migration technology) on real images, so as to expand the training sample size and alleviate overfitting.

[0053] The original ResNet is based on ImageNet pre-training, and the improved model uses a medical domain pre-training + task fine-tuning strategy, i.e., first pre-train on a large-scale skin disease image dataset to make the model learn general features of skin and lesions; then fine-tune on a psoriasis special dataset (containing more than 100,000 clinically annotated images) to update the attention module and output layer parameters, so that the model quickly adapts to the specific features of psoriasis (such as oyster shell scales, homomorphous reaction lesions).

[0054] These improvements directly address the adaptation of the original ResNet in scenarios such as small medical samples, complex lesion features, and limited mobile deployment, providing precise lesion feature data support for subsequent follow-up priority prediction and operation guidance.

[0055] For example, the injection site in the patient's skin image is identified as the abdomen, and the extracted area features are weighted and calculated according to the weight coefficient of the abdomen in the Psoriasis Area and Severity Index score. Assuming the weight coefficient of the abdomen is 0.3, and the model extracts a lesion area of 2 square centimeters, the weighted lesion area is 0.6 square centimeters.

[0056] At the same time, when extracting erythema degree and edema grade features, the model simultaneously identifies whether there are features related to the injection site reaction in the image. For example, the model identifies that there is mild erythema and slight edema around the injection site, with erythema degree quantization value of 0.2 and edema grade quantization value of 0.1.

[0057] The processing module performs multi-dimensional analysis on the collected full-process home injection operation video. First, the video is segmented into a preset number of key time segments according to the steps of the standardized operation process, specifically into 7 key time segments, corresponding to injection preparation, skin cleaning, skin disinfection, injection site selection, needle insertion operation, drug injection, needle removal, and waste disposal.

[0058] During segmentation, the continuity of operation actions in the video is monitored in real time by the inter-frame difference method. For example, when the patient is performing the skin disinfection step, if the operation interruption time exceeds the preset time (such as 30 seconds), the demonstration module is triggered to play the standardized operation video segment corresponding to this segment and superimpose a text prompt to continue the skin disinfection operation as soon as possible, while recording the interruption reason (such as the patient answering the phone) to the patient operation archive.

[0059] Operation feature extraction: extract operation features for each time segment.

[0060] Skin disinfection area coverage: calculate the contact area ratio of the disinfection cotton piece and the skin by the semantic segmentation algorithm. For example, it is identified that the pixel ratio of the contact area of the disinfection cotton piece and the skin is 80%, reaching the preset threshold (such as 70%).

[0061] At the same time, the processing module synchronously calls the infrared thermal imaging analysis function based on the RGB-IR dual-channel imaging of the smartphone camera (which requires the user to purchase additionally and obtain the user's permission; if some models of smartphones do not support this function, or the user does not purchase it, the change in pixels can also be analyzed, i.e. the alcohol contact area changes from diffuse reflection to specular reflection, which has applications in live face recognition), to identify the temperature change of the contact area of the disinfection cotton piece. If the contact area coverage reaches the preset value, but the temperature change does not reach the threshold (such as a temperature change of less than 0.5 degrees Celsius), it is determined as formal disinfection, and the demonstration module immediately plays the correct disinfection technique video and voice prompts to ensure that the disinfection cotton piece is in full contact with the skin and stays for enough time, while associating the result to the sterile operation compliance score.

[0062] The needle insertion angle is the angle between the needle and the skin plane. The processing module detects the smartphone camera's shooting angle offset angle θ in real time, and corrects the needle insertion angle through a skin texture deformation compensation model. For example, it is detected that the shooting angle offset angle is 10°, the original needle insertion angle is 30°, and the skin elasticity coefficient is 0.1, then the corrected angle is 30°×cos10°+0.1×|90°-30°|, which is about 35.5°.

[0063] The corrected angle deviates from the standard range (e.g., 15°-30° is the recommended standard), and exceeds the preset time (e.g., 5 seconds), triggering a vibration alarm and projecting a standard angle trajectory line on the screen. At the same time, the processing module constructs a three-dimensional coordinate system through binocular vision algorithm, and displays the needle-skin plane angle value in real time on the operation interface of the smartphone screen.

[0064] If the needle insertion angle deviates from the preset standard range, in addition to outputting the angle deviation prompt, the deviation direction is analyzed synchronously, triggering the acquisition module to focus on shooting the real-time image of the needle insertion site for subsequent judgment of whether there is a risk of drug exosmosis. At the same time, the angle deviation data is associated with the injection site reaction risk model, and the monitoring frequency of the site is automatically increased when the angle is abnormal.

[0065] The injection speed is obtained by calculating the ratio of the moving distance of the syringe plunger to the time through frame difference method. For example, the system calculates the injection speed as 0.5 cm / s, which is compared with the preset threshold range (e.g., 0.3-0.7 cm / s) in the standardized operation process.

[0066] The needle withdrawal dwell time is obtained by recognizing the time difference between the needle being completely inserted and withdrawn through the target detection algorithm. For example, the needle withdrawal dwell time is recognized as 10 seconds, which is compared with the preset threshold range (e.g., 8-12 seconds).

[0067] Compliance analysis and risk assessment: compare the characteristics of each time sequence segment with the threshold range of the standardized operation process in the storage module, and output the compliance of each step. For example, the skin disinfection area coverage rate of 80% meets the threshold range, and it is determined that the step is compliant; if the needle insertion angle deviates from the standard range, it is determined that the step is not compliant.

[0068] Combine the error frequency of the same step in the patient's historical operation to give a weighted score to the non-compliant step, and get the step risk value. Assuming that the patient's needle insertion angle error frequency in the historical operation is 5 times, the system calculates the risk value of this step as 0.8 (full score 1) according to the weighted scoring algorithm.

[0069] The processing module extracts the lesion area weighted value 0.6, the erythema degree quantification value 0.2, the edema grade quantification value 0.1, and the average value 0.5 of the step risk value in the last 3 injection operations recorded by the acquisition module.

[0070] The model standardizes the input parameters and assigns weights using a dynamic weight allocation algorithm. The lesion area weighted value accounts for 40%, the erythema degree quantification value accounts for 20%, the edema grade quantification value accounts for 15%, and the average value of the step risk value accounts for 25%. Assuming that the step risk value of the last injection is 0.6, the adjusted weight is 0.27 according to the time decay coefficient 0.9, and the comprehensive risk score is calculated.

[0071] According to the score, the follow-up priority and the recommended follow-up time are determined: if the score exceeds the preset maximum value (such as 0.7), a high priority is output, and the recommended follow-up time is within 3 days. At the same time, the demonstration module is triggered to automatically add a slow-motion demonstration segment of a high-risk step when the next injection is requested. If the score is in the preset interval (such as 0.4-0.7), a medium priority is output, and the recommended follow-up time is 7-14 days. If the score is lower than the preset minimum value (such as 0.4), a low priority is output, and the recommended follow-up time is 1-3 months.

[0072] If it is identified that the weighted value of the lesion area in the interval between two consecutive follow-ups of the same body part increases by more than a preset threshold (such as 0.2), the priority is automatically upgraded by one level, and the recommended follow-up time is shortened.

[0073] According to the follow-up priority and step risk value output by the processing module, the feedback module generates feedback information. For example, if the comprehensive risk score is 0.5, it is determined to be a medium priority, and the recommended follow-up time is 10 days.

[0074] The feedback information includes follow-up reminders and recommended follow-up times, labels of non-compliant steps, and operation optimization suggestions. For example, the feedback information shows: In your current injection operation, the needle insertion angle does not meet the standard range, and it is recommended that you refer to the slow-motion demonstration video pushed by the system for operation in the next injection.

[0075] For predicted high-risk time segments, personalized guidance information is pushed in advance through the demonstration module. For example, if it is predicted that the patient has a high risk in the drug injection step, detailed operation videos and precautions for this step are pushed in advance.

[0076] If the predicted high-risk time segment is still determined to be non-compliant, the processing module analyzes whether there is a quality problem in the operation video. For example, through an image clarity evaluation algorithm, it is detected whether the video is blurred due to hand shaking, and through a key operation area detection algorithm, it is determined whether the feature extraction is incomplete due to the deviation of the shooting angle.

[0077] If there is a video quality problem, the processing module switches the guidance mode. For example, for the steps corresponding to the blurred video, pre-stored 3D animation demonstrations are called instead of video comparison; for the steps with angle deviation, shooting angle correction guidelines are generated and the decomposition diagram of the step is pushed synchronously.

[0078] If the video quality problem is ruled out, the processing module determines that there is an operation cognitive bias, triggers the demonstration module to play a deep analysis video of the step, and records the bias type to the patient's cognitive archive for optimizing the guidance strategy next time.

[0079] When the processing module obtains the current step risk value in real time and reaches the preset high-risk threshold (such as 0.8), or the follow-up priority output by the skin image evaluation unit is high, or the acquisition module synchronously collects the patient's voice information during the operation, the processing module performs semantic analysis on the voice information and extracts the help keywords or confused tone features contained therein, and automatically triggers the remote guidance warning.

[0080] For example, during the injection process, the patient may use help keywords such as: I don’t know how to adjust this angle or how to do this (a voice recognition library is preset, which can be achieved through the existing networked voice assistant and will not be repeated here), triggering a remote guidance warning.

[0081] The pre-processed help information is pushed to the doctor's terminal, including the characteristics of the patient's skin lesions during this injection, video clips of high-risk steps, historical correction records of similar errors, and current follow-up priorities.

[0082] The processing module collects real-time images of the patient (specifically, by collecting facial images similar to those during video chats and performing facial expression recognition). Using facial expression recognition algorithms, it analyzes the patient's emotional state and identifies any positive facial expressions that indicate a willingness to receive guidance. For example, it identifies a patient's facial expression as focused and without anxiety.

[0083] Perform background analysis on the image (i.e., analyze whether the patient's environment is bright enough and whether there are fast-moving objects) to determine whether the operating environment meets the basic requirements for remote guidance. For example, check that the operating environment is well-lit and free of distractions.

[0084] Combined with the results of the patient's speech and semantic analysis, if there are no rejection-related words in the speech and the image analysis meets the conditions, it is determined that remote guidance can be connected; otherwise, the remote guidance request is temporarily stored and a delayed confirmation option is pushed through the feedback module.

[0085] After the remote guidance is connected, the processing module automatically starts the recording function and performs structured marking on the remote guidance process of the doctor's terminal (there can be many ways to process structured marking, for example, it can be split by marking words. In this embodiment, the splitting is done by extracting the audio track, performing text recognition on the voice content, and obtaining the split time nodes on the audio track based on the similarity after semantic aggregation, and then marking and splitting the entire remote guidance process).

[0086] After recording, the video is stored in association with the error type of the operation of the patient and the characteristics of the lesion site. When a subsequent patient triggers the same type of remote guidance requirement, the processing module preferentially retrieves the matching historical recording video and plays it through the demonstration module in picture-in-picture form. If the patient indicates through voice or operation feedback that real-time guidance is still needed, the processing module switches to the doctor terminal and synchronously pushes the historical video to the doctor terminal as a reference.

[0087] Meanwhile, the processing module periodically checks the recorded video to ensure that the content is consistent with the latest operation specification (i.e., updates the operation video). Through the specific implementation of the above modules, the patient home subcutaneous injection auxiliary guidance system based on a smartphone according to the present application can provide comprehensive and personalized home injection auxiliary guidance for patients, improve the safety and accuracy of patient home injection, and provide remote guidance support for doctors, thereby optimizing the treatment experience of patients.

[0088] The above is only an embodiment of the present application, and the present application is not limited to the field involved in this embodiment. Well-known specific structures and characteristics in the scheme are not described in detail, and the person skilled in the art knows all the ordinary technical knowledge in the field of the present application before the filing date or the priority date, can know all the prior art in the field, and has the ability to apply conventional experimental means before that date. The person skilled in the art can improve and implement the present scheme based on the disclosure given in the present application, and some typical known structures or known methods should not be an obstacle for the person skilled in the art to implement the present application. It should be noted that, for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should also be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application. The protection scope of the present application should be subject to the content of its claims, and the specific implementation in the specification can be used to explain the content of the claims.

Claims

1. A smartphone-based patient-home subcutaneous injection assistance guidance system, characterized in that: A smart phone terminal is included, and the smart phone terminal includes: A storage module, used for storing preset standardized operating procedures for subcutaneous injections at home; A demonstration module, configured to show a complete demonstration video of the standardized operating procedure to the patient upon receiving a first injection request; The acquisition module is used to collect the patient's skin image and the video of the entire home injection process after playing the complete demonstration video; The processing module is used to pre-process the skin image, call the pre-trained skin lesion recognition deep learning model, extract the lesion area, erythema degree, and edema grade features, input the features into the preset follow-up demand prediction model, and output the follow-up priority and recommended follow-up time; The operation video is then subjected to a multi-dimensional analysis, and the video is segmented into a preset number of key time-series segments according to the steps of the standardized operation process. Operational features are extracted for each time-series segment. The operation features include calculating the proportion of the contact area between the disinfection cotton pad and the skin using a semantic segmentation algorithm to obtain the area coverage of the skin disinfection, then calculating the angle between the needle and the skin plane to obtain the insertion angle, and then calculating the ratio of the syringe plunger movement distance to time using the frame difference method to obtain the injection speed. The target detection algorithm is used to identify the time difference between complete needle insertion and needle removal to obtain the needle removal dwell time. Compare the characteristics of each time series segment with the threshold range of the standardized operation process in the storage module to output the compliance of a single step; combine the error frequency of the same step in the patient's historical operation, and perform a weighted score on the non-compliant steps to obtain the step risk value; The feedback module is used to generate feedback information based on the follow-up priority and step risk value.

2. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 1, characterized in that: The demonstration module is also used to retrieve the patient's historical operation error records when receiving a non-first injection request, and only display the demonstration video clips corresponding to the operation steps in the records whose error rate exceeds the preset value; the feedback information includes follow-up reminders and recommended follow-up time, annotations of non-compliant steps and operation optimization suggestions.

3. The patient home subcutaneous injection auxiliary guidance system based on a smartphone according to claim 2, characterized in that: The processing module preprocesses the skin image, including: performing illumination equalization on the skin image to eliminate ambient light interference, and then preliminarily locating the suspected skin lesion area through an adaptive threshold segmentation algorithm; the pre-trained skin lesion recognition deep learning model is an improved ResNet model optimized for psoriasis skin lesions. When extracting the area, erythema degree, and edema grade features of the skin lesions, the skin lesion recognition deep learning model first identifies the body part corresponding to the skin image through image feature matching, and then performs a weighted calculation on the extracted area features in combination with the weight coefficient of the part in the psoriasis area and severity index score. At the same time, when extracting the erythema degree and edema grade features, the skin lesion recognition deep learning model simultaneously identifies whether there are features related to injection site reactions in the image.

4. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 3, characterized in that: The processing module is also used to extract the weighted value of the skin lesion area, the quantitative value of the degree of erythema, the quantitative value of the degree of edema, and the average of the step risk values ​​of the last three injection operations recorded by the acquisition module; The follow-up demand prediction model outputs the follow-up priority and recommended follow-up time in the following manner: the input parameters are standardized and weighted using a dynamic weight allocation algorithm, with the weighted value of the lesion area accounting for 40%, the quantitative value of the erythema degree accounting for 20%, the quantitative value of the edema grade accounting for 15%, and the average step risk value accounting for 25%. The weight of the step risk value of the most recent injection is adjusted according to the time decay coefficient; then the comprehensive risk score is calculated, and when the score exceeds the preset maximum value, a high priority is output, and the recommended follow-up time is within 3 days. At the same time, the demonstration module is triggered to automatically add a slow-motion demonstration clip of the high-risk step when the next injection request is made; when the score is within the preset interval, a medium priority is output, and the recommended follow-up time is 7-14 days; when the score is lower than the preset minimum value, a low priority is output, and the recommended follow-up time is 1-3 months; when it is recognized that the weighted value of the lesion area increases by more than a preset threshold within two consecutive follow-up intervals on the same body part, the priority is automatically increased by one level, and the recommended follow-up time is shortened.

5. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 4, characterized in that: The video is divided into a preset number of key timing segments according to the steps of the standardized operation process, specifically: divided into 7 key timing segments, corresponding to pre-injection preparation, skin cleaning, skin disinfection, injection site selection, needle insertion, injection of liquid medicine, needle removal and waste disposal; during the segmentation process, the processing module monitors the continuity of the operation actions in the video in real time through the inter-frame difference method. If the operation interruption time of a certain timing segment exceeds the preset time, the demonstration module is automatically triggered to play the standardized operation video segment corresponding to the segment and superimpose text prompts, and the reason for the interruption is recorded in the patient's operation file.

6. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 5, characterized in that: When calculating the proportion of the contact area between the disinfection cotton pad and the skin through the semantic segmentation algorithm, the processing module synchronously calls the infrared thermal imaging analysis function based on the RGB-IR dual-channel imaging of the smartphone camera to identify the temperature changes in the contact area of ​​the disinfection cotton pad. If the contact area coverage rate reaches the preset value but the temperature change does not reach the threshold, it is determined to be a formal disinfection, and the demonstration module immediately plays a video of the correct disinfection technique and provides a voice prompt; at the same time, the result is associated with the sterile operation compliance score.

7. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 6, characterized in that: When the processing module calculates the angle between the needle tip and the skin plane, it detects the smartphone's camera viewing angle offset angle θ in real time and uses the skin texture deformation compensation model to correct the needle insertion angle: Correction angle = original angle × cosθ + skin elasticity coefficient × |90° - original angle|; If the angle deviates from the standard range after correction for a period exceeding the preset time, a vibration alarm will be automatically triggered and a standard angle trajectory line will be projected on the screen; The processing module constructs a three-dimensional coordinate system through a binocular vision algorithm, and displays the angle value between the needle and the skin plane in real time on the operating interface of the smartphone screen. If the needle insertion angle deviates from the preset standard range, in addition to outputting an angle deviation prompt, the deviation direction is simultaneously analyzed, and the acquisition module is triggered to focus on capturing real-time images of the needle insertion site for subsequent judgment of whether there is a risk of drug extravasation. At the same time, the angle deviation data is associated with the injection site reaction risk model, and the next monitoring frequency of the site is automatically increased when the angle is abnormal.

8. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 7, characterized in that: Before comparing the features of each time series segment with the threshold range of the standardized operation process in the storage module, the processing module first constructs a high-risk operation prediction model based on the patient's historical operation data, which includes the error frequency, step risk value and corresponding PASI score change of each time series segment. The high-risk operation prediction model outputs high-risk time series segments that may have non-compliant operations during the current injection through a machine learning algorithm and sorts them by risk probability; Before outputting the compliance of a single step, the processing module pushes personalized guidance information in advance through the demonstration module for predicted high-risk time sequence segments; After obtaining the compliance results for a single step, if the predicted high-risk time sequence segment is still judged as non-compliant, the processing module automatically analyzes the operation video obtained by the acquisition module to see if there are any quality issues. This includes using an image clarity assessment algorithm to detect whether the video is blurred due to hand shaking, and using a key operation area detection algorithm to determine whether incomplete feature extraction is caused by shooting angle deviation. If there is a problem with the video quality, the processing module will immediately switch the guidance mode. For the steps corresponding to the blurred video, the pre-stored 3D animation demonstration will be called instead of the video comparison. For the steps with angle deviation, the shooting angle correction guidance will be generated and the decomposition diagram of the step will be pushed synchronously. If the video quality problem is not found, the processing module will determine that it is an operation cognitive deviation, triggering the demonstration module to play the preset in-depth analysis video of the step, and record the deviation type in the patient's cognitive file for optimizing the next guidance strategy.

9. The smartphone-based home subcutaneous injection auxiliary guidance system for patients according to claim 8, characterized in that: The processing module is also used to obtain the current step risk value in real time, if the step risk value reaches a preset high risk threshold; The follow-up priority for the skin image assessment unit output is high; The acquisition module synchronously collects the patient's voice information during the operation, and the processing module performs semantic analysis on the voice information to extract the help keywords or confused tone features contained therein; When any of the above conditions is met, the processing module automatically triggers a remote guidance warning and pushes pre-processed help information to the doctor's terminal. The help information includes: the characteristics of the patient's skin lesions during this injection, video clips of high-risk steps, historical correction records of similar errors, and current follow-up priority.

10. The smart phone-based home subcutaneous injection auxiliary guidance system for patients according to claim 9, characterized in that: The processing module is also used for the acquisition module to capture real-time images of the patient, and the processing module analyzes the patient's emotional state through a facial expression recognition algorithm to identify whether there are positive expression characteristics indicating acceptance of guidance; Perform background analysis on the image to determine whether the operating environment meets the basic conditions for remote guidance. Combined with the results of the patient's speech and semantic analysis, if there are no rejection words in the speech and the image analysis meets the conditions, it is determined that remote guidance can be connected. Otherwise, the remote guidance request is temporarily stored and a delayed confirmation option is pushed through the feedback module. After the remote guidance is connected, the processing module automatically starts the recording function and performs structured marking on the remote guidance process of the doctor terminal; After the recording is completed, the processing module associates and stores the video with the patient's error type and skin lesion location characteristics; When the same patient subsequently triggers the same remote guidance needs, the processing module will give priority to retrieving the matching historical recorded video and playing it in picture-in-picture format through the demonstration module; if the patient indicates through voice or operation feedback that real-time guidance is still needed, the processing module will transfer the call to the doctor's terminal and simultaneously push the historical video to the doctor's terminal as a reference; at the same time, the processing module regularly performs standardized verification on the recorded video to ensure that the content is consistent with the latest operating specifications.

Images