Skin disease precision diagnosis and treatment informatization follow-up visit management method and system

Abstract

The invention relates to the technical field of skin disease detection, discloses a difficult and miscellaneous skin disease screening and diagnosis method, and particularly relates to a skin disease precision diagnosis and treatment informatization follow-up management method and system. The system is provided with a doctor end and a patient end, detection is carried out based on an automatic detection method through a skin image uploaded by the patient end, and a detection result is obtained. The automatic detection method is realized based on a detection model, the detection model is provided with a first branch network and a second branch network, and transmission and fusion between shallow and deep feature maps are realized through the first branch network and the second branch network, so that classification accuracy is realized.

Description

Technical Field

[0001] The present application relates to the technical field of skin disease detection, and is a method for screening and diagnosing difficult and complicated skin diseases, specifically, an information-based follow-up management method and system for accurate diagnosis and treatment of skin diseases. Background Art

[0002] In the medical field, diseases related to the skin are collectively referred to as skin diseases. Such diseases are common and frequently occurring diseases that have a great impact on human health, such as eczema, moles, basal cell carcinoma, and melanoma. As the largest organ in the human body, the importance of the skin lies in its direct contact with the external environment. It is the first barrier to the physiological health of the human body. It can not only defend against pathogens, but also regulate body temperature, absorb and secrete. With the rapid development of today's social economy, the advancement of industrialization, and the constant changes in the natural environment, human production and lifestyles have undergone major changes, and more and more factors have induced and caused skin diseases. Various skin diseases have also emerged, and the incidence rate has also been gradually increasing every year. Today, skin diseases have become the most common diseases in humans.

[0003] As for the treatment of skin diseases, especially complex skin diseases, if the patient is only examined in the hospital when the disease occurs, it is easy to delay treatment due to untimely detection, thus missing the best treatment time.

[0004] Therefore, at the current stage, a precise treatment system is needed for skin diseases, especially difficult skin diseases. It can establish a connection between patients and doctors through follow-up and diagnose the patient's disease by obtaining the patient's skin condition in real time. Summary of the invention

[0005] In order to solve the above technical problems, the embodiments of the present application provide an information-based follow-up management method and system for precise diagnosis and treatment of skin diseases, which improves the diagnostic efficiency and accuracy by establishing a relationship between the patient and the doctor and obtaining the test results by detecting the patient's skin images through an automated detection method.

[0006] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, a method for follow-up management of accurate diagnosis and treatment of skin diseases is provided, which is applied to a server, and comprises: obtaining a regional skin image of a management object, performing morphological processing on the regional skin image to remove hair features in the regional skin image, and performing enhancement processing on the regional skin image after removal to obtain a skin image to be processed; inputting the skin image to be processed into a detection model, extracting features of the skin image to be processed, and determining a classification result corresponding to the skin image to be processed based on the features; the detection model comprises a first branch network and a second branch network, the first branch network being used to extract initial features of the skin image to be processed, and generate shallow features rich in texture and background information and generative features rich in semantic information, the second branch network being used to fuse the common features output by the first branch network with the features output by the upper layer in the second branch network, and pass them to the next layer; the results corresponding to the first branch network and the second branch network are weighted summed to calculate the final classification result.

[0008] Furthermore, the morphological processing of the regional skin image to remove hair features in the regional skin image includes: binarizing the regional skin image to obtain a mask of hair features; dilating and corroding the regional skin image to obtain an initial skin image, and subtracting the initial skin surface image from the regional skin image to obtain a difference image; removing hair from the difference image based on the mask to obtain an initial target skin image, acquiring empty points of the initial target skin image, and repairing the empty points to obtain a complete target surface image, which is the regional skin image after hair removal.

[0009] Furthermore, the skin image of the area after hair removal is enhanced to obtain the skin image to be processed, including: performing color space transformation on the skin image of the area, converting the RGB color space corresponding to the skin image of the area into the HSV color space, and removing noise in the skin image of the area after the color space conversion, to obtain the skin image to be processed after enhancement processing.

[0010] Furthermore, the noise in the skin image of the area after the color space conversion is removed, including: splitting the HSV color space into H channel component, S channel component and V channel component, and performing median filtering noise reduction on the S channel component, enhancing the denoised S channel component to calculate the updated pixel value of the S channel component, and merging the H channel component pixel value, the V channel component pixel value and the updated pixel value of the S channel component and converting them back to the HSV color space.

[0011] Furthermore, the median filtering denoising process for the S channel component comprises: obtaining the grayscale median of the skin image of the area, and denoising the S channel component based on the grayscale median; the enhancement process for the denoised S channel component to calculate the updated pixel value of the S channel component comprises: statistically calculating the corresponding distribution states of multiple grayscale values ​​in the S channel component after denoising, determining a transformation function based on the distribution state, and merging similar grayscale values ​​based on the transformation function to obtain a new pixel value as the updated pixel value.

[0012] Furthermore, the first branch network is connected by multiple first feature extraction modules, and the second branch network is connected by multiple second feature extraction modules; and the first feature modules and the second feature modules in the first branch network and the second branch network are connected through a common feature extraction module, and the common feature extraction module is used to extract common features from the feature maps of adjacent layers in the first branch network and the second branch network, and then the feature map of each layer in the first branch network and the common feature map obtained after processing by the common feature extraction module are fused through a feature fusion network to obtain a fused feature map, and the fused feature map is passed to the corresponding second feature extraction module in the second branch network.

[0013] Furthermore, the common feature extraction module uses convolution and upsampling operations to set the number of channels and size of the input feature map to the one with the largest channel or size in the two feature maps, and uses convolution and downsampling to restore the number of channels and size of the two feature maps at output.

[0014] Furthermore, the feature fusion network includes a first input terminal and a second input terminal, the first input terminal and the second input terminal respectively receive a common feature map and a feature map, the first input terminal is connected to a first processing branch and a second processing branch respectively receive the common feature map, the second output terminal is provided with a first segment branch and a second segment branch, the first segment branch is connected to the output terminal of the first input terminal and to the second segment branch, the output terminal of the second segment branch is connected to the output terminal of the second processing branch and fused through matrix addition.

[0015] Furthermore, the second processing branch includes a first type of processing branch and a second type of processing branch, and the first type of processing branch and the second type of processing branch are selected based on the received common feature map size; when the common feature map size is less than or equal to a preset feature map size, the first type of processing branch is selected to be connected to the second segment branch, and when the common feature map size is larger than the preset feature map size, the second type of processing branch is selected to be connected to the second segment branch.

[0016] In a second aspect, an information-based follow-up management system for accurate diagnosis and treatment of skin diseases is provided, the system comprising a patient end, a doctor end and a server, the patient end and the doctor end are respectively connected to the server, the server is provided with a condition detection device, for executing any of the above-mentioned information-based follow-up management methods for accurate diagnosis and treatment of skin diseases, the device comprising: an image preprocessing module, for acquiring a regional skin image of a management object, and performing morphological processing on the regional skin image to remove hair features in the regional skin image, and performing enhancement processing on the regional skin image after removal to obtain a skin image to be processed; a recognition module, for inputting the skin image to be processed into a detection model, extracting features of the skin image to be processed, and determining a classification result corresponding to the skin image to be processed based on the features.

[0017] According to a third aspect, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described methods when executing the computer program.

[0018] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above items is implemented.

[0019] In the technical solution provided in the embodiment of the present application, a doctor end and a patient end are set for the system, and the skin image uploaded by the patient end is detected based on an automated detection method, and the detection result is obtained. Among them, the automated detection method is implemented based on a detection model, wherein the detection model is provided with a first branch network and a second branch network, and the first branch network and the second branch network are used to realize the transmission and fusion between the shallow and deep feature maps to achieve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] The methods, systems and / or programs in the accompanying drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example numbers represent similar mechanisms in the various views of the accompanying drawings.

[0022] Figure 1It is a structural diagram of an information-based follow-up management system for accurate diagnosis and treatment of skin diseases provided in an embodiment of the present application.

[0023] Figure 2 It is a flow chart of the information-based follow-up management method for precise diagnosis and treatment of skin diseases provided in the embodiment of the present application.

[0024] Figure 3 It is a schematic diagram of the structure of an information-based follow-up management device for accurate diagnosis and treatment of skin diseases provided in an embodiment of the present application.

[0025] Figure 4 It is a schematic diagram of the structure of the diagnostic equipment provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to better understand the above technical scheme, the technical scheme of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0027] In the following detailed description, numerous specific details are set forth by way of example in order to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that the present application may be practiced without these details. In other cases, well-known methods, procedures, systems, compositions and / or circuits have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.

[0028] Flowcharts are used in the present application to illustrate the execution process performed by the system according to the embodiment of the present application. It should be clearly understood that the execution process of the flowchart may not be performed in order. On the contrary, these execution processes may be performed in reverse order or simultaneously. In addition, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0029] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0030] (1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0031] (2) Based on is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0032] Before describing the technical solution, it is necessary to explain the terms that appear in the embodiments of the present application:

[0033] Common map: common feature map;

[0034] Fusion map: fusion feature map;

[0035] Feature map: feature map.

[0036] We have witnessed the rapid development of mobile medical care in recent years. In 2011, the World Health Organization (WHO) defined mobile medical care as providing support for medical and public health practices through mobile devices such as mobile phones, patient monitoring devices, PDAs and other wireless devices. In the Internet field, mobile medical care mostly exists in the form of medical apps based on Android and IOS operating systems. Unlike the traditional medical model, mobile medical care has made full use of the popularity and portability of mobile devices, vigorously developed products and information systems such as electronic call systems, electronic medical records, online registration, and electronic scheduling, and successfully introduced them into the daily work of medical institutions, which not only improved the work efficiency of medical staff, but also brought great convenience to patients in seeing a doctor. With the integration of 4G services and medical information technology, mobile application products for health have grown rapidly, and patients use these software to monitor themselves, thereby improving their quality of life.

[0037] At present, the development of informatization in the medical industry is mainly focused on hospital management informatization and clinical management informatization, while the development of screening before the patient is diagnosed and admitted to the hospital and follow-up continuity care services after treatment is still in its infancy. Relevant studies have shown that pre-hospitalization screening and follow-up continuity care can detect the patient's disease development status early, improve the patient's health status and reduce their re-hospitalization rate, but the existing services cannot meet the needs of patients. There are many ways to follow up on continuity care services, including telephone follow-up, visits, online consultations, etc. Based on the current development of information technology and people's behavior patterns, continuing care based on the network platform is generally recognized by the public. Users do not need to master special skills or carry complex equipment to enable patients to maintain two-way contact with doctors, and patients can also obtain relevant health knowledge.

[0038] Based on the above technical background, the embodiment of the present application provides a method for screening and evaluating gastrointestinal tumors based on computer technology, which screens the possibility of the patient to be diagnosed, collects images of high-likelihood patients, and automatically identifies the acquired gastrointestinal endoscopic images to determine abnormal areas and classifies the abnormal areas to determine whether they have the corresponding symptoms.

[0039] For details, see Figure 1 According to an embodiment of the present application, a skin disease precision diagnosis and treatment information follow-up management system 100 is provided, which includes a server 110 and a patient terminal 120 and a doctor terminal 130 communicating with the server.

[0040] In the embodiment of the present application, the patient side includes four modules: follow-up management, doctor-patient interaction, information and personal center. Among them, the follow-up management module records all follow-up questionnaires and image acquisition requests received by the patient. The patient can view the answer details of the answered form, answer questions online and upload the answer results, and realize the skin image acquisition of the patient's target area by calling the image acquisition device in the patient-side device. The functions of the doctor-patient interaction module include viewing doctor information and instant messaging. Doctor information includes the doctor's name, title, department and introduction; instant messaging provides patients with the most convenient platform for communicating with doctors. Like chat software, it supports multiple data formats such as text, emoticons, voice, and pictures. The information module provides patients with relevant health information of their departments, which is displayed in the form of a list. The information is published and managed by the platform administrator or doctor. The personal center module includes medication plan management, password modification, feedback, and medical history records. Among them, for medication plan management, patients can add, edit and end medication plans in the medication plan function list, and can also view their own medication records. A medication plan consists of six elements: drug name, medication method, dosage, dosage unit, start date, and medication time. Up to six medication times can be added for the same drug in one day. Changing the password can enhance account security. After identity verification, the password can be reset. Feedback allows users to submit their opinions or suggestions during the use of the system in the form of text or pictures, which will be resolved by the system administrator. Medical history information is used for patients who visit the hospital. The system will automatically enter the patient's medical information and diagnosis report for subsequent review by the patient.

[0041] In an embodiment of the present application, the doctor side includes three modules, namely, a follow-up management module, a doctor-patient interaction module and a personal center. The follow-up management module provides doctors with the functions of viewing forms, issuing image acquisition commands and creating forms. All sent forms are displayed on this page in the form of a list, and the patient's answers can be viewed on the details page; at the same time, the doctor can create a new follow-up questionnaire on the mobile terminal and send it directly to the patient. The functions of the doctor-patient interaction module include patient management, instant messaging and viewing the patient's medication status. These functions provide convenience for doctors to understand the patient's current physical condition at any time and get in touch with the patient. Through the information fed back by the patient, the doctor can provide the patient with a more targeted rehabilitation plan. The personal center module includes information viewing, password modification, feedback, and message reminders. Doctors can set up the system according to their personal habits to enhance the user experience.

[0042] In the embodiment of the present application, the follow-up management method mainly obtains skin images uploaded by the patient based on an image acquisition device, and identifies and classifies the skin images to obtain information on whether the skin images have skin diseases and the corresponding skin disease types, thereby achieving the technical purpose of remote automated detection and improving the effect of follow-up and the accuracy of diagnosis.

[0043] Specifically, in the embodiment of the present application, the image acquisition device is a camera device built into the patient-side device, wherein the patient-side device may be a mobile terminal such as a mobile phone.

[0044] See also Figure 2 , the information-based follow-up management method for accurate diagnosis and treatment of skin diseases provided in the embodiment of the present application includes the following steps:

[0045] Step S210: Acquire a regional skin image of the management object, perform morphological processing on the regional skin image to remove hair features in the regional skin image, and perform enhancement processing on the regional skin image after removal to obtain a skin image to be processed.

[0046] In the embodiment of the present application, because the collected images are collected by the patient-side device itself, and are not collected by clinical high-pixel devices or skin electron microscopes, the image performance and anti-interference capabilities are poor, resulting in the final image having large noise and lack of expression. In addition, because the skin surface is covered by hair and the patient's non-professional ability, the collected images have more hair features, and the presence of hair features reduces the accuracy of subsequent feature processing.

[0047] Therefore, before detecting and identifying skin images, it is necessary to remove the interference information in the image, namely the hair, and perform noise reduction and image enhancement processing on the image after removing the interference information to improve the image's performance.

[0048] Specifically, the removal of hair features is mainly done through morphological means, which specifically includes the following steps:

[0049] The skin surface image is binarized to obtain a mask related to the interference image.

[0050] The skin surface image is expanded. For the expansion operation, a window of fixed size is set. In the embodiment of the present application, the window shape is selected as a cross, and the center of the window is the anchor point. The maximum value pixel is taken from the window and assigned to the anchor point. The window is traversed through all pixels in the image. The image after expansion is eroded. A window of fixed size is also set. The minimum value pixel is taken from the window and assigned to the anchor point. All pixels in the image are traversed. The initial skin surface image is obtained after the expansion operation and the erosion operation.

[0051] Then, a black hat operation is performed on the initial skin surface image and the skin surface image, wherein the black hat operation is to subtract the initial skin surface image from the skin surface image to obtain a difference image.

[0052] Based on the mask, the difference image is interfered with and an initial target surface image is obtained. The empty points of the initial target surface image are obtained, and the empty points are repaired to obtain a complete target surface image. The empty point repair process is implemented based on the FMM algorithm. The FMM algorithm first processes the pixel points on the edge of the area to be repaired, and then advances inward layer by layer until all the pixel points are repaired to obtain the final target surface image.

[0053] The enhancement processing of the target surface image after hair removal to obtain the skin image to be processed includes: performing color space transformation on the skin image of the region, converting the RGB color space corresponding to the skin image of the region into the HSV color space, and removing noise in the skin image of the region after the color space conversion, to obtain the skin image to be processed after the enhancement processing.

[0054] Images are formed by combining channels of different categories. In digital image processing, the general idea is to split the image by channel, process the three basic components separately and then merge them. In the embodiment of the present application, the images are based on the RGB color space and consist of three basic components: R (red), G (green), and B (blue). However, the three basic components cannot be processed digitally, and the specific details of the image are difficult to convert into a processable digital representation. It is difficult to obtain a good processing effect by directly processing in the RGB color space. Therefore, the image needs to be converted in the color space first.

[0055] There are many kinds of color spaces, the most common ones are YUV, CMY, HSV, HSI, etc. Different color spaces are based on different usage effects. In this article, the image will be processed based on the HSV color space. The advantage of this color space is that it separates color information and grayscale information, which is more in line with human visual characteristics and is more conducive to image processing. The H, S, and V of the HSV color space represent hue (Hue), saturation (Saturation), and value (Value), respectively. Hue H refers to the wavelength transmitted or reflected by an object, that is, it is identified by color; color saturation S refers to the intensity or purity of the color, measured from 0% (grayscale) to 100% (fully saturated); value V refers to the relative brightness of the color. In the embodiment of the present application, the processing is mainly performed on the grayscale of the image, that is, the color saturation S component is mainly processed.

[0056] Specifically, the HSV color space is split into an H channel component, an S channel component and a V channel component, and the S channel component is subjected to median filtering denoising processing, the denoised S channel component is enhanced to calculate the updated pixel value of the S channel component, and the H channel component pixel value, the V channel component pixel value and the updated pixel value of the S channel component are merged and converted back to the HSV color space.

[0057] Among them, for the median filter noise reduction process, by setting the filter, the pixel grayscale values ​​in the filter window are sorted, and then the middle value of the arrangement order is used as the output result. Since the characteristic of noise is that there is a grayscale difference with the surrounding grayscale values, when arranged by size, the grayscale value of the noise will be concentrated on the left or right end. When the median output is used, the final output excludes the influence of noise. Therefore, here we first set the size of the window template, then align the window with the pixels to be processed in the target area, and finally sort the pixels within the window area and calculate the median of the window. The median is used as the output within the window range to process the S component, and finally the processed S component is merged with the H component and the V component, and converted back to the RGB color space.

[0058] Due to the limitations of shooting conditions, clinical images may suffer from various adverse consequences such as insufficient image contrast and large differences in brightness and darkness due to factors such as lighting and shooting angle during the shooting process, making many details in the image unclear and reducing the performance of subsequent classification algorithms. Therefore, it is necessary to enhance the image to enhance the contrast differences between different levels.

[0059] In an embodiment of the present application, a histogram equalization method is used for enhancement processing, specifically, the corresponding distribution states of multiple grayscale values ​​in the S channel component after noise reduction are statistically calculated, a transformation function is determined based on the distribution state, and similar grayscale values ​​are merged based on the transformation function to obtain a new pixel value as the updated pixel value.

[0060] Step S220. Input the skin image to be processed into the detection model, extract the features of the skin image to be processed, and determine the classification result corresponding to the skin image to be processed based on the features.

[0061] In an embodiment of the present application, the detection model includes a first branch network and a second branch network that are symmetrically arranged. Among them, the first branch network is a network arranged on the left, and the second branch network is a network arranged on the right, wherein the first branch network can be an original network branch, and the second branch network is a fusion network branch. The first branch network is responsible for the initial feature extraction of the input image, which is used to generate shallow features rich in texture and background information, as well as generative features rich in semantic information. The second branch network on the right uses a fusion block to fuse the common features on the left with the features output by the upper layer, and pass it to the next layer. The final results of the two branches are weighted and summed to calculate the final classification result. Among them, the results of the two branch networks are given different weights by two learnable parameters γ1 and γ2. In addition, for the problem of inconsistent sizes of deep and shallow feature maps, the embodiment of the present application scales the feature maps of smaller sizes.

[0062] Among them, the first branch network is connected by multiple first feature extraction modules, the second branch network is connected by multiple second feature extraction modules, and the first feature modules and the second feature modules in the first branch network and the second branch network are connected through a common feature extraction module. In an embodiment of the present application, the common feature extraction module is used to extract common features from the feature maps of adjacent layers in the first branch network and the second branch network, and then the feature map of each layer in the first branch network and the common feature map obtained after processing by the common feature extraction module are fused through a feature fusion network to obtain a fused feature map, and the fused feature map is passed to the corresponding second feature extraction module in the second branch network.

[0063] Specifically, the skin image to be processed after denoising is input into the first branch network and the second branch network respectively. First, the first feature extraction is performed on the skin image to be processed to obtain two feature maps. The input channels and sizes of the feature maps extracted by the different first branch networks and second branch networks are different. For the two feature maps with different input channels and sizes, the common feature extraction module uses convolution and upsampling operations to set the number of channels and size of the input feature map to the one with the largest channel or size in the two feature maps. At output, convolution and downsampling are used to restore the number of channels and size of the two feature maps, and the specific calculation process is presented based on the following formula:

[0064]

[0065] In the embodiment of the present application, for the output of the common feature extraction module The number of channels and size of the feature map are based on the feature map of the original network layer a, and the result of feature fusion with the feature map of the layer b. In the formula, R(), M(), G(), and F() are all ConvBNReLU with a convolution kernel of 3×3, and the subscripts l and h represent the corresponding processing input f l and f h The input is and If f h The size and f l If the sizes of f are different, first h Upsample it to make it consistent with f l The length and width are the same, R() will h and f l The number of channels is unified as C = Max (C1, C2), and finally F l and F h Restore the number of channels of the two layers to the size C1 and C2 at the time of input respectively. At the final output, if the initial size is different, Downsampling is performed to make the two branches the same as the input, that is,

[0066] In the embodiment of the present application, the feature map generated by each layer of the first branch network is respectively input into the common feature extraction module with the feature maps of the upper and lower layers to extract the common features, and two common feature fusion feature maps are obtained. Finally, the feature map of the first branch network and the fusion feature map are added and averaged to obtain the final common feature map, which is expressed as shown in the following formula:

[0067] Among them common i represents the result of common feature extraction at the i-th layer, and Mean(·) represents the averaging function.

[0068] In an embodiment of the present application, the feature fusion network includes a first input terminal and a second input terminal, the first input terminal and the second input terminal respectively receive a common feature map and a feature map, the first input terminal is connected to a first processing branch and a second processing branch respectively receive the common feature map, the second output terminal is provided with a first segment branch and a second segment branch, the first segment branch is connected to the output terminal of the first input terminal and to the second segment branch, the output terminal of the second segment branch is connected to the output terminal of the second processing branch and fused through matrix addition.

[0069] In the embodiment of the present application, two fusion methods are provided, and the fusion process is controlled by setting the advance. Specifically, when the size of the common feature map is less than or equal to the preset feature map size, the first type of processing branch is selected to be connected with the second segment branch, and when the size of the common feature map is greater than the preset feature map size, the second type of processing branch is selected to be connected with the second segment branch.

[0070] For the output x of each layer of the feature fusion network i And the common feature extraction part passed in each layer i In the present embodiment, x i and common i Perform a dot multiplication operation to get fusion i , as shown below:

[0071] fusion i =x i ·common i , where x i , fusion i , common i ∈R C×W×H Afterwards, this paper will discuss x i , fusion i and common i Use 1×1 convolutional coding and then reshape the block to merge the length and width to get x′ i fusion i , common′ i ∈R C×N , where N = W × H. Then x′ i fusion i The encoded result is subjected to matrix product operation to obtain the correlation graph CP i :

[0072] CP i =(common′ i ) T ×x′ i , the obtained CP i ∈R N×N , the present application embodiment has a i After softmax and fusion′ i Matrix multiplication is performed to obtain the fusion graph FP, and the process is shown in the following formula:

[0073] FP i =(fusion i ′) T ×Soft max(CP i ).

[0074] In the embodiment of the present application, the characteristics of the lesion area are highlighted based on the obtained fusion map, and the specific disease type is determined based on the characteristics of the non-lesion area. Specifically, this method can realize the transmission and fusion between the shallow and deep feature maps, and fully express the lesion characteristics.

[0075] See also Figure 3 , with respect to the disease condition detection device 300 disposed in the server, it specifically includes the following structure:

[0076] The image preprocessing module 310 is used to obtain a regional skin image of a management object, perform morphological processing on the regional skin image to remove hair features in the regional skin image, and perform enhancement processing on the regional skin image after removal to obtain a skin image to be processed;

[0077] The recognition module 320 is used to input the skin image to be processed into the detection model, extract the features of the skin image to be processed, and determine the classification result corresponding to the skin image to be processed based on the features.

[0078] See also Figure 4 , the above method can also be integrated into the provided dermatology diagnosis device 400, and the device can have relatively large differences due to different configurations or performances, and can include one or more processors 401 and memory 402, and the memory 402 can store one or more storage applications or data. Among them, the memory 402 can be a temporary storage or a permanent storage. The application stored in the memory 402 can include one or more modules (not shown in the figure), and each module can include a series of computer executable instructions in the diagnostic device 400. Furthermore, the processor 401 can be configured to communicate with the memory 402, and the dermatology diagnosis device 400 executes a series of computer executable instructions in the memory 402. The dermatology diagnosis device 400 can also include one or more power supplies 403, one or more wired / wireless network interfaces 404, one or more input / output interfaces 405, one or more keyboards 406, etc.

[0079] In a specific embodiment, the skin disease diagnosis device 400 includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the skin disease diagnosis device 400, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following:

[0080] Acquire a regional skin image of a management object, perform morphological processing on the regional skin image to remove hair features in the regional skin image, and perform enhancement processing on the regional skin image after removal to obtain a skin image to be processed;

[0081] The skin image to be processed is input into a detection model, features of the skin image to be processed are extracted, and a classification result corresponding to the skin image to be processed is determined based on the features.

[0082] The following is a detailed introduction to the various components of the processor:

[0083] In this embodiment, the processor is an application specific integrated circuit (ASIC), or is configured to implement one or more integrated circuits of the embodiments of the present application, such as one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays (field programmable gate array, FPGA).

[0084] Optionally, the processor may execute various functions by running or executing a software program stored in the memory and calling data stored in the memory, such as executing the above-mentioned Figure 1 The method shown.

[0085] In a specific implementation, as an embodiment, the processor may include one or more microprocessors.

[0086] Among them, the memory is used to store the software program that executes the solution of the present application, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0087] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor, or may exist independently and be coupled to the processing unit through the interface circuit of the processor, and the embodiments of the present application do not specifically limit this.

[0088] It should be noted that the structure of the processor shown in this embodiment does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0089] In addition, the technical effects of the processor can refer to the technical effects of the method described in the above method embodiment, which will not be repeated here.

[0090] It should be understood that the processor in the embodiments of the present application may be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0092] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0093] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0094] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks.

[0101] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for follow-up management of accurate diagnosis and treatment of skin diseases, characterized in that: The method is applied to a server, and the method comprises: Acquire a regional skin image of a management object, perform morphological processing on the regional skin image to remove hair features in the regional skin image, and perform enhancement processing on the regional skin image after removal to obtain a skin image to be processed; The skin image to be processed is input into the detection model, the features of the skin image to be processed are extracted, and the classification result corresponding to the skin image to be processed is determined based on the features; the detection model includes a first branch network and a second branch network, the first branch network is used to extract the initial features of the skin image to be processed, and generate shallow features rich in texture and background information and generative features rich in semantic information, the second branch network is used to fuse the common features output by the first branch network with the features output by the upper layer of the second branch network, and pass it to the next layer; the results corresponding to the first branch network and the second branch network are weighted summed to calculate the final classification result.

2. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 1, characterized in that: The morphological processing of the regional skin image to remove the hair features in the regional skin image includes: binarizing the regional skin image to obtain a mask of the hair features; dilating and corroding the regional skin image to obtain an initial skin image, and subtracting the initial skin surface image from the regional skin image to obtain a difference image; removing hair from the difference image based on the mask to obtain an initial target skin image, acquiring empty points of the initial target skin image, and repairing the empty points to obtain a complete target surface image, which is the regional skin image after the hair is removed.

3. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 2, characterized in that: The enhancing process of the skin image of the region after hair removal to obtain the skin image to be processed includes: performing color space transformation on the skin image of the region, converting the RGB color space corresponding to the skin image of the region into the HSV color space, and removing noise from the skin image of the region after the color space conversion to obtain the skin image to be processed after the enhanced process.

4. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 3, characterized in that: The method of removing noise from the skin image of the area after converting the color space includes: splitting the HSV color space into an H channel component, an S channel component and a V channel component, and performing a median filtering noise reduction process on the S channel component, performing an enhancement process on the denoised S channel component, calculating an updated pixel value of the S channel component, and merging the H channel component pixel value, the V channel component pixel value and the updated pixel value of the S channel component and converting them back to the HSV color space.

5. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 4, characterized in that: The median filtering denoising process for the S channel component comprises: obtaining the grayscale median of the skin image of the area, and denoising the S channel component based on the grayscale median; the enhancement process for the denoised S channel component to calculate the updated pixel value of the S channel component comprises: counting the corresponding distribution states of multiple grayscale values ​​in the S channel component after denoising, determining a transformation function based on the distribution state, and merging similar grayscale values ​​based on the transformation function to obtain a new pixel value as the updated pixel value.

6. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 5, characterized in that: The first branch network is connected by multiple first feature extraction modules, and the second branch network is connected by multiple second feature extraction modules; and the first feature modules and the second feature modules in the first branch network and the second branch network are connected through a common feature extraction module, and the common feature extraction module is used to extract common features from the feature maps of adjacent layers in the first branch network and the second branch network, and then the feature map of each layer in the first branch network and the common feature map obtained after processing by the common feature extraction module are fused through a feature fusion network to obtain a fused feature map, and the fused feature map is passed to the corresponding second feature extraction module in the second branch network.

7. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 6, characterized in that: The common feature extraction module uses convolution and upsampling operations to set the number of channels and size of the input feature map to the one with the largest channel or size in the two feature maps, and uses convolution and downsampling to restore the number of channels and size of the two feature maps at output.

8. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 7, characterized in that: The feature fusion network includes a first input terminal and a second input terminal, the first input terminal and the second input terminal receive a common feature map and a feature map respectively, the first input terminal is connected to a first processing branch and a second processing branch to receive the common feature map respectively, the second output terminal is provided with a first segment branch and a second segment branch, the first segment branch is connected to the output terminal of the first input terminal and to the second segment branch, the output terminal of the second segment branch is connected to the output terminal of the second processing branch and fused through matrix addition.

9. The information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to claim 8, characterized in that: The second processing branch includes a first type of processing branch and a second type of processing branch, and the first type of processing branch and the second type of processing branch are selected based on the size of the received common feature map; when the size of the common feature map is less than or equal to the preset feature map size, the first type of processing branch is selected to be connected to the second segment branch, and when the size of the common feature map is larger than the preset feature map size, the second type of processing branch is selected to be connected to the second segment branch.

10. An information-based follow-up management system for accurate diagnosis and treatment of skin diseases, characterized in that: The system includes a patient end, a doctor end and a server, wherein the patient end and the doctor end are respectively connected to the server, and a disease condition detection device is provided in the server for executing the information-based follow-up management method for accurate diagnosis and treatment of skin diseases according to any one of claims 1 to 9, wherein the device includes: An image preprocessing module is used to obtain a regional skin image of a management object, perform morphological processing on the regional skin image to remove hair features in the regional skin image, and perform enhancement processing on the regional skin image after removal to obtain a skin image to be processed; The recognition module is used to input the skin image to be processed into the detection model, extract the features of the skin image to be processed, and determine the classification result corresponding to the skin image to be processed based on the features.

Images