Pathological model edge calculation box equipment based on edge calculation

Abstract

The invention provides pathological model edge calculation box equipment based on edge calculation. The pathological model edge calculation box equipment comprises at least one image acquisition end and an inference end, wherein the image acquisition end is used for acquiring an image of a pathological tissue sample; and the inference end carries out identification and inference on the received image sent by the image acquisition end and returns a result to the image acquisition end. According to the pathological model edge computing box device based on edge computing, local data processing is carried out on the edge device, sensitive data can be prevented from being transmitted to a cloud end, privacy of a user and a patient can be protected, data transmission is not needed due to local processing, and extra cost is avoided.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to a pathological model edge computing box device based on edge computing. Background Art

[0002] In recent years, with the rapid development and widespread application of deep learning technology, especially the rise of convolutional neural networks (CNN), it has achieved great success in various fields including medical image analysis. Deep learning models are trained with a large amount of pathological image data, which can automatically learn and extract features in images, accurately locate and segment lesion areas, and provide doctors with reliable auxiliary diagnostic tools.

[0003] The existing assistance to doctors all adopts the cloud computing model. Although the traditional cloud computing model can perform analysis and calculation of pathological images on cloud servers, it also faces problems such as privacy security, high latency, and high data transmission costs, which limit its application in real-time diagnosis and mobile medical fields. This is because cloud computing needs to transmit a large amount of pathological image data from edge devices to cloud servers for processing, resulting in long data transmission delays and high server room maintenance costs. At the same time, since pathological images may contain sensitive personal health information, cloud computing needs to ensure privacy security during data transmission and processing, adding additional privacy protection measures and costs. Summary of the invention

[0004] One of the purposes of the present invention is to provide a pathology model edge computing box device based on edge computing. By performing local data processing on the edge device, sensitive data can be avoided from being transmitted to the cloud, which helps to protect the privacy of users and patients. Moreover, since it is processed locally, no data transmission is required, thus avoiding additional costs.

[0005] An embodiment of the present invention provides a pathology model edge computing box device for edge computing, comprising: at least one image acquisition end and an inference end;

[0006] Among them, the image acquisition end acquires images of pathological tissue samples; the inference end recognizes and infers the images sent by the received image acquisition end and returns the results to the image acquisition end.

[0007] Preferably, the image acquisition end includes: an electron microscope and a PC terminal.

[0008] Preferably, the PC performs the following operations:

[0009] The image of the pathological tissue sample is collected by an electron microscope to obtain an initial image;

[0010] Preprocess the initial image;

[0011] The preprocessed initial image is sent to the inference end through the TCP / IP network protocol.

[0012] Preferably, the preprocessing includes: image brightness adjustment and / or selection of a region of interest.

[0013] Preferably, the PC terminal also performs the following operations:

[0014] Match the preprocessed initial image with the temporary image in the temporary library;

[0015] When there is a temporarily stored image matching the initial image, a local recognition and reasoning result of the initial image is generated and output according to the recognition and reasoning result associated with the temporarily stored image.

[0016] Preferably, a deep learning model is configured at the inference end, wherein the deep learning model includes: a combination of one or more of a model for cell detection, a model for cell counting, a model for pathology classification, and a model for lesion detection.

[0017] Preferably, the inference end further performs the following operations:

[0018] Retrieving historical images of the object corresponding to the initial image and corresponding recognition reasoning results;

[0019] A growth graph is constructed based on the retrieved historical images, the recognition and reasoning results corresponding to the historical images, the initial image and the recognition and reasoning results corresponding to the initial image, and is output to the image acquisition end.

[0020] Preferably, the inference end further performs the following operations:

[0021] The accuracy is calculated based on the output of the neurons of each model and the output of the model during the generation of the recognition reasoning result corresponding to the initial image;

[0022] When the accuracy is less than or equal to the preset accuracy threshold, the initial image is sent to the server;

[0023] Receiving the recognition and reasoning result of the server for the initial image and verifying the recognition and reasoning result of the reasoning end based on the received recognition and reasoning result;

[0024] If the verification fails, access the server to update the configured deep learning model.

[0025] Preferably, the inference end further performs the following operations:

[0026] When the recognition reasoning result is a result in the pre-configured verification library, the initial image is sent to the server for recognition reasoning.

[0027] Preferably, the inference end further performs the following operations:

[0028] When the inference result is identified as a result in a pre-configured verification library, a pre-configured inference terminal communication list of other adjacent devices is retrieved;

[0029] Send the initial image to a preset number of inference terminals arranged in descending order of difference, and receive the returned recognition inference results;

[0030] Based on the returned recognition and reasoning results, a joint analysis display interface is generated and output.

[0031] The present invention has the following advantages:

[0032] 1. By performing local data processing on edge devices, sensitive data can be prevented from being transmitted to the cloud, helping to protect the privacy of users and patients.

[0033] 2. Compared with cloud computing, edge computing reduces data transmission time and delay. Edge devices effectively utilize local computing resources, perform data processing and analysis near the device, reduce data transmission volume and transmission time, and improve real-time performance. This not only reduces the cost of data transmission and processing, but also saves bandwidth costs.

[0034] 3. Edge computing shows obvious advantages in inference time. Compared with the cloud computing mode that transmits data to cloud servers for processing, the inference time on edge devices is shorter. This reduces the delay in result feedback and helps to achieve the requirements of real-time diagnosis and mobile medical care.

[0035] 4. Edge computing reduces dependence on network connections. Even when the network is unstable or disconnected, edge devices can still process data and make decisions, improving the reliability of the system. It can reduce the maintenance costs of server rooms in medical institutions.

[0036] 5. The portability of edge devices also makes them suitable for teaching, for example, by medical researchers. Traditional medical teaching usually relies on large servers and computer equipment, which imposes certain limitations on the mobility and flexibility of the teaching process.

[0037] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0038] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0040] Figure 1 It is a schematic diagram of a pathological model edge computing box device based on edge computing in an embodiment of the present invention;

[0041] Figure 2 This is a topological diagram of a pathological model edge computing box device based on edge computing in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0043] The embodiment of the present invention provides a pathological model edge computing box device based on edge computing, such as Figure 1 to Figure 2 As shown, it includes: at least one image acquisition end 1 and a reasoning end 2;

[0044] The image acquisition end 1 acquires images of pathological tissue samples; the inference end 2 performs recognition and inference on the images received from the image acquisition end and returns the results to the image acquisition end.

[0045] The image acquisition terminal 1 includes an electron microscope and a PC terminal.

[0046] The PC performs the following operations:

[0047] The image of the pathological tissue sample is collected by an electron microscope to obtain an initial image;

[0048] Preprocess the initial image;

[0049] The preprocessed initial image is sent to the inference end through the TCP / IP network protocol.

[0050] The preprocessing includes: adjusting the image brightness and / or selecting a region of interest.

[0051] A deep learning model is configured at the inference end 2, wherein the deep learning model includes: a model for cell detection, a model for cell counting, a model for pathology classification, and a model for lesion detection, or one or more combinations thereof.

[0052] The working principle and beneficial effects of the above technical solution are:

[0053] The edge computing box device of the pathological model based on edge computing provided in this embodiment includes: at least one image acquisition end and an inference end; wherein, the case image acquisition end (i.e., the image acquisition end) is used for the acquisition of pathological images, and the image acquisition of pathological tissue samples is realized by connecting the electron microscope to the computer, and the acquired images are transmitted to the computer end for image preprocessing. After obtaining the inference results of the pathological image from the inference end, post-processing operations such as visualization can also be performed for medical researchers to analyze and study. The edge computing inference end (i.e., the inference end) is used for the inference of pathological images, and has the computing power and storage resources required for inference. As a computing platform for pathological image recognition and inference, a trained deep learning model is deployed on the device, such as cell detection, cell counting, pathological classification, lesion detection, etc., which is responsible for the recognition and inference tasks of the received pathological images, and returns the results to the computer at the image acquisition end. Overall process: First, an electron microscope is used to acquire high-resolution images of pathological specimens. The acquired images are transmitted to the computer through an electron microscope system connected to the computer, and then image preprocessing operations are performed in the application on the computer end, such as image brightness adjustment, selection of regions of interest, etc. Next, the computer establishes a TCP / IP network protocol connection with the edge device at the inference end for data communication and transmission. When the computer receives the pathological image data from the electron microscope, it transmits the image data to the inference end through the TCP / IP protocol, and uses the deployed model at the inference end to perform inference operations. After the inference is completed, the recognition and inference results of the pathological image are returned to the computer to achieve two-way communication and transmission. Finally, the computer performs post-processing steps such as parsing and visualization of the inference results for reference by medical researchers. The inference end and the acquisition end are in a one-to-many relationship, that is, an edge device at the inference end can be used for inference by multiple acquisition ends at the same time. Through the above steps, we have implemented a pathological image recognition and inference solution based on edge computing, which distributes the acquisition and recognition tasks of pathological images on different devices, and realizes the collaborative processing of computing tasks and the effective use of resources.

[0054] In one embodiment, the PC terminal further performs the following operations:

[0055] Match the preprocessed initial image with the temporary image in the temporary library;

[0056] When there is a temporarily stored image matching the initial image, a local recognition and reasoning result of the initial image is generated and output according to the recognition and reasoning result associated with the temporarily stored image.

[0057] The execution steps of the PC end provided in this embodiment are to more quickly identify and infer the initial image. By building a local storage temporary library on the PC end, a small number of (0-20) temporary images and the recognition and inference results corresponding to the temporary images are stored in the temporary library; by directly matching the initial image with the temporary images stored in the temporary library, the initial image is quickly predicted, and the recognition and inference results of the inference end can be verified to a certain extent. The temporary images in the temporary library are

[0058] In order to realize the growth analysis display of pathological trends, in one embodiment, the reasoning end further performs the following operations:

[0059] Retrieving historical images of the object corresponding to the initial image and corresponding recognition reasoning results;

[0060] A growth graph is constructed based on the retrieved historical images, the recognition and reasoning results corresponding to the historical images, the initial image and the recognition and reasoning results corresponding to the initial image, and is output to the image acquisition end.

[0061] In order to ensure the validity of the model at the inference end, in one embodiment, the inference end further performs the following operations:

[0062] The accuracy is calculated based on the output of the neurons of each model and the output of the model during the generation of the recognition inference result corresponding to the initial image; the accuracy is the ratio between the number of neurons whose output of the neurons of each model is the same as the output of the model and the total number of neurons;

[0063] When the accuracy is less than or equal to the preset accuracy threshold, the initial image is sent to the server;

[0064] Receive the recognition and reasoning result of the initial image from the server and verify the recognition and reasoning result of the reasoning end based on the received recognition and reasoning result; the verification mainly determines whether the reasoning result sent by the server is the same as the reasoning result of the reasoning end; if they are the same, the verification passes; otherwise, it fails;

[0065] If the verification fails, access the server to update the configured deep learning model.

[0066] This embodiment screens the initial image by triggering, and verifies the feedback result obtained by sending the screened initial image to the server. When the recognition reasoning results are inconsistent, the verification fails, and it is necessary to access the server to configure the deep learning model to update.

[0067] In one embodiment, the inference end further performs the following operations:

[0068] When the recognition reasoning result is a result in the pre-configured verification library, the initial image is sent to the server for recognition reasoning.

[0069] In this embodiment, a verification library is constructed on the inference end. The doctor configures the results in the verification library in advance. When the result of the inference on the inference end is the result in the library, it is uploaded to the server for verification. Since the inference end is an edge computing device after all, due to its performance (limited computing power), the doctor can configure the verification library to provide server-side computing power support for the initial image of the recognition reasoning result that needs to be re-confirmed, so as to ensure the accuracy and effectiveness of the recognition reasoning result.

[0070] In one embodiment, the inference end further performs the following operations:

[0071] When the inference result is identified as a result in a pre-configured verification library, a pre-configured inference terminal communication list of other adjacent devices is retrieved;

[0072] Send the initial image to a preset number of inference terminals arranged in descending order of difference, and receive the returned recognition inference results;

[0073] Based on the returned recognition and reasoning results, a joint analysis display interface is generated and output.

[0074] This embodiment configures the communication list of other reasoning terminals on the reasoning terminal. When verification is required, the initial image can be sent out through communication with other reasoning terminals, and feedback can be received to verify the recognition reasoning result of the initial image. When selecting the reasoning terminal, the difference can be arranged in order from large to small, and the reasoning terminals with the top N differences can be extracted as communication targets; wherein the difference is quantified by comprehensively analyzing the difference of the model of the reasoning terminal, specifically including model parameters (model type, number of neurons, etc.), loss function, difference in data used for training (the proportion of normal tissue images and pathological tissue images, the proportion of tissue images of each type of pathological type), etc. The steps for determining the difference are as follows: calculate the similarity of the labeled data sets between the two models; determine the difference based on the similarity; the difference is the value obtained by subtracting the similarity from 1; wherein the labeled data set is constructed and associated with the model through the following steps, and the specific steps are as follows: use the model parameters, loss function, and data used for training as data to be analyzed; perform feature extraction on the analyzed data to construct a labeled data set; the parameters in the labeled data set include: parameters representing each model parameter, parameters representing the type of loss function, parameters representing the proportion of normal tissue images and pathological tissue images in the training data, and parameters representing the proportion of tissue images of each type of pathological type.

[0075] In one embodiment, the joint analysis display interface also displays the confidence parameters of each inference end for the inference results returned by it;

[0076] The confidence parameter is determined by the following steps:

[0077] Obtain the number of processed records on the inference end, the accuracy of the processed records, the proportion of processed records that are the same as the returned inference results (the ratio of the number of identical records to the total number of records), and the time of the last update as analysis data;

[0078] Extract features from the analysis data and construct an analysis parameter set based on the features; the analysis parameter set includes: a parameter indicating the number of processed records, a parameter indicating the accuracy of processed records, a parameter indicating the proportion of processed records that are the same as the returned inference results, and a parameter indicating the time difference between the last updated time and the current time;

[0079] According to the analysis parameter set, the corresponding confidence parameters are retrieved from the pre-configured confidence parameter analysis library;

[0080] Among them, the confidence parameter analysis set is analyzed and configured in advance by professionals, and the analysis parameter set and the confidence parameter are associated one-to-one in the confidence parameter analysis library; in the process of constructing the confidence parameter analysis set, the larger the parameter of the number of processed records, the larger the confidence parameter; the higher the parameter representing the accuracy of the processed records, the larger the confidence parameter; the larger the parameter representing the proportion of processed records that are the same as the returned inference results, the larger the confidence parameter; the smaller the parameter representing the time difference between the last updated time and the current time, the larger the confidence parameter; the display of confidence parameters facilitates the screening of results by the corresponding staff at the inference end.

[0081] In one embodiment, when receiving the initial image from another inference end, determining the processing timing of the initial image according to the current working state;

[0082] When the inference end is idle, the initial image is processed immediately;

[0083] When the inference end is in working state, the idle time intervals between the previous N processing times and the time spent on the previous N processing times are obtained; when the average value of the idle time intervals between the current N processing times is greater than or equal to half of the average value of the time spent on the previous N processing times, the initial image is processed when the time spent on the next processing is less than or equal to the average value of the time spent on the previous N processing times. Where N is an integer greater than or equal to 2.

[0084] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A pathological model edge computing box device based on edge computing, characterized in that: include: At least one image acquisition end and inference end; Among them, the image acquisition end acquires images of pathological tissue samples; The inference end performs recognition inference on the image sent by the received image acquisition end and returns the result to the image acquisition end.

2. The pathological model edge computing box device based on edge computing according to claim 1 is characterized in that: The image acquisition end includes: electron microscope and PC terminal.

3. The pathological model edge computing box device based on edge computing according to claim 2 is characterized in that: Perform the following operations on the PC: The image of the pathological tissue sample is collected by an electron microscope to obtain an initial image; Preprocess the initial image; The preprocessed initial image is sent to the inference end through the TCP / IP network protocol.

4. The pathological model edge computing box device based on edge computing according to claim 3 is characterized in that: The preprocessing includes: adjusting image brightness and / or selecting a region of interest.

5. The pathological model binaural computing box device based on edge computing according to claim 1 is characterized in that: The PC also performs the following operations: Match the preprocessed initial image with the temporary image in the temporary library; When there is a temporarily stored image matching the initial image, a local recognition and reasoning result of the initial image is generated and output according to the recognition and reasoning result associated with the temporarily stored image.

6. The pathological model edge computing box device based on edge computing according to claim 1, characterized in that: A deep learning model is configured at the inference end, wherein the deep learning model includes: a model for cell detection, a model for cell counting, a model for pathology classification, and a model for lesion detection, or one or more combinations thereof.

7. The pathological model edge computing box device based on edge computing according to claim 1, characterized in that: The inference side also performs the following operations: Retrieving historical images of the object corresponding to the initial image and corresponding recognition reasoning results; A growth graph is constructed based on the retrieved historical images, the recognition and reasoning results corresponding to the historical images, the initial image and the recognition and reasoning results corresponding to the initial image, and is output to the image acquisition end.

8. The pathological model edge computing box device based on edge computing according to claim 1, characterized in that: The inference side also performs the following operations: The accuracy is calculated based on the output of the neurons of each model and the output of the model during the generation of the recognition reasoning result corresponding to the initial image; When the accuracy rate is less than or equal to a preset accuracy threshold, sending the initial image to the server; Receiving the recognition and reasoning result of the server for the initial image and verifying the recognition and reasoning result of the reasoning end based on the received recognition and reasoning result; If the verification fails, access the server to update the configured deep learning model.

9. The pathological model edge computing box device based on edge computing according to claim 1, characterized in that: The inference side also performs the following operations: When the recognition reasoning result is a result in the pre-configured verification library, the initial image is sent to the server for recognition reasoning.

10. The pathological model edge computing box device based on edge computing according to claim 9, characterized in that: The inference side also performs the following operations: When the inference result is identified as a result in a pre-configured verification library, a pre-configured inference terminal communication list of other adjacent devices is retrieved; Send the initial image to a preset number of inference terminals arranged in descending order of difference, and receive the returned recognition inference results; Based on the returned recognition and reasoning results, a joint analysis display interface is generated and output.

Images