Image processing method and device, equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of research and development design and image processing, is applied to medical image target detection and labeling scenes, and relates to an image processing method, device and equipment and a storage medium thereof.The method comprises the steps that a target image is input into a pre-trained improved detection model, and a preliminary detection result is obtained; displaying the target image and the preliminary detection result to a preset human-computer interaction interface; generating a final detection result; automatically marking the image into a target image to obtain a marked image; and generating a detection report, and sending the detection report to the target receiving end. The image processing method is applied to the field of medical image detection, focus detection information can be obtained more accurately by adopting an improved detection model, and a doctor can perform personalized operation and correction on a target image conveniently in combination with a personalized operation function and a labeling tool provided in a human-computer interaction interface, so that the medical image detection efficiency is improved. On the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

Description

Technical Field

[0001] The present application relates to the field of R&D design and image processing technology, and is applied to medical image target detection and annotation scenarios, and in particular to an image processing method, device, equipment and storage medium thereof. Background Art

[0002] Interpreting medical images is a highly specialized task that requires lengthy training by radiologists, which not only limits the number of people who can perform the job, but also makes the diagnostic process time-consuming and labor-intensive.

[0003] Although there have been many studies applying deep learning methods to medical image detection, currently, systems such as MABEL are able to generate medical image prediction results in a relatively short period of time. The MABEL system has achieved 96.5% and 97.3% for mass detection and calcification detection in the internal test set, respectively. This means that there is still a certain proportion of false negatives, that is, images with lesions are mistakenly marked as normal. This is an important consideration for clinical screening, which may lead to misdiagnosis and increase the amount of error correction tasks for doctors. Therefore, the existing MABEL system still has the problem of inaccurate detection results in medical image processing, which leads to an increase in the amount of error correction tasks for doctors. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image processing method, apparatus, device and storage medium thereof to solve the problem that the detection results of the existing MABEL system in medical image processing are not accurate enough, resulting in an increase in the amount of error correction tasks for doctors.

[0005] In order to solve the above technical problems, the present application provides an image processing method, which adopts the following technical solutions:

[0006] An image processing method comprises the following steps:

[0007] Get the target image;

[0008] Inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result;

[0009] Displaying the target image and the preliminary detection result on a preset human-computer interaction interface;

[0010] Obtaining a labeling result made by a target user on the human-computer interaction interface using a preset labeling tool for the target image;

[0011] The preliminary detection result is improved in combination with the annotation result to generate a final detection result;

[0012] Automatically annotating the final detection result into the target image to obtain an annotated image;

[0013] A detection report is generated according to the annotated image, and the detection report is sent to a target receiving end.

[0014] Furthermore, before executing the step of inputting the target image into the pre-trained improved detection model to obtain a preliminary detection result, the method further includes:

[0015] Acquiring training samples, wherein the training samples include a batch of medical images with annotated lesion information;

[0016] The training samples are input into an improved detection model to be trained, and detection training is performed on the improved detection model to obtain a pre-trained improved detection model, wherein the improved detection model includes a detection model based on an image target detection network.

[0017] Furthermore, the improved detection model includes an improved DeepLab series image target detection model with ResNet-50 as the main network structure, introducing FPN deep learning network and Non-LocalNet, and the step of inputting the training sample into the improved detection model to be trained, performing detection training on the improved detection model, and obtaining a pre-trained improved detection model specifically includes:

[0018] Using the FPN deep learning network to extract and fuse multi-scale feature maps of all training images in the training sample according to the feature pyramid method, to obtain all fused training images;

[0019] Using the ResNet-50 backbone network structure, image feature extraction is performed on all the fused training images to obtain image feature extraction results corresponding to all the training images;

[0020] Capturing the long-range dependency of image features respectively contained in all training images according to the image feature extraction results corresponding to all training images respectively by the Non-LocalNet;

[0021] According to the image feature extraction results corresponding to all training images and the long-range dependencies of the image features contained in all training images, the ResNet-50 backbone network structure is used as the detection backbone network, and target detection is performed on all fused training images to obtain detection results;

[0022] Using the DIoU loss function, the loss value of the detection result compared with the labeled lesion information in the training sample is calculated;

[0023] If the loss value meets the preset loss requirement, the pre-training of the improved detection model to be trained is completed, and a pre-trained improved detection model is obtained;

[0024] If the loss value does not meet the preset loss requirement, the improved detection model to be trained is subjected to parameter tuning and pre-training is performed again until the loss value meets the preset loss requirement.

[0025] Furthermore, the step of inputting the target image into the pre-trained improved detection model to obtain a preliminary detection result specifically includes:

[0026] Using the FPN deep learning network to extract and fuse multi-scale feature maps of the target image in a feature pyramid manner to obtain a fused target image;

[0027] Using the ResNet-50 backbone network structure, image feature extraction is performed on the fused target image to obtain an image feature extraction result corresponding to the target image;

[0028] Capturing the long-range dependency of the image features contained in the target image from the image feature extraction result corresponding to the target image according to the Non-LocalNet;

[0029] According to the image feature extraction result corresponding to the target image and the long-range dependency of the image features contained in the target image, the ResNet-50 backbone network structure is used as the detection backbone network to perform target detection on the fused target image to obtain the lesion information contained in the target image.

[0030] Furthermore, after executing the step of obtaining the annotation result made by the target user on the human-computer interaction interface using a preset annotation tool for the target image, the method further includes:

[0031] By parsing, identifying whether the annotation result is a null value;

[0032] The step of improving the preliminary detection result in combination with the annotation result to generate the final detection result specifically includes:

[0033] If the labeling result is a null value, the preliminary detection result is directly set as the final detection result;

[0034] If the annotation result is not a null value, the preliminary detection result is adjusted according to the adjustment field in the annotation result, and the adjusted result is obtained as the final detection result.

[0035] Furthermore, the step of automatically annotating the final detection result into the target image to obtain the annotated image specifically includes:

[0036] Starting a preset automatic annotation tool, wherein the automatic annotation tool includes an AI-based automatic auxiliary annotation tool;

[0037] Identifying lesion information contained in the final detection result, wherein the lesion information includes position information of the image region where the lesion is located and size information of the image region where the lesion is located;

[0038] The lesion information is annotated into the target image to obtain the annotated image.

[0039] Furthermore, the step of generating a detection report according to the annotated image and sending the detection report to a target receiving end specifically includes:

[0040] Adding the annotated image to the image position reserved in the test report generation template;

[0041] Identifying the marked area position information and the marked area size information from the marked image;

[0042] The marked area position information and the marked area size information are added to the text position reserved in the test report generation template in the form of text description information;

[0043] Obtaining a test report generation template after image addition and text description information supplementation, and outputting the test report based on the test report generation template;

[0044] According to the receiving address of the target receiving end, the test report is sent to the target receiving end, wherein the target receiving end includes a mobile phone end, a WeChat applet end or a preset test report printing end.

[0045] In order to solve the above technical problems, the embodiment of the present application also provides an image processing device, which adopts the following technical solution:

[0046] An image processing device, comprising:

[0047] A target image acquisition module, used for acquiring a target image;

[0048] A preliminary detection module, used for inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result;

[0049] A preliminary detection result display module, used to display the target image and the preliminary detection result on a preset human-computer interaction interface;

[0050] A labeling result acquisition module, used to acquire the labeling result made by the target user on the human-computer interaction interface with respect to the target image through a preset labeling tool;

[0051] A final test result acquisition module, used to improve the preliminary test result in combination with the annotation result to generate a final test result;

[0052] An automatic labeling module, used for automatically labeling the final detection result into the target image to obtain a labeled image;

[0053] The detection report generation and sending module is used to generate a detection report according to the labeled image and send the detection report to a target receiving end.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0055] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned image processing method when executing the computer-readable instructions.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0057] A computer-readable storage medium stores computer-readable instructions, which implement the steps of the image processing method described above when executed by a processor.

[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0059] The image processing method described in the embodiment of the present application is as follows: acquiring a target image; inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result; displaying the target image and the preliminary detection result to a preset human-computer interaction interface; acquiring the annotation result made by the target user on the target image through a preset annotation tool in the human-computer interaction interface; improving the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotating the final detection result to the target image to obtain an annotated image; generating an inspection report based on the annotated image, and sending the inspection report to the target receiving end. The image processing method described in the present application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information, and combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0062] Figure 2 is a flow chart of an embodiment of an image processing method according to the present application;

[0063] Figure 3 is a flowchart of a specific embodiment of pre-training an improved detection model in the image processing method described in the present application;

[0064] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;

[0065] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0066] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 206 is shown;

[0067] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 207 is shown;

[0068] Figure 8 is a schematic structural diagram of an embodiment of an image processing device according to the present application;

[0069] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0074] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0076] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0077] It should be noted that the image processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the image processing device is generally arranged in the server.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image processing method according to the present application. The image processing method comprises the following steps:

[0080] Step 201, acquiring a target image.

[0081] In this embodiment, the target image includes a medical image, specifically, for example, a breast X-ray image, which is used for identifying early breast lesions.

[0082] Step 202: input the target image into the pre-trained improved detection model to obtain a preliminary detection result.

[0083] In this embodiment, the improved detection model, for example, improves the target detection model of the DeepLab series, wherein the DeepLab series includes DeepLabv1, DeepLabv2, DeepLabv3, and DeepLabv3+.

[0084] A more accurate preliminary detection result is obtained by inputting the target image into a pre-trained improved detection model.

[0085] Step 203: display the target image and the preliminary detection result on a preset human-computer interaction interface.

[0086] Specifically, the output port of the improved detection model is connected to the preset human-computer interaction interface, so that after the improved detection model obtains a preliminary detection result, the preliminary detection result can be mapped to the human-computer interaction interface.

[0087] In this embodiment, the preset human-computer interaction interface provides the function of performing personalized operations on the target image. For example, doctors can easily view prediction results, adjust confidence thresholds, measure the distance between lesions and nipples in breast X-ray images, etc., which greatly improves the interaction experience between doctors and the system. At the same time, it can zoom in / out the target image, change the display of prediction results, etc., which is more in line with actual clinical workflow.

[0088] Step 204 : obtaining the annotation result made by the target user on the human-computer interaction interface using a preset annotation tool for the target image.

[0089] Specifically, the target user includes a doctor user who reviews the preliminary test results and the target image. The preset annotation tool is a processing component built into the human-computer interaction interface. The annotation tool allows the doctor to quickly locate the potential area using the patient's medical record information and select a suitable drawing tool to outline the lesion. In addition, an AI-assisted annotation function is also provided to further simplify the annotation process and reduce the workload of doctors.

[0090] Step 205, improving the preliminary detection result in combination with the annotation result to generate a final detection result.

[0091] In this embodiment, the preliminary test results are improved in combination with the annotation results to generate the final test results, so that the doctor can correct the preliminary test results and then generate the final test results, thus ensuring that the final test results are controlled by professionals.

[0092] Step 206: automatically annotate the final detection result into the target image to obtain an annotated image.

[0093] Specifically, the use of AI-assisted labeling function further simplifies the labeling process and reduces the workload of doctors.

[0094] Step 207: Generate a detection report based on the labeled image, and send the detection report to a target receiving end.

[0095] In this embodiment, a test report is generated based on the annotated image, and the test report is sent to a target receiving end, thereby ensuring that the patient can receive the test report in time.

[0096] In this embodiment, by acquiring a target image; inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result; displaying the target image and the preliminary detection result on a preset human-computer interaction interface; acquiring the annotation result made by the target user on the target image through a preset annotation tool on the human-computer interaction interface; improving the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotating the final detection result to the target image to obtain an annotated image; generating a detection report based on the annotated image, and sending the detection report to the target receiving end. The image processing method described in this application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information, and combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0097] Continue to refer Figure 3 In some optional implementations, a step of pre-training the improved detection model is also included before step 202. Figure 3 It is a flowchart of a specific embodiment of pre-training the improved detection model in the image processing method described in the present application, comprising the following steps:

[0098] Step 301, obtaining training samples, wherein the training samples include a batch of medical images with lesion information annotated;

[0099] Step 302: input the training samples into the improved detection model to be trained, perform detection training on the improved detection model, and obtain a pre-trained improved detection model, wherein the improved detection model includes a detection model based on an image target detection network.

[0100] In this embodiment, the improved detection model includes an improved DeepLab series of image target detection models that uses ResNet-50 as the main network structure and introduces FPN deep learning network and Non-LocalNet.

[0101] Specifically, by introducing FPN deep learning network and Non-LocalNet in the existing DeepLab series of image target detection models, ResNet-50 is still used as the main network structure, and image feature extraction and lesion detection can be performed on medical images using ResNet-50 as the main network structure. The residual network is used, and the gradient vanishing and degradation problems of the image target detection model can be avoided in a jump connection manner, thereby ensuring the reliability of the detection results; moreover, the introduction of FPN (Feature Pyramid Networks) deep learning network can perform multi-scale feature extraction on medical images, improve the accuracy of medical image feature extraction, and thus more accurately assist in lesion information detection; since Non-LocalNet is a neural network model based on a self-attention mechanism, the introduction of Non-LocalNet can capture the long-range dependency between the extracted image features. By improving the conventional detection model, the improved detection model is obtained, which not only realizes the extraction of medical image features from a multi-scale perspective, but also can capture the long-range dependency between the extracted image features, thereby ensuring the accuracy of target detection.

[0102] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes the following steps:

[0103] Step 401, using the FPN deep learning network to extract and fuse multi-scale feature maps of all training images in the training sample according to the feature pyramid method, to obtain all fused training images;

[0104] Specifically, the multi-scale feature map extraction according to the feature pyramid method includes two different extraction methods: a bottom-up extraction method, that is, starting from the input image, gradually extracting feature maps, the resolution of the feature maps gradually decreases, but the number of channels increases; a top-down extraction method, that is, starting from the feature map of the highest layer, adding it to the feature map of the next layer through upsampling, and so on, until the resolution of the input image is restored. The above two feature map extraction methods can be set by doctors according to actual needs.

[0105] Step 402, using the ResNet-50 backbone network structure to perform image feature extraction on all the fused training images, and obtain image feature extraction results corresponding to all the training images;

[0106] Step 403, capturing the long-range dependency of the image features respectively contained in all the training images according to the image feature extraction results respectively corresponding to all the training images by the Non-LocalNet;

[0107] Step 404, according to the image feature extraction results corresponding to all the training images and the long-range dependency of the image features respectively contained in all the training images, the ResNet-50 backbone network structure is used as the detection backbone network, and target detection is performed on all the fused training images to obtain detection results;

[0108] Step 405, using the DIoU loss function to calculate the loss value of the detection result compared to the labeled lesion information in the training sample;

[0109] In this embodiment, the DIoU loss function is adopted, which can make the loss function of the model converge more quickly and accurately compared with the traditional IoU loss, thereby improving the efficiency of model training.

[0110] Step 406: If the loss value meets the preset loss requirement, the pre-training of the improved detection model to be trained is completed, and a pre-trained improved detection model is obtained;

[0111] Step 407, if the loss value does not meet the preset loss requirement, the improved detection model to be trained is subjected to parameter tuning and pre-training is performed again until the loss value meets the preset loss requirement.

[0112] By pre-training the improved detection model, it is ensured that the pre-trained improved detection model outputs more accurate detection results.

[0113] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0114] Step 501, using the FPN deep learning network to extract and fuse multi-scale feature maps of the target image in a feature pyramid manner to obtain a fused target image;

[0115] Step 502, using the ResNet-50 backbone network structure to perform image feature extraction on the fused target image to obtain an image feature extraction result corresponding to the target image;

[0116] Step 503, capturing the long-range dependency of the image features contained in the target image from the image feature extraction result corresponding to the target image according to the Non-LocalNet;

[0117] Step 504, based on the image feature extraction result corresponding to the target image and the long-range dependency of the image features contained in the target image, the ResNet-50 backbone network structure is used as the detection backbone network to perform target detection on the fused target image to obtain the lesion information contained in the target image.

[0118] The improved detection model is used to detect lesion information on the target image, ensuring more accurate acquisition of lesion detection information to assist doctors in their work and reduce their workload.

[0119] In this embodiment, after executing the step of obtaining the annotation result made by the target user on the human-computer interaction interface by using a preset annotation tool for the target image, the method further includes: identifying whether the annotation result is a null value by parsing;

[0120] In this embodiment, the step of improving the preliminary detection result in combination with the annotation result to generate the final detection result specifically includes: if the annotation result is a null value, directly setting the preliminary detection result as the final detection result; if the annotation result is not a null value, adjusting the preliminary detection result according to the adjustment field in the annotation result, and obtaining the adjusted result as the final detection result.

[0121] Specifically, the preliminary detection result is adjusted according to the marking result, and when the preliminary detection result is sent incorrectly, correction processing can be performed in time, thereby ensuring that the final detection result is output under human control.

[0122] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 206 shown includes the following steps:

[0123] Step 601, starting a preset automatic labeling tool, wherein the automatic labeling tool includes an AI-based automatic auxiliary labeling tool;

[0124] Step 602, identifying the lesion information contained in the final detection result, wherein the lesion information includes the position information of the image region where the lesion is located and the size information of the image region where the lesion is located;

[0125] Step 603: annotate the lesion information into the target image to obtain the annotated image.

[0126] In this embodiment, AI-assisted labeling function is used to further simplify the labeling process and reduce the workload of doctors.

[0127] Continue to refer Figure 7 , Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 207 shown includes the following steps:

[0128] Step 701, adding the annotated image to the image position reserved in the test report generation template;

[0129] Step 702, identifying the position information and size information of the marked area from the marked image;

[0130] Step 703, adding the marked area position information and the marked area size information to the reserved text position of the test report generation template in the form of text description information;

[0131] Step 704, obtaining a test report generation template after the image is added and the text description information is supplemented, and outputting the test report based on the test report generation template;

[0132] Step 705, sending the test report to the target receiving end according to the receiving address of the target receiving end, wherein the target receiving end includes a mobile phone end, a WeChat applet end, or a preset test report printing end.

[0133] Specifically, by combining with a preset test report generation template, the annotated image is added to the image position reserved by the test report generation template, and the annotated area position information and the annotated area size information are supplemented to the text position reserved by the test report generation template in the form of text description information, and then the test report generation template after image addition and text description information supplement is obtained, and the test report is output based on the test report generation template. The test report generation is more intelligent and automated, reducing the workload of doctors.

[0134] This application obtains a target image; inputs the target image into a pre-trained improved detection model to obtain a preliminary detection result; displays the target image and the preliminary detection result on a preset human-computer interaction interface; obtains the annotation result made by the target user on the target image in the human-computer interaction interface through a preset annotation tool; improves the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotates the final detection result to the target image to obtain an annotated image; generates an inspection report based on the annotated image, and sends the inspection report to the target receiving end. The image processing method described in this application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information. Combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0135] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0136] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0137] In an embodiment of the present application, a target image is obtained; the target image is input into a pre-trained improved detection model to obtain a preliminary detection result; the target image and the preliminary detection result are displayed on a preset human-computer interaction interface; the target user obtains the annotation result made on the target image in the human-computer interaction interface through a preset annotation tool; the preliminary detection result is improved in combination with the annotation result to generate a final detection result; the final detection result is automatically annotated into the target image to obtain an annotated image; a detection report is generated according to the annotated image, and the detection report is sent to the target receiving end. The image processing method described in the present application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information, and combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0138] Further references Figure 8 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of an image processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0139] like Figure 8 As shown, the image processing device 800 described in this embodiment includes: a target image acquisition module 801, a preliminary detection module 802, a preliminary detection result display module 803, a labeling result acquisition module 804, a final detection result acquisition module 805, an automatic labeling module 806 and a detection report generation and sending module 807. Among them:

[0140] The target image acquisition module 801 is used to acquire the target image;

[0141] A preliminary detection module 802 is used to input the target image into a pre-trained improved detection model to obtain a preliminary detection result;

[0142] A preliminary detection result display module 803 is used to display the target image and the preliminary detection result on a preset human-computer interaction interface;

[0143] The annotation result acquisition module 804 is used to acquire the annotation result made by the target user on the human-computer interaction interface with respect to the target image through a preset annotation tool;

[0144] The final detection result acquisition module 805 is used to improve the preliminary detection result in combination with the annotation result to generate the final detection result;

[0145] An automatic labeling module 806 is used to automatically label the final detection result to the target image to obtain a labeled image;

[0146] The detection report generation and sending module 807 is used to generate a detection report according to the labeled image and send the detection report to a target receiving end.

[0147] This application obtains a target image; inputs the target image into a pre-trained improved detection model to obtain a preliminary detection result; displays the target image and the preliminary detection result on a preset human-computer interaction interface; obtains the annotation result made by the target user on the target image in the human-computer interaction interface through a preset annotation tool; improves the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotates the final detection result to the target image to obtain an annotated image; generates an inspection report based on the annotated image, and sends the inspection report to the target receiving end. The image processing method described in this application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information. Combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0150] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0151] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c which are interconnected through a system bus. It should be noted that: Fig. 9 Only a computer device 9 having a component memory 9a, a processor 9b, and a network interface 9c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0152] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0153] The memory 9a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk equipped on the computer device 9, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 9a can also include both the internal storage unit of the computer device 9 and its external storage device. In this embodiment, the memory 9a is generally used to store the operating system and various application software installed on the computer device 9, such as a computer-readable instruction of an image processing method, etc. In addition, the memory 9a can also be used to temporarily store various types of data that have been output or are to be output.

[0154] The processor 9b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run the computer-readable instructions stored in the memory 9a or process data, such as computer-readable instructions for running the image processing method.

[0155] The network interface 9c may include a wireless network interface or a wired network interface. The network interface 9c is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0156] The computer device proposed in this embodiment belongs to the field of research and development design and image processing technology, and is applied to medical image target detection and annotation scenarios. This application obtains a target image; inputs the target image into a pre-trained improved detection model to obtain a preliminary detection result; displays the target image and the preliminary detection result to a preset human-computer interaction interface; obtains the annotation result made by the target user on the target image in the human-computer interaction interface through a preset annotation tool; improves the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotates the final detection result to the target image to obtain an annotated image; generates an inspection report based on the annotated image, and sends the inspection report to the target receiving end. The image processing method described in this application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information, and then combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0157] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the image processing method as described above.

[0158] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and image processing technology, and is applied to medical image target detection and annotation scenarios. This application obtains a target image; inputs the target image into a pre-trained improved detection model to obtain a preliminary detection result; displays the target image and the preliminary detection result to a preset human-computer interaction interface; obtains the annotation result made by the target user on the target image in the human-computer interaction interface through a preset annotation tool; improves the preliminary detection result in combination with the annotation result to generate a final detection result; automatically annotates the final detection result to the target image to obtain an annotated image; generates an inspection report based on the annotated image, and sends the inspection report to the target receiving end. The image processing method described in this application is applied to the field of medical image detection, and the improved detection model can be used to more accurately obtain lesion detection information, and then combined with the personalized operation function and annotation tool provided in the human-computer interaction interface, it is convenient for doctors to perform personalized operations and corrections on the target image, and on the basis of intelligent detection of medical images, the workload of doctors is reasonably reduced.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0160] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image processing method, characterized in that: The steps include: Get the target image; Inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result; Displaying the target image and the preliminary detection result on a preset human-computer interaction interface; Obtaining a labeling result made by a target user on the human-computer interaction interface using a preset labeling tool for the target image; The preliminary detection result is improved in combination with the annotation result to generate a final detection result; Automatically annotating the final detection result into the target image to obtain an annotated image; A detection report is generated according to the annotated image, and the detection report is sent to a target receiving end.

2. The image processing method according to claim 1, characterized in that: Before executing the step of inputting the target image into the pre-trained improved detection model to obtain a preliminary detection result, the method further includes: Acquiring training samples, wherein the training samples include a batch of medical images with annotated lesion information; The training samples are input into an improved detection model to be trained, and detection training is performed on the improved detection model to obtain a pre-trained improved detection model, wherein the improved detection model includes a detection model based on an image target detection network.

3. The image processing method according to claim 2, characterized in that: The improved detection model includes an improved DeepLab series image target detection model with ResNet-50 as the main network structure, introducing FPN deep learning network and Non-Local Net, and the step of inputting the training sample into the improved detection model to be trained, performing detection training on the improved detection model, and obtaining a pre-trained improved detection model specifically includes: Using the FPN deep learning network to extract and fuse multi-scale feature maps of all training images in the training sample in a feature pyramid manner to obtain all fused training images; Using the ResNet-50 backbone network structure, image feature extraction is performed on all the fused training images to obtain image feature extraction results corresponding to all the training images; Capturing the long-range dependency of the image features respectively contained in all the training images according to the image feature extraction results respectively corresponding to all the training images by the Non-Local Net; According to the image feature extraction results corresponding to all training images and the long-range dependencies of the image features contained in all training images, the ResNet-50 backbone network structure is used as the detection backbone network, and target detection is performed on all fused training images to obtain detection results; Using the DI oU loss function, the loss value of the detection result compared with the labeled lesion information in the training sample is calculated; If the loss value meets the preset loss requirement, the pre-training of the improved detection model to be trained is completed, and a pre-trained improved detection model is obtained; If the loss value does not meet the preset loss requirement, the improved detection model to be trained is subjected to parameter tuning and pre-training is performed again until the loss value meets the preset loss requirement.

4. The image processing method according to claim 3, characterized in that: The step of inputting the target image into the pre-trained improved detection model to obtain a preliminary detection result specifically includes: Using the FPN deep learning network to extract and fuse multi-scale feature maps of the target image in a feature pyramid manner to obtain a fused target image; Using the ResNet-50 backbone network structure, image feature extraction is performed on the fused target image to obtain an image feature extraction result corresponding to the target image; capturing the long-range dependency of the image features contained in the target image from the image feature extraction result corresponding to the target image according to the Non-Local Net; According to the image feature extraction result corresponding to the target image and the long-range dependency of the image features contained in the target image, the ResNet-50 backbone network structure is used as the detection backbone network to perform target detection on the fused target image to obtain the lesion information contained in the target image.

5. The image processing method according to claim 1, characterized in that: After executing the step of obtaining the annotation result made by the target user on the human-computer interaction interface using a preset annotation tool for the target image, the method further includes: By parsing, identifying whether the annotation result is a null value; The step of improving the preliminary detection result in combination with the annotation result to generate the final detection result specifically includes: If the labeling result is a null value, the preliminary detection result is directly set as the final detection result; If the annotation result is not a null value, the preliminary detection result is adjusted according to the adjustment field in the annotation result, and the adjusted result is obtained as the final detection result.

6. The image processing method according to claim 4, characterized in that: The step of automatically annotating the final detection result into the target image to obtain the annotated image specifically includes: Starting a preset automatic annotation tool, wherein the automatic annotation tool includes an AI-based automatic auxiliary annotation tool; Identifying lesion information contained in the final detection result, wherein the lesion information includes position information of the image region where the lesion is located and size information of the image region where the lesion is located; The lesion information is annotated into the target image to obtain the annotated image.

7. The image processing method according to claim 1, characterized in that: The step of generating a detection report according to the labeled image and sending the detection report to a target receiving end specifically includes: Adding the annotated image to the image position reserved in the test report generation template; Identifying the marked area position information and the marked area size information from the marked image; The marked area position information and the marked area size information are added to the text position reserved in the test report generation template in the form of text description information; Obtaining a test report generation template after image addition and text description information supplementation, and outputting the test report based on the test report generation template; According to the receiving address of the target receiving end, the test report is sent to the target receiving end, wherein the target receiving end includes a mobile phone end, a WeChat applet end or a preset test report printing end.

8. An image processing device, characterized in that: include: A target image acquisition module, used for acquiring a target image; A preliminary detection module, used for inputting the target image into a pre-trained improved detection model to obtain a preliminary detection result; A preliminary detection result display module, used to display the target image and the preliminary detection result on a preset human-computer interaction interface; A labeling result acquisition module, used to acquire the labeling result made by the target user on the human-computer interaction interface with respect to the target image through a preset labeling tool; A final test result acquisition module, used to improve the preliminary test result in combination with the annotation result to generate a final test result; An automatic labeling module, used for automatically labeling the final detection result into the target image to obtain a labeled image; The detection report generation and sending module is used to generate a detection report according to the labeled image and send the detection report to a target receiving end.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images