An Image Training Set Transfer and Generation Method and System

Abstract

The present invention provides an image training set migration generation method and system, including: obtaining an original picture to be converted; inputting the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted. In the case of less training data when constructing an image training set, the present invention obtains pictures with a unified style through a style transfer algorithm for pictures collected from an open-source database or other devices, ensuring the data quality of the image training set and the network training effect, and reducing the workload of artificially synthesizing pictures.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and particularly relates to an image training set migration generation method and system. Background Art

[0002] In the medical field, capsule endoscopes have the advantages of being painless, non-invasive, and having a large amount of image information captured, and have extensive application value.

[0003] The prior art uses manual methods to identify the original pictures taken by capsule endoscopes and classify the original pictures. In order to identify the original pictures more accurately and efficiently, a model needs to be constructed. However, the model usually needs to be trained before use, and after the training is completed, the model can be used to identify the original pictures. The training data is the top priority in the current machine learning network training, and the data volume and data quality are the key points in network training. In practical applications, only using the data collected by itself, the data volume is difficult to meet the training requirements. As a result, the number of certain categories collected is small, affecting the final training effect.

[0004] The currently common processing methods are as follows. The first method is to perform oversampling processing, image augmentation processing, or weighted processing on the loss function for the pictures of the categories with small numbers to increase the number of training pictures or the picture weights. The second method is to use some open-source data or data collected by other medical devices to increase the number of training pictures.

[0005] The above two methods can, to a certain extent, increase the number of pictures of the categories with small numbers in the training set. However, for the first method, only oversampling processing, image augmentation processing, or weighted processing on the loss function is performed on the pictures. Although the number and diversity of the pictures are increased to a certain extent, due to the limited number of original pictures, the improvement space of these methods is limited to a certain extent, especially for scenarios where the number of pictures is originally small or the actual scenarios are relatively diverse. This method has little help. For the second method, although the data collected by using open-source data or other medical devices can also expand the training set, generally speaking, due to the differences in devices, the exposure, gain, white balance during image acquisition, and subsequent image processing, etc., can all lead to differences in the effects of the pictures taken, and there are certain differences between the pictures. How to unify these differences is a major problem currently faced.

[0006] Therefore, how to avoid the above defects, improve the processing efficiency of training pictures, and thus improve the accuracy of constructing an image training set has become an urgent problem to be solved. Summary of the Invention

[0007] The present invention provides an image training set migration generation method and system to solve the defects existing in the prior art.

[0008] In a first aspect, the present invention provides an image training set migration generation method, including:

[0009] Obtain the original picture to be converted;

[0010] Input the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0011] In one embodiment, the image training set migration generation model is obtained through the following steps:

[0012] Obtain an image training set for training a preset model; the preset model is a classification network model for identifying original pictures;

[0013] Determine the preset style picture and the set of pictures to be converted in the image training set;

[0014] Based on the preset style picture and the set of pictures to be converted, train the style transfer network to obtain the image training set migration generation model.

[0015] In one embodiment, the obtaining of the image training set for training a preset model includes:

[0016] Obtain the original pictures taken by a preset detection device and sample pictures having preset associated features with the original pictures;

[0017] Merge the original pictures and the sample pictures to obtain the image training set.

[0018] In one embodiment, the determining of the preset style picture and the set of pictures to be converted in the image training set includes:

[0019] Determine the picture source style corresponding to the result to be detected, and determine the preset style picture based on the picture source style;

[0020] Determine the remaining pictures in the image training set except the preset style pictures as the set of pictures to be converted.

[0021] In one embodiment, the training of the style transfer network based on the preset style picture and the set of pictures to be converted to obtain the image training set migration generation model includes:

[0022] Input the preset style picture and the set of pictures to be converted into the style transfer network;

[0023] Perform style conversion on the set of pictures to be converted based on the preset style picture to obtain a set of converted pictures;

[0024] Adjust the style weight and content weight of the set of converted pictures to obtain the image training set migration generation model.

[0025] In one embodiment, the preset detection device includes a capsule endoscope.

[0026] In a second aspect, the present invention further provides an image training set migration generation system, including:

[0027] An acquisition module, configured to acquire the original pictures to be converted;

[0028] A conversion module, configured to input the original pictures to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0029] In a third aspect, the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the image training set migration generation method as described in any one of the above are implemented.

[0030] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image training set migration generation method as described in any one of the above are implemented.

[0031] In a fifth aspect, the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the image training set migration generation method as described in any one of the above are implemented.

[0032] The image training set migration generation method and system provided by the present invention, in the case of less training data when constructing an image training set, through pictures collected from an open-source database or other devices, and through a style transfer algorithm, obtain pictures with a unified style, ensuring the data quality of the image training set and the network training effect, and reducing the workload of artificially synthesized pictures. Description of the Drawings

[0033] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0034] Figure 1 It is a schematic flowchart of the method for generating an image training set migration provided by the present invention;

[0035] Figure 2 It is an example diagram of a capsule endoscopy picture to be converted provided by the present invention;

[0036] Figure 3 It is an example diagram of a capsule endoscopy style picture provided by the present invention;

[0037] Figure 4 It is an example diagram of a converted capsule endoscopy picture provided by the present invention;

[0038] Figure 5 It is a schematic structural diagram of the system for generating an image training set migration provided by the present invention;

[0039] Figure 6 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0041] In view of the defects existing in the prior art, the present invention proposes a method for generating an image training set migration. Figure 1 It is a schematic flowchart of the method for generating an image training set migration provided by the present invention, as Figure 1 shown, including:

[0042] S1. Obtain the original picture to be converted;

[0043] S2. Input the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0044] Specifically, the present invention proposes to obtain a training set for training a preset model. It should be noted that the original images in the training set are taken by a capsule endoscope. The working process of the capsule endoscope is described as follows:

[0045] The capsule endoscope enters the digestive tract through the mouth and is naturally discharged from the anus.

[0046] The battery life of the capsule endoscope is limited, and its effective working range is the mouth, esophagus, stomach, duodenum, small intestine, and a part of the large intestine.

[0047] Each activity of the capsule endoscope generates in-domain inspection images and out-of-domain inspection images.

[0048] The in-domain inspection images are the shooting results of a certain section of the digestive tract.

[0049] The out-of-domain inspection images are the images taken incidentally by the capsule endoscope in addition to the in-domain inspection images.

[0050] All images can be automatically recognized without any manual intervention (including image preprocessing).

[0051] After the images are recognized, the images taken by the capsule endoscope are divided into six major categories (125 subcategories) and automatically saved in 125 image folders. Among them, the six major categories can be:

[0052] The first major category: a first type of out-of-domain classification label (10 categories);

[0053] The second major category: a second type of out-of-domain classification label (13 categories);

[0054] The third major category: a first target image classification label based on local structural features (14 categories);

[0055] The fourth major category: a first target image classification label for hole-like structures (8 categories);

[0056] The fifth major category: a first target image classification label based on global structural features (24 categories);

[0057] The sixth major category: a second target image classification label (56 categories).

[0058] Through the capsule endoscope, different parts of the digestive tract such as the mouth, esophagus, stomach, duodenum, small intestine, and large intestine can be automatically recognized.

[0059] The number of original images that each capsule endoscope can take each time can be 2,000 to 3,000, that is, the number of images in the image set obtained by the capsule endoscope.

[0060] The original pictures (JPG format) taken by the capsule endoscope that can be exported from the hospital information system without any processing.

[0061] In order to obtain an image training set with a unified style and a sufficient number of images, the present invention needs to train an image training set transfer generation model. A large number of original pictures to be converted are input into the trained image training set transfer generation model to obtain pictures with a unified style. A certain number of pictures with a unified style obtained are used to construct an image training set for training a preset model.

[0062] When constructing the image training set, in the case of less training data, the present invention obtains pictures with a unified style through the style transfer algorithm from pictures collected from an open-source database or other devices, ensuring the data quality of the image training set and the network training effect, and reducing the workload of artificially synthesized pictures.

[0063] Based on the above embodiments, the image training set transfer generation model proposed by the present invention is obtained through the following steps:

[0064] Obtain an image training set for training a preset model; the preset model is a classification network model for identifying original pictures;

[0065] Determine the preset style pictures and the set of pictures to be converted in the image training set;

[0066] Based on the preset style pictures and the set of pictures to be converted, train the style transfer network to obtain the image training set transfer generation model.

[0067] Specifically, first obtain a large number of image training sets, which are used to train a preset model. The preset model is generally used to obtain the final image processing results, such as image classification results, image recognition results, and image detection results, etc.

[0068] Select the preset style pictures as the benchmark from the image training set, and use the remaining pictures as the set of pictures to be converted. Select the style transfer network as the benchmark model, and then train the style transfer network with the preset style pictures and the set of pictures to be converted to obtain the image training set transfer generation model.

[0069] By constructing and training an image training set transfer generation model specifically for obtaining an image training set, the present invention can effectively solve the problem of low accuracy of image training data and realize the automatic construction and generation of pictures, reducing the workload of artificially synthesized pictures.

[0070] Based on any of the above embodiments, the obtaining of the image training set for training a preset model includes:

[0071] Obtain the original image captured by a preset detection device and a sample image having a preset associated feature with the original image;

[0072] Merge the original image and the sample image to obtain the image training set.

[0073] Specifically, for the image training set provided by the present invention, it mainly includes two parts: one is the image directly captured by the current capsule endoscope, and the other is the image obtained by shooting with a variety of capsule endoscope brands of other brands and an open-source endoscope image database, or a traditional gastroscope.

[0074] Here, by migrating some data from other open-source data or other medical devices, the required training set data can be quickly obtained, and the style of the obtained images will not vary greatly from the image data of the self-training set due to changes in the model number and shooting parameters of the collector.

[0075] Merge the multiple picture sample data obtained from the above two channels to obtain the image training set.

[0076] In the case of less training data, the present invention uses some open-source databases or images collected by other devices to obtain the required foreground images, thereby solving the problem of insufficient training data.

[0077] Based on any of the above embodiments, the determining the preset style image and the set of images to be converted in the image training set includes:

[0078] Determine the picture source style corresponding to the result to be detected, and determine the preset style image based on the picture source style;

[0079] Determine the remaining images in the image training set except the preset style image as the set of images to be converted.

[0080] Specifically, select one of the picture sources in the image training set as the unified style, that is, the reference image that needs to be converted into this style. The converted images refer to the images among the various capsule endoscope brands used in the hospital, some open-source endoscope images, and traditional gastroscopes, except for the style images. The rest are the images with the required conversion style and become the converted images.

[0081] The present invention can quickly unify the picture style, save the picture training time, and improve the network training effect by selecting the style image to train and fuse other converted images.

[0082] Based on any of the above embodiments, the training the style transfer network based on the preset style image and the set of images to be converted to obtain the image training set migration generation model includes:

[0083] Input the preset style image and the set of images to be converted into the style transfer network;

[0084] Perform style conversion on the set of images to be converted based on the preset style image to obtain a set of converted images;

[0085] Adjust the style weight and content weight of the set of converted images to obtain the image training set transfer generation model.

[0086] Specifically, the present invention uses a style image and a converted image to train the style transfer network, where the style transfer network adopts Linear Style Transfer (Learning Linear Transformations for Fast Image and Video Style Transfer).

[0087] Perform style conversion on the converted image through the style transfer network, and at the same time adjust the style and content weights so that the style and content of the converted image achieve a balanced effect.

[0088] When performing the above migration work compared with other algorithms, it is necessary to perform new model training or parameter adjustment on different data sources according to the actual image situation before data migration can be carried out. However, for the style transfer algorithm proposed by the present invention, since it directly extracts the style of the style image and the image content of the migrated image for fusion, it can be directly applied to all other open-source data that has not been trained or data collected by other medical devices without a large amount of network training or parameter adjustment work.

[0089] Figure 2 is an image to be converted obtained by a capsule endoscope, and is converted according to Figure 3 the set style image, and the final converted image shown in Figure 4 is obtained after the conversion is completed.

[0090] All the images obtained by the present invention through the open-source database or other devices are processed by the style transfer algorithm to unify the style to the style of the images collected by our own devices, which better guarantees the quality of the training data and the training effect of the network.

[0091] The image training set transfer generation system provided by the present invention will be described below. The image training set transfer generation system described below can be mutually corresponding and referred to the image training set transfer generation method described above.

[0092] Figure 5 is a schematic structural diagram of the image training set transfer generation system provided by the present invention. As Figure 5 shown, it includes: an acquisition module 51 and a conversion module 52, where:

[0093] The acquisition module 51 is used to acquire the original picture to be converted; the conversion module 52 is used to input the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0094] In the case of less training data when constructing an image training set, the present invention obtains pictures with a unified style through a style transfer algorithm for the pictures collected from an open-source database or other devices, ensuring the data quality of the image training set and the network training effect, and reducing the workload of synthetic pictures.

[0095] Figure 6 An example of the physical structure diagram of an electronic device is shown as Figure 6 shown. The electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 complete communication with each other through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the image training set migration generation method, which includes: acquiring the original picture to be converted; inputting the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0096] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image training set migration generation method provided by the above-mentioned various methods. The method includes: obtaining an original picture to be converted; inputting the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0098] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the image training set migration generation method provided by the above-mentioned various methods. The method includes: obtaining an original picture to be converted; inputting the original picture to be converted into a pre-trained image training set migration generation model to obtain an image training set; wherein, the image training set migration generation model is obtained by training a style transfer network based on a preset style picture and a set of pictures to be converted.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0100] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image training set transfer and generation method, characterized in that Including: Obtain the original image to be converted; Input the original image to be converted into a pre-trained image training set transfer generation model to obtain an image training set; wherein, the image training set transfer generation model is obtained by training a style transfer network based on a preset style image and a set of images to be converted; The image training set transfer generation model is obtained through the following steps: Obtain an image training set for training a preset model; the preset model is a classification network model for identifying original images; the image training set for training the preset model is obtained in the following manner: obtain the original images taken by a capsule endoscope and sample images having preset associated features with the original images; merge the original images and the sample images to obtain the image training set; the image training set includes two parts: one is the images directly taken by the current capsule endoscope, and the other is the images obtained by shooting with various capsule endoscope brands of other brands and an open-source endoscope image database, or traditional gastroscopy; Determine the preset style image and the set of images to be converted in the image training set; Based on the preset style image and the set of images to be converted, train the style transfer network to obtain the image training set transfer generation model.

2. The method for migrating and generating an image training set according to claim 1, wherein The determination of the preset style image and the set of images to be converted in the image training set includes: Determine the source style of the image corresponding to the result to be detected, and determine the preset style image based on the source style of the image; Determine the remaining images in the image training set except the preset style image as the set of images to be converted.

3. The method for migrating and generating an image training set according to claim 1, wherein The training of the style transfer network based on the preset style image and the set of images to be converted to obtain the image training set transfer generation model includes: Input the preset style image and the set of images to be converted into the style transfer network; Based on the preset style image, perform style conversion on the set of images to be converted to obtain a set of converted images; Adjust the style weight and content weight of the set of converted images to obtain the image training set transfer generation model.

4. An image training set migration and generation system, characterized in that, Including: An acquisition module for acquiring the original image to be converted; A conversion module for inputting the original image to be converted into a pre-trained image training set transfer generation model to obtain an image training set; wherein, the image training set transfer generation model is obtained by training a style transfer network based on a preset style image and a set of images to be converted; The image training set transfer generation model is obtained through the following steps: Obtain an image training set for training a preset model; the preset model is a classification network model for identifying original pictures; the image training set for training the preset model is obtained through the following method: obtain the original pictures taken by a capsule endoscope and sample pictures having preset associated features with the original pictures; merge the original pictures and the sample pictures to obtain the image training set; the image training set includes two parts: one is the pictures directly taken by the current capsule endoscope, and the other is the pictures obtained by taking pictures with multiple capsule endoscope brands of other brands and an open-source endoscope picture database, or traditional gastroscopy. Determine the preset style pictures and the set of pictures to be converted in the image training set. Based on the preset style pictures and the set of pictures to be converted, train the style transfer network to obtain the image training set migration generation model.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the image training set migration generation method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image training set migration generation method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image training set migration generation method according to any one of claims 1 to 3.

Images