Upper digestive tract endoscopy image generation method, device, electronic device and storage medium

Abstract

The present invention discloses a method, device, electronic device and storage medium for generating an upper gastrointestinal endoscopic image, wherein the method comprises: obtaining an original upper gastrointestinal image of a patient and characteristic information of the patient; determining identification feature information according to an identification feature model, the original upper gastrointestinal image and the characteristic information of the patient, wherein the identification feature model is generated by training training samples including historical gastrointestinal images and historical patient characteristic information; determining a target area of ​​the original upper gastrointestinal image according to the identification feature information to generate an upper gastrointestinal endoscopic image. The embodiment of the present invention processes the characteristic information of the patient and the gastrointestinal image to obtain identification feature information, achieves accuracy in upper gastrointestinal image feature extraction, generates a highlight target area through identification feature information, enhances the display effect of the upper gastrointestinal endoscopic image, and assists doctors in reducing the misdiagnosis rate and missed diagnosis rate of the examination.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for generating an upper gastrointestinal endoscopic image. Background Art

[0002] Upper gastrointestinal endoscopy is widely used in the diagnosis of diseases of the esophagus, stomach, duodenum, etc. Traditional upper gastrointestinal endoscopy images only include specific images collected from the upper gastrointestinal tract. When doctors use upper gastrointestinal endoscopy images for examination, they completely rely on their professional attitude, operation techniques, and experience. The same upper gastrointestinal endoscopy image may have different examination results under the interpretation of different doctors. In recent years, with the rapid development of artificial intelligence technology represented by deep learning, experts in the field have gradually introduced deep learning into the generation process of upper gastrointestinal endoscopy images.

[0003] When existing upper gastrointestinal endoscopic images are generated based on deep learning, only the collected endoscopic images are used as a single input. The quality of endoscopic image generation depends entirely on the richness of the training data, and the endoscopic images of different patients are homogenized without considering the individual differences of patients. For example, the same endoscopic image generation method is used for a 70-year-old patient with a family history of genetic diseases and a 30-year-old patient with no family history of genetic diseases. As a result, the generated upper gastrointestinal endoscopic images cannot truly reflect the patient's condition. When doctors use the above endoscopic images to diagnose upper gastrointestinal symptoms, it is easy to cause missed diagnosis and misdiagnosis. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for generating upper gastrointestinal endoscopic images, so as to realize rapid generation of upper gastrointestinal endoscopic images, perform personalized adjustment according to patient information, enhance the accuracy of upper gastrointestinal endoscopic image generation, and assist doctors in making reliable diagnosis.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating an upper gastrointestinal endoscopic image, the method comprising: acquiring an original upper gastrointestinal image of a patient and characteristic information of the patient; determining identification feature information based on an identification feature model, the original upper gastrointestinal image and the characteristic information of the patient, wherein the identification feature model is generated by training with training samples including historical upper gastrointestinal images and historical patient characteristic information; determining a target area of ​​the original upper gastrointestinal image based on the identification feature information to generate an upper gastrointestinal endoscopic image.

[0006] In a second aspect, an embodiment of the present invention provides an upper gastrointestinal endoscopic image generation device, the device comprising:

[0007] A data acquisition module, used to acquire the original upper digestive tract image of the patient and the characteristic information of the patient;

[0008] A model processing module, used to determine identification feature information according to an identification feature model, the original upper consumption tract image and the feature information of the patient, wherein the identification feature model is generated by training a training sample including historical consumption tract images and historical patient feature information;

[0009] An image generation module is used to determine the target area of ​​the original digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image.

[0010] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0011] one or more processors;

[0012] a memory for storing one or more programs,

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for generating an upper gastrointestinal endoscopic image as described in any one of the embodiments of the present invention.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for generating an upper gastrointestinal endoscopic image as described in any one of the embodiments of the present invention.

[0015] The embodiment of the present invention collects the original upper gastrointestinal tract image and the patient's characteristic information, processes the collected original upper gastrointestinal tract image and the patient's characteristic information through an identification feature model to obtain identification feature information, determines the target area in the original upper gastrointestinal tract image according to the identification feature information to generate an upper gastrointestinal endoscopic image, enhances the original upper gastrointestinal tract image through the patient's characteristic information attributes, improves the generation quality of the gastrointestinal image, truly reflects the patient's upper gastrointestinal health condition, assists doctors in examination by highlighting the characteristic area, and reduces the missed diagnosis rate and misdiagnosis rate of the doctor's examination. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a method for generating an upper gastrointestinal endoscopic image provided by Embodiment 1 of the present invention;

[0017] Figure 2 is a structural diagram of an identification feature model provided in the first embodiment of the present invention;

[0018] Figure 3 is a flow chart of a method for generating an upper gastrointestinal endoscopic image provided by Embodiment 2 of the present invention;

[0019] Figure 4 is a schematic structural diagram of an upper gastrointestinal endoscopic image generating device provided in Embodiment 3 of the present invention;

[0020] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention are shown in the accompanying drawings, rather than all structures. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0022] Embodiment 1

[0023] Figure 1 is a flow chart of a method for generating an upper gastrointestinal endoscopic image provided by Embodiment 1 of the present invention. The embodiment of the present invention can be applied to the case of generating an upper gastrointestinal image to assist doctors in diagnosis. The method can be executed by an upper gastrointestinal endoscopic image generating device, which can be implemented in the form of sampling hardware and / or software. Figure 1 The method provided in the embodiment of the present invention specifically includes the following steps:

[0024] Step 110: Acquire the original upper digestive tract image of the patient and the characteristic information of the patient.

[0025] The original upper gastrointestinal image may be image data of the patient's upper gastrointestinal tract collected by endoscopy, and the original digestive image may include pictures and videos. The patient's characteristic information may be information reflecting the patient's characteristics, including age, gender, upper gastrointestinal symptoms, family genetic disease history of the upper gastrointestinal tract, other high-risk factors, and the like.

[0026] Specifically, the original upper gastrointestinal images of the patient can be collected, which can include collecting them directly from the patient's body through an endoscope or collecting them from generated endoscopic images. For example, the upper gastrointestinal endoscopic images of the patient that have been generated in the past can be collected by scanning. The patient's age, gender, upper gastrointestinal symptoms, family genetic disease history of the upper gastrointestinal tract, other high-risk factors and other information can also be obtained as the patient's characteristic information. The above patient's characteristic information can be input by the patient himself or collected from the case.

[0027] Step 120: determining identification feature information according to an identification feature model, the original upper digestive tract image and the patient's feature information, wherein the identification feature model is generated by training with training samples including historical digestive tract images and historical patient feature information.

[0028] Among them, the identification feature model can be a deep learning model that processes the original digestive tract images and the patient's characteristic information to obtain the identification feature information. The identification feature model can specifically include a convolutional neural network model. The identification feature model can include a convolution layer, a normalization layer, and a nonlinear activation function, etc. The identification feature model can specifically include one or more neural network models. For example, the identification feature model is composed of two neural network models, one neural network model is used to process the original upper digestive tract images, and the other neural network model is used to process the patient's physical information. The historical digestive tract images can be upper digestive tract images that have been diagnosed by doctors in the past, and the characteristic information of historical patients can be characteristic information of patients who have undergone upper digestive tract diagnosis in the past. The training sample can be a data set that includes at least a threshold number of historical digestive tract images and characteristic information of historical patients.

[0029] In an embodiment of the present invention, the acquired original upper gastrointestinal image and the patient's body information can be input into the gastrointestinal identification feature model, and the identification feature information output by the gastrointestinal identification feature model can be obtained, wherein the identification feature information can be used to emphasize the target area in the upper gastrointestinal image, for example, the target area can be the patient's Helicobacter pylori infection area or the lesion area such as gastric ulcer. Specifically, the identification feature model can be generated in advance through training samples, and the training samples include historical gastrointestinal images and past patient information exceeding a threshold number. The historical gastrointestinal images can be annotated with lesions by experienced doctors, which can reflect the location of lesions in each historical gastrointestinal image.

[0030] Step 130: Determine the target area of ​​the original digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image.

[0031] The target area may be an image area corresponding to the identification feature information, and the target area may represent an image area where a lesion exists in the original digestive tract image.

[0032] In an embodiment of the present invention, the corresponding target area can be searched in the original digestive tract image according to the identification feature information, and the target area can be marked in the original digestive tract image by drawing a frame, changing the color, or changing the background, and the marked image can be used as an upper digestive tract endoscopic image. In an exemplary embodiment, the identification feature information can be a binary image of dimension n*n, each element can represent a position area in the original digestive tract image, a component element of 0 can indicate that there is no lesion at the corresponding position, and a component element of 1 can indicate that there is a lesion at the corresponding position. The lesion position in the original digestive tract image can be determined by the above-mentioned binary image, and the lesion position can be circled with a red line, and the processed original digestive tract image can be used as an upper digestive tract endoscopic image.

[0033] The embodiments of the present invention obtain the original upper gastrointestinal tract image of the patient and the characteristic information of the patient, input the original upper gastrointestinal tract image and the characteristic information of the patient into an identification feature model to obtain the identification feature information, determine the target area in the original upper gastrointestinal tract image according to the identification feature information, and generate an upper gastrointestinal endoscopic image according to the target area, thereby improving the generation quality of the upper gastrointestinal endoscopic image, enriching the display information of the upper gastrointestinal image according to the characteristic information of the patient, enhancing the accuracy of the upper gastrointestinal endoscopic image, assisting doctors in upper gastrointestinal tract diagnosis, and reducing the missed diagnosis rate and misdiagnosis rate during the doctor's examination process.

[0034] Furthermore, on the basis of the above-mentioned embodiments of the invention, determining the identification feature information corresponding to the original upper gastrointestinal tract image and the patient's body information according to the identification feature model includes: processing the original upper gastrointestinal tract image to generate a convolution feature map; processing the patient's feature information to generate a feature vector; processing the feature vector and the convolution feature map to generate a fusion feature matrix; processing the fusion feature matrix to generate identification feature information.

[0035] In the embodiment of the present invention, the identification feature model may include four neural network models: a convolution feature extraction branch, a fully connected feature extraction branch, a feature fusion branch, and a feature decoding branch. Figure 2 In the identification feature model, the convolution feature extraction branch and the fully connected feature extraction branch process the original upper gastrointestinal tract image and the patient identity information respectively, and use the processing results as the input information of the feature fusion branch. Finally, the feature decoding branch can decode the output result of the feature fusion branch to obtain the tract identification feature information.

[0036] Specifically, the convolutional feature extraction branch can be composed of a convolutional layer, a batch normalization layer, and a nonlinear activation function. The input original upper gastrointestinal tract image can be processed by multiple superposition combinations to extract the multi-dimensional convolutional feature map of the image step by step. For example, taking the convolutional feature extraction branch of the U-net network structure as an example, assuming that the resolution of the original upper gastrointestinal tract image input is 1024*1024 and the number of channels is 3, that is, the dimension of the input image S0 is 3*1024*1024. The input image is processed by a convolution layer, a batch normalization layer, and a combination of nonlinear functions to obtain a feature map S1 with a dimension of 64*512*512. Then S1 is again processed by a convolution layer, a batch normalization layer, and a combination of nonlinear functions to obtain a feature map S2 with a dimension of 128*256*256. Similarly, S2 is used as input to obtain a feature map S3 with a dimension of 256*128*128, S3 is used as input to obtain a feature map S4 with a dimension of 512*64*64, and S4 is used as input to obtain a feature map S5 with a dimension of 1024*32*32.

[0037] In an exemplary embodiment, a fully connected feature extraction branch may be composed of a fully connected layer and a nonlinear activation function. The patient's physical information input into the fully connected feature extraction branch may be subjected to multiple superposition and combination processes to obtain a feature vector with a higher dimension. For example, the patient's physical information may be encoded to obtain a 1*6-dimensional feature G0, which is processed through a fully connected layer and a nonlinear activation function to obtain a feature G1 with a dimension of 1*1024. G1 is then processed through a fully connected layer and a nonlinear activation function to obtain a feature G2 with a dimension of 1*128. G1 is again used as input and processed through a fully connected layer and a nonlinear activation function to obtain a feature G3 with a dimension of 1*256. Similarly, G1 is used as input to obtain a feature G4 with a dimension of 1*512 and a feature G5 with a dimension of 1*1024.

[0038] Specifically, the feature fusion branch can obtain the corresponding fused feature matrix by combining the convolution feature map and feature vector of the convolution feature extraction branch and the fully connected feature extraction branch. For example, the feature vector can be directly used as one dimension of the convolution feature map, or the elements in the feature vector can be grouped into weight values, and the product of the weight value and the convolution feature map is determined as the fused feature matrix.

[0039] In the embodiment of the present invention, the feature decoding branch is composed of an upsampling layer, a convolution layer, a batch normalization layer, and a nonlinear activation function. The fused feature matrix output by the feature fusion branch can be processed by multiple superpositions of the convolution layer, the upsampling layer, the batch normalization layer, and the nonlinear activation function to obtain a feature map with a gradually improved resolution, and then the final identification feature information is obtained through the excitation function.

[0040] Further, based on the above-mentioned embodiment of the invention, the feature vector and the convolution feature map are processed to generate a fusion feature matrix, including:

[0041] The constituent elements of the feature vector are used as weight values ​​to determine the multiplication result of the weight value and the corresponding channel of the convolution feature map; the multiplication result is processed by at least one convolution, batch normalization and nonlinear activation function to generate a fusion feature matrix.

[0042] In an embodiment of the present invention, the feature fusion branch combines the convolution feature map and feature vector of the attention mechanism convolution feature extraction branch and the fully connected feature extraction branch to obtain the corresponding fusion feature matrix. Exemplarily, each element of the feature vector G5 can be used as an attention weight to multiply the corresponding channel of the convolution feature map S5, and then processed by a combination of a convolution layer, a batch normalization layer, and a nonlinear function to obtain a feature map A5 with a dimension of 1024*32*32. Similarly, G4 is used as an attention weight to fuse with S4 to obtain a feature map A4 with a dimension of 512*64*64, G3 is fused with S3 to obtain a feature map A3 with a dimension of 256*128*128, and G2 is fused with S2 to obtain a feature map A2 with a dimension of 128*256*256.

[0043] Furthermore, based on the above-mentioned embodiment of the invention, the identification feature information includes lesion location information and lesion attribute information.

[0044] In an embodiment of the present invention, the identification feature information may indicate a position in the original upper gastrointestinal tract image that needs to be identified, specifically a position where an upper gastrointestinal tract lesion exists. The identification feature information may include a lesion position and lesion attribute information, wherein the lesion position may indicate information on the location of the lesion, which may be coordinate information in the original upper gastrointestinal tract image; the lesion attribute information may indicate information on lesion attributes, which may include information such as the name of the disease to which the lesion belongs and the size of the lesion.

[0045] Embodiment 2

[0046] Figure 3 is a flow chart of a method for generating an upper gastrointestinal endoscopic image provided by Embodiment 2 of the present invention. This embodiment of the present invention is a specific embodiment of the above-mentioned invention embodiment, and the generation of training samples and the identification of target areas are specified. Figure 3 The method provided in the embodiment of the present invention specifically includes the following steps:

[0047] Step 210: Obtain historical digestive tract images and historical patient feature information corresponding to the historical digestive tract images.

[0048] In an embodiment of the present invention, historical digestive tract images and historical patient characteristic information can be collected through historical cases or a network shared database. It can be understood that each historical digestive tract image can have its corresponding historical patient characteristic information, and one historical patient characteristic information can correspond to multiple historical digestive tract images.

[0049] Step 220: annotate the lesions in the historical digestive tract images to generate lesion samples and non-lesion samples.

[0050] The lesion samples may be an image set of historical digestive tract images containing upper digestive tract lesions, and the non-lesion samples may be an image set of historical digestive tract images that do not contain upper digestive tract lesions.

[0051] In an embodiment of the present invention, the lesions in the historical digestive tract images can be annotated. For example, the annotation method can include circling the lesion location and adding text information at the lesion location. The historical digestive tract image annotation method can be manually performed by an experienced doctor, or automatically generated by a computer through image recognition technology. After the historical digestive tract images are annotated, the historical digestive tract images can be divided into lesion samples and non-lesion samples according to whether there are lesions in the images.

[0052] Step 230: classify and quantify the historical patient characteristic information according to the attribute information to obtain an attribute information vector.

[0053] In an embodiment of the present invention, historical patient characteristic information can be divided into different information categories according to different dimensions of corresponding attribute information, each information category can be scored, and the scoring result can be used as the attribute information vector corresponding to the historical patient characteristic information. For example, past patient information can be divided according to age, gender, upper gastrointestinal symptoms, upper gastrointestinal family genetic disease history, other family genetic disease history and other high-risk factors, and each item can be scored. Past patient information can be converted into a multi-dimensional attribute information vector. An information classification scoring table can be expressed as follows:

[0054] Table 1 Information classification scoring table

[0055]

[0056] Step 240: Merge lesion samples, non-lesion samples and attribute information vectors as training samples according to the correspondence between historical digestive tract images and historical patient feature information.

[0057] Specifically, since there is a correspondence between historical digestive tract images and historical patient feature information, correspondingly, the historical digestive tract images in the determined lesion samples and non-lesion samples may have their own corresponding attribute information vectors, and the attribute information vectors may be merged with the corresponding lesion samples and non-lesion samples, and the merged lesion samples and non-lesion samples may be used as training samples. Further, after obtaining the training samples, the neural network model may be trained through the training samples to generate an identification feature model.

[0058] Step 250: Acquire the original upper digestive tract image of the patient and the characteristic information of the patient.

[0059] Step 260: Perform at least one preprocessing operation of brightness adjustment, rotation, and translation on the original upper digestive tract image.

[0060] In an embodiment of the present invention, the original upper gastrointestinal tract image may be unclear due to the limitations of the acquisition means and acquisition equipment. The original upper gastrointestinal tract image can be subjected to brightness adjustment, rotation, translation and other operations to improve the display quality of the original upper gastrointestinal tract image, so as to enhance the accuracy of the upper gastrointestinal endoscopic image in displaying the target area.

[0061] Step 270: determine identification feature information according to the identification feature model, the original upper consumption tract image and the patient's feature information, wherein the identification feature model is generated by training with training samples including historical consumption tract images and historical patient feature information.

[0062] The identification feature model may be generated in advance through training with training samples, and the generated identification feature model may be saved.

[0063] Specifically, the original upper digestive tract image and the patient's characteristic information can be input into a pre-trained identification feature model for processing, and the output result of the identification feature model can be used as the identification feature information. It is understandable that the identification feature information can include lesion location information and lesion attribute information corresponding to the lesion in the original upper digestive tract image.

[0064] Step 280: Search the original digestive tract image for a target area corresponding to the lesion location information in the identification feature information.

[0065] The target area may be an area in the original digestive tract image that needs to be highlighted, and the target area may be a lesion or an organ.

[0066] In an embodiment of the present invention, the lesion location information in the identification feature information can be extracted, wherein the lesion location information can indicate the location of the lesion, and can be the location coordinates of the lesion in the original upper digestive tract image, and the location coordinates can be the coordinates of the edge of the lesion, or can be the coordinates of all lesions. The corresponding target area is obtained through the location coordinates, for example, the connection area of ​​the location coordinates can be used as the target area.

[0067] Step 290: prominently mark the target area and display the lesion attribute information in the marked feature information on the original digestive tract image to generate an upper digestive tract endoscopic image.

[0068] Specifically, after the target area is determined, a prominent mark can be made at the corresponding position of the original upper digestive tract image. For example, the method of prominently marking the target area may include increasing the brightness of the target area or reducing the brightness of the area other than the target area, and circling the target area in the original upper digestive tract image. The lesion attribute information in the identification feature information can be extracted, and the lesion attribute information can be displayed in the original upper digestive tract image of the significantly marked target area, and can be marked around the target area or marked in the target area, so as to realize the display of lesion attribute information such as lesion type and lesion size.

[0069] The embodiment of the present invention obtains historical digestive tract images and historical patient characteristic information, generates lesion samples and non-lesion samples after marking lesions on historical digestive tract images, classifies and quantifies historical patient characteristic information according to attribute information to generate attribute information vectors, uses lesion samples, non-lesion samples and attribute information vectors as training samples, collects original upper digestive tract images of patients and characteristic information of patients, processes the original upper digestive tract images using pre-operations such as brightness adjustment, rotation and translation, processes the pre-processed original upper digestive tract images and patient body information using an identification feature model generated according to training samples to obtain tract identification feature information, determines a target position in the original digestive tract image based on lesion position information of the identification feature information, significantly identifies the target position and displays the lesion attribute information in the original digestive tract image to realize the generation of upper digestive tract endoscopic images, improves the display quality of digestive tract endoscopic images, generates endoscopic images through personalized patient body information, improves the display accuracy of target areas in endoscopic images, assists doctors in upper digestive tract examinations, and reduces doctors' missed diagnosis rate and misdiagnosis rate.

[0070] Embodiment 3

[0071] Figure 4 This is a schematic diagram of the structure of an upper gastrointestinal endoscopic image generation device provided in the third embodiment of the present invention, which can execute the upper gastrointestinal endoscopic image generation method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. The device can be implemented by software and / or hardware, specifically including: a data acquisition module 301, a model processing module 302 and an image generation module 303.

[0072] The data acquisition module 301 is used to acquire the original upper digestive tract image of the patient and the characteristic information of the patient.

[0073] The model processing module 302 is used to determine the identification feature information based on the identification feature model, the original upper consumption tract image and the characteristic information of the patient, wherein the identification feature model is generated by training a training sample including historical consumption tract images and historical patient characteristic information.

[0074] The image generation module 303 is used to determine the target area of ​​the original digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image.

[0075] In the embodiment of the present invention, the original upper gastrointestinal tract image of the patient and the characteristic information of the patient are acquired through the data acquisition module, the model processing module processes the original upper gastrointestinal tract image and the patient's body information through the identification feature model to obtain the identification feature information, the image generation module determines the corresponding target area in the original upper gastrointestinal tract image according to the identification feature information, and uses the image after the target area is marked as the upper gastrointestinal endoscopic image, thereby improving the quality of upper gastrointestinal endoscopic image generation, improving the personalization of feature identification according to the patient's characteristic information, enhancing the accuracy of upper gastrointestinal endoscopic images, assisting doctors in upper gastrointestinal tract diagnosis, and reducing the missed diagnosis rate and misdiagnosis rate during the doctor's examination process.

[0076] Further, based on the above-mentioned embodiment of the invention, the model processing module 302 includes:

[0077] The convolution feature extraction branch is used to process the original upper digestive tract image to generate a convolution feature map.

[0078] The fully connected feature extraction branch is used to process the patient feature information and generate a feature vector.

[0079] The feature fusion branch is used to process the feature vector and the convolution feature map to generate a fused feature matrix.

[0080] The feature decoding branch is used to process the fused feature matrix to generate identification feature information.

[0081] Furthermore, based on the above-mentioned embodiments of the invention, the feature fusion branch is specifically used to: use the constituent elements of the feature vector as weight values, determine the multiplication result of the weight value and the corresponding channel of the convolution feature map; and generate a fused feature matrix after subjecting the multiplication result to at least one convolution, batch normalization and nonlinear activation function processing.

[0082] Furthermore, on the basis of the above-mentioned embodiments of the invention, it also includes: a sample generation module, which is specifically used to: obtain historical digestive tract images and historical patient characteristic information corresponding to the historical digestive tract images; annotate the lesions in the historical digestive tract images to generate lesion samples and non-lesion samples; classify and quantify the historical patient characteristic information according to the attribute information as an attribute information vector; and merge the lesion samples, the non-lesion samples and the attribute information vector as training samples according to the correspondence between the historical digestive tract images and the historical patient characteristic information.

[0083] Further, based on the above-mentioned embodiment of the invention, the identification feature information in the model processing module 302 includes lesion location information and lesion attribute information.

[0084] Further, based on the above-mentioned embodiment of the invention, the image generation module 303 includes:

[0085] The region determination unit is used to search the target region corresponding to the lesion location information in the identification feature information in the original digestive tract image.

[0086] An image generating unit is used to significantly mark the target area and display the lesion attribute information in the marking feature information on the original digestive tract image to generate an upper digestive tract endoscopic image.

[0087] Furthermore, based on the above-mentioned embodiment of the invention, the device further includes: a preprocessing module, which is used to perform at least one preprocessing operation of brightness adjustment, rotation, and translation on the original upper digestive tract image.

[0088] Embodiment 4

[0089] Figure 5 is a schematic diagram of the structure of an electronic device provided by Embodiment 4 of the present invention, such as Figure 5 As shown, the electronic device includes a processor 40, a memory 41, an input device 42 and an output device 43; the number of processors 40 in the electronic device can be one or more. Figure 5 A processor 40 is taken as an example; the processor 40, the memory 41, the input device 42 and the output device 43 in the electronic device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0090] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the upper gastrointestinal endoscopic image generation method in the embodiment of the present invention (for example, the data acquisition module 301, the model processing module 302 and the image generation module 303 in the upper gastrointestinal endoscopic image generation device). The processor 40 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 41, that is, realizes the above-mentioned upper gastrointestinal endoscopic image generation method.

[0091] The memory 41 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include a memory remotely arranged relative to the processor 40, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0092] The input device 42 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 43 may include a display device such as a display screen.

[0093] Embodiment 5

[0094] Embodiment 5 of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute a method for generating an upper gastrointestinal endoscopic image when executed by a computer processor, the method comprising:

[0095] Acquire the patient's original upper gastrointestinal tract image and the patient's characteristic information; determine the identification feature information based on the identification feature model, the original upper gastrointestinal tract image and the patient's characteristic information, wherein the identification feature model is generated by training with training samples including historical upper gastrointestinal tract images and historical patient characteristic information; determine the target area of ​​the original upper gastrointestinal tract image based on the identification feature information to generate an upper gastrointestinal tract endoscopic image.

[0096] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided in an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the upper gastrointestinal endoscopic image generation method provided in any embodiment of the present invention.

[0097] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0098] It is worth noting that in the above-mentioned embodiment of the upper gastrointestinal endoscopic image generating device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0099] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for generating an upper gastrointestinal endoscopic image, characterized in that: The method comprises: Acquire the original upper gastrointestinal tract image and characteristic information of the patient; Determine identification feature information according to an identification feature model, the original upper digestive tract image and feature information of the patient, wherein the identification feature model is generated by training with training samples including historical digestive tract images and historical patient feature information; Determining the target area of ​​the original upper digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image includes: marking the target area in the original upper digestive tract image by using at least one of a frame, a color change, and a background change; The determining of the identification feature information according to the identification feature model, the original upper digestive tract image and the feature information of the patient comprises: Processing the original upper digestive tract image to generate a convolution feature map; Processing the patient characteristic information to generate a characteristic vector; Processing the feature vector and the convolution feature map to generate a fused feature matrix; Processing the fused feature matrix to generate identification feature information; Wherein, the identification feature information of the fused feature matrix is ​​obtained by combining the convolution feature map and the feature vector; The step of processing the feature vector and the convolution feature map to generate a fusion feature matrix includes: Taking the constituent elements of the feature vector as weight values, determining the multiplication result of the weight value and the corresponding channel of the convolution feature map; The multiplication result is processed by at least one convolution, batch normalization and nonlinear activation function to generate a fusion feature matrix.

2. The method according to claim 1, characterized in that Generate training samples including historical digestive tract images and historical patient characteristics, including: Acquire historical digestive tract images and historical patient characteristic information corresponding to the historical digestive tract images; annotating lesions in the historical digestive tract images to generate lesion samples and non-lesion samples; Classifying and quantifying the historical patient characteristic information according to the attribute information as an attribute information vector; The lesion samples, the non-lesion samples and the attribute information vector are combined as training samples according to the correspondence between the historical digestive tract images and the historical patient characteristic information.

3. According to the method of claim 1, the identification feature information includes lesion location information and lesion attribute information.

4. The method according to claim 1 or 3, characterized in that: The step of determining the target area of ​​the original upper digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image includes: Searching for a target area corresponding to the lesion location information in the identification feature information in the original upper digestive tract image; The target area is marked significantly and the lesion attribute information in the marked feature information is displayed on the original upper gastrointestinal tract image to generate an upper gastrointestinal endoscopic image.

5. The method according to any one of claims 1 to 3, characterized in that: Also includes: The original upper digestive tract image is subjected to at least one preprocessing operation of brightness adjustment, rotation, and translation.

6. An upper digestive tract endoscopy image generation device, characterized in that: The device comprises: A data acquisition module, used to acquire the original upper digestive tract image of the patient and the characteristic information of the patient; A model processing module, used to determine identification feature information according to an identification feature model, the original upper digestive tract image and the feature information of the patient, wherein the identification feature model is generated by training a training sample including historical digestive tract images and historical patient feature information; An image generation module, used for determining the target area of ​​the original upper digestive tract image according to the identification feature information to generate an upper digestive tract endoscopic image, including: marking the target area in the original upper digestive tract image by using at least one of a frame, a color change, and a background change; The model processing module includes: A convolution feature extraction branch, used for processing the original upper digestive tract image to generate a convolution feature map; A fully connected feature extraction branch, used to process the patient feature information and generate a feature vector; A feature fusion branch, used for processing the feature vector and the convolution feature map to generate a fused feature matrix; A feature decoding branch is used to process the fused feature matrix to generate identification feature information; Wherein, the identification feature information of the fused feature matrix is ​​obtained by combining the convolution feature map and the feature vector; The feature fusion branch is specifically used to: use the constituent elements of the feature vector as weight values, determine the multiplication result of the weight value and the corresponding channel of the convolution feature map; and generate a fusion feature matrix after processing the multiplication result through at least one convolution, batch normalization and nonlinear activation function.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the upper gastrointestinal endoscopic image generation method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating an upper gastrointestinal endoscopic image as described in any one of claims 1 to 5 is implemented.

Images