Method and device for recognizing part, image processing equipment and endoscope system

Abstract

The invention is suitable for the technical field of image processing, and provides a part recognition method and device, image processing equipment and an endoscope system.The method comprises the steps that a to-be-recognized image collected by the digestive tract through an endoscope is obtained; inputting the to-be-recognized image into the part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the to-be-recognized image; if it is determined that multiple alimentary canal parts exist in the to-be-recognized image according to the initial part recognition result, alimentary canal part segmentation processing is conducted on the to-be-recognized image based on the attention heat map, and multiple segmented target images are obtained; and performing part identification on each target image, and determining a target part corresponding to each target image. Compared with the prior art, the method can accurately judge whether a plurality of alimentary canal parts exist in the collected to-be-recognized image or not, and when it is determined that the plurality of alimentary canal parts exist, the plurality of parts are accurately recognized, and the recognition accuracy of the plurality of parts is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a part recognition method, device, image processing equipment and endoscope system. Background Art

[0002] An endoscope system is a detection instrument that has an endoscope, image processing equipment, light source equipment, and display equipment. It can enter the stomach through the mouth or enter the body through other natural orifices to detect the condition of the patient's digestive tract. At present, when performing endoscopic examinations, in order to effectively reduce the missed detection rate of endoscopic examinations and improve the quality of endoscopic examinations, it is necessary to perform digestive tract site (hereinafter referred to as site) recognition processing on endoscopic images to determine the inspected and uninspected sites. However, in actual endoscopic examinations, the field of view of the endoscope lens is not completely standard, and the same endoscopic image may contain multiple sites at the same time. Current related technologies cannot accurately identify in this case.

[0003] It can be seen that the current related technologies often have low recognition accuracy for multiple parts in endoscopic images and are difficult to meet actual needs. Summary of the invention

[0004] The embodiments of the present application provide a part recognition method, apparatus, image processing equipment and endoscope system, which can accurately determine whether there are multiple digestive tract parts in the collected image to be recognized, and when it is determined that there are multiple digestive tract parts, the multiple parts can be accurately recognized, thereby improving the recognition accuracy of multiple parts.

[0005] In a first aspect, an embodiment of the present application provides a part recognition method, comprising:

[0006] Acquire the image to be identified collected from the digestive tract through an endoscope;

[0007] The image to be identified is input into the part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to identify different digestive tract parts of the digestive tract;

[0008] If it is determined according to the initial part recognition result that there are multiple digestive tract parts in the image to be recognized, the digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain multiple target images after segmentation;

[0009] Part recognition is performed on each target image respectively to determine the target part corresponding to each target image.

[0010] The embodiment of the present application provides a method for part recognition, which obtains an image to be recognized collected from the digestive tract through an endoscope; inputs the image to be recognized into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be recognized; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to recognize different digestive tract parts of the digestive tract; if it is determined according to the initial part recognition result that there are multiple digestive tract parts in the image to be recognized, then the digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain multiple target images after segmentation; and part recognition is performed on each target image respectively to determine the target part corresponding to each target image. Compared with the prior art, the present method can accurately determine whether there are multiple digestive tract parts in the collected image to be recognized, and when it is determined that there are multiple digestive tract parts, the multiple parts can be accurately recognized, thereby improving the recognition accuracy of multiple parts.

[0011] In a possible implementation of the first aspect, the initial part recognition result includes multiple confidences, each confidence is uniquely associated with a digestive tract part; if it is determined according to the initial part recognition result that there are multiple digestive tract parts in the image to be recognized, then digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain multiple segmented target images, the method further includes:

[0012] When multiple confidence levels include multiple target confidence levels whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidence levels belong to a preset digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified; wherein each digestive tract part included in a single preset digestive tract part combination is a physiologically adjacent part.

[0013] In the above implementation, by comparing the differences between multiple confidence levels and detecting whether the digestive tract parts associated with all target confidence levels belong to a preset digestive tract part combination, it is determined whether there are multiple digestive tract parts in the image to be identified, thereby improving the accuracy of judging whether there are multiple digestive tract parts in the image to be identified.

[0014] In an implementation of the first aspect, when the multiple confidences include multiple target confidences whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidences belong to a preset digestive tract part combination, determining that there are multiple digestive tract parts in the image to be identified includes:

[0015] Get the preset benchmark quantity;

[0016] Filter out the confidence level with the largest number of benchmarks from multiple confidence levels;

[0017] When the differences between the reference number of confidences are all less than the first threshold, the reference number of confidences screened out are all used as target confidences, and when it is detected that the digestive tract parts respectively associated with the reference number of confidences all belong to the digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified;

[0018] When the difference between the confidence levels of the benchmark number is greater than or equal to the first threshold, the benchmark number is adjusted, and the process returns to the step of selecting the confidence level with the largest benchmark number from multiple confidence levels and subsequent steps until the adjusted benchmark number reaches a preset upper or lower limit.

[0019] In the above implementation, the accuracy of determining whether there are multiple digestive tract parts in the image to be identified is further improved.

[0020] In an implementation of the first aspect, adjusting the reference quantity includes:

[0021] If the digestive tract is the stomach, the base quantity is adjusted in descending order until the adjusted base quantity reaches the lower limit; wherein the lower limit is 2;

[0022] If the digestive tract is not the stomach, the base quantity is adjusted in ascending order until the adjusted base quantity reaches the upper limit of the quantity.

[0023] In the above implementation, for different digestive tract parts, the image processing device can adjust the reference number according to different adjustment methods, thereby improving the adjustment flexibility and adjustment accuracy of the reference number.

[0024] In one implementation of the first aspect, digestive tract segmentation is performed on the image to be identified based on the attention heat map to obtain multiple segmented target images, including:

[0025] Based on the attention heat map, determine a target area where each of the multiple digestive tract parts is located in the image to be identified;

[0026] Determine the first centroid of each target area;

[0027] According to each first centroid, the image to be recognized is subjected to image segmentation processing to obtain target images corresponding to each of the multiple parts.

[0028] In the above implementation, the image to be identified is segmented using the first centroids of the target areas of the various digestive tract sites in the image to be identified, thereby improving the segmentation accuracy of the image to be identified and accurately obtaining the target images corresponding to the various sites.

[0029] In an implementation of the first aspect, determining a target area where each digestive tract part among a plurality of digestive tract parts is located in an image to be identified based on the attention heat map includes:

[0030] Grayscale the attention heat map to obtain a grayscale image;

[0031] According to the number of parts in the grayscale image, the attention heat map is binarized to obtain the same number of target regions as the number of parts.

[0032] In the above implementation, by graying and binarizing the attention heat map, the accuracy of determining each target area is improved.

[0033] In an implementation of the first aspect, image segmentation processing is performed on the image to be identified according to each first centroid to obtain target images corresponding to each of the multiple parts, including:

[0034] When the number of target areas is 2, the perpendicular midline of the line connecting the first centroids is determined as the cutting line for image segmentation processing of the image to be recognized;

[0035] The image to be identified is segmented according to the cutting lines to obtain target images corresponding to multiple parts.

[0036] In the above embodiment, when the number of target areas is 2, the image processing device can directly determine the perpendicular bisector of the lines connecting each first centroid as the cutting line, and perform image segmentation processing on the image to be identified based on the cutting line, thereby improving the processing efficiency and segmentation accuracy of the image segmentation of the image to be identified, so that different parts can be segmented into the same area as much as possible.

[0037] In an implementation of the first aspect, image segmentation processing is performed on the image to be identified according to each first centroid to obtain target images corresponding to each of the multiple parts, including:

[0038] When the number of target areas is greater than 2, each first centroid is connected according to a preset order to obtain a centroid figure;

[0039] Determine the second centroid of the centroid figure;

[0040] Determine the lines connecting the second centroid and the midpoint of each side of the centroid figure as a plurality of cutting lines for image segmentation processing of the image to be recognized;

[0041] The image to be identified is segmented according to the multiple cutting lines to obtain target images corresponding to the multiple parts.

[0042] In the above embodiment, when the number of target areas is greater than 2, the image processing device can determine the cutting line for dividing each target area based on the centroid of the centroid figure formed by each first centroid and the length of each side, thereby improving the cutting accuracy of each target area so that different parts can be divided into the same area as much as possible.

[0043] In one implementation of the first aspect, digestive tract segmentation is performed on the image to be identified based on the attention heat map to obtain multiple segmented target images, including:

[0044] The multiple segmented images after the segmentation process are respectively subjected to image filling process to obtain multiple target images of the same size as the image to be recognized.

[0045] In the above embodiment, image filling processing is performed on the multiple segmented images after segmentation processing so that the size of the target image is the same as the size of the image to be recognized, thereby improving the accuracy of part recognition of each target image.

[0046] In a second aspect, an embodiment of the present application provides a part recognition device, including:

[0047] A first acquisition unit is used to acquire an image to be identified collected from the digestive tract through an endoscope;

[0048] A first processing unit is used to input the image to be identified into a part recognition model for processing, and obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to identify different digestive tract parts of the digestive tract;

[0049] A second processing unit is configured to perform digestive tract segmentation processing on the image to be identified based on the attention heat map to obtain multiple target images after segmentation if it is determined according to the initial part recognition result that the image to be identified has multiple digestive tract parts;

[0050] The recognition unit is used to perform part recognition on each target image respectively to determine the target part corresponding to each target image.

[0051] In an implementation of the second aspect, the initial part recognition result includes a plurality of confidence levels, each confidence level is uniquely associated with a digestive tract part; and the part recognition device further includes:

[0052] The first determination unit is used to determine that there are multiple digestive tract parts in the image to be identified when multiple confidence levels include multiple target confidence levels whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidence levels belong to a preset digestive tract part combination; wherein each digestive tract part included in a single preset digestive tract part combination is a physiologically adjacent part.

[0053] In an implementation manner of the second aspect, the first determining unit specifically includes:

[0054] A second obtaining unit, used to obtain a preset reference quantity;

[0055] A screening unit, used for screening out a confidence degree with a maximum number of benchmarks from a plurality of confidence degrees;

[0056] a second determination unit, configured to, when the differences between the reference number of confidences are all less than the first threshold, use the reference number of confidences screened out as target confidences, and when it is detected that the digestive tract parts respectively associated with the reference number of confidences all belong to the digestive tract part combination, determine that there are multiple digestive tract parts in the image to be identified;

[0057] The first adjustment unit is used to adjust the baseline number when the difference between the baseline number of confidences is greater than or equal to a first threshold, and return to execute the step of screening out the baseline number of maximum confidences from multiple confidences and subsequent steps until the adjusted baseline number reaches a preset upper limit or lower limit.

[0058] In an implementation manner of the second aspect, the first adjustment unit specifically includes:

[0059] The second adjustment unit is used for adjusting the reference quantity in descending order if the digestive tract is the stomach, until the adjusted reference quantity reaches the lower limit of quantity; wherein the lower limit of quantity is 2;

[0060] The third adjusting unit is used to adjust the reference quantity in ascending order if the digestive tract is not the stomach, until the adjusted reference quantity reaches the upper limit of the quantity.

[0061] In an implementation of the second aspect, the second processing unit specifically includes:

[0062] A third determination unit is used to determine, based on the attention heat map, a target area where each digestive tract part among the multiple digestive tract parts is located in the image to be identified;

[0063] A fourth determining unit, used to determine a first centroid of each target area;

[0064] The third processing unit is used to perform image segmentation processing on the image to be recognized according to each first centroid to obtain target images corresponding to each of the multiple parts.

[0065] In an implementation manner of the second aspect, the third determining unit specifically includes:

[0066] A fourth processing unit, configured to grayscale the attention heat map to obtain a grayscale image;

[0067] The fifth processing unit is used to perform binarization processing on the attention heat map according to the number of parts in the grayscale image to obtain target areas with the same number as the number of parts.

[0068] In an implementation of the second aspect, the third processing unit specifically includes:

[0069] A fifth determining unit, configured to determine, when the number of target areas is 2, a perpendicular midline of a line connecting the first centroids as a cutting line for performing image segmentation processing on the image to be recognized;

[0070] The sixth processing unit is used to perform image segmentation processing on the image to be identified according to the cutting line to obtain target images corresponding to the multiple parts.

[0071] In an implementation of the second aspect, the third processing unit specifically includes:

[0072] A connection unit, used for connecting each first centroid according to a preset order to obtain a centroid graph when the number of target areas is greater than 2;

[0073] A sixth determining unit, used for determining a second centroid of the centroid figure;

[0074] A seventh determination unit, used for determining each connecting line between the second centroid and the midpoint of each side length of the centroid figure as a plurality of cutting lines for performing image segmentation processing on the image to be recognized;

[0075] The seventh processing unit is used to perform image segmentation processing on the image to be identified according to the multiple cutting lines to obtain target images corresponding to the multiple parts.

[0076] In an implementation of the second aspect, the second processing unit specifically includes:

[0077] The eighth processing unit is used to perform image filling processing on the multiple segmented images after the segmentation processing to obtain multiple target images with the same size as the image to be recognized.

[0078] In a third aspect, an embodiment of the present application provides an image processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the part recognition method as described in any one of the first aspects above is implemented.

[0079] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the part recognition method as described in any one of the above-mentioned first aspects.

[0080] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an image processing device, the image processing device can execute the part recognition method described in any one of the first aspects above.

[0081] In a sixth aspect, an embodiment of the present application provides an endoscope system, comprising: an endoscope, a display device, and an image processing device as described in the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0083] Figure 1 is a schematic structural diagram of an endoscope system provided in one embodiment of the present application;

[0084] Figure 2 is a flowchart of an implementation method of a part recognition method provided in an embodiment of the present application;

[0085] Figure 3 is a flow chart of an implementation of a part recognition method provided by another embodiment of the present application;

[0086] Figure 4 is a flow chart of an implementation of a part recognition method provided in yet another embodiment of the present application;

[0087] Figure 5 is a flow chart of an implementation of a part recognition method provided by another embodiment of the present application;

[0088] Figure 6 is a flow chart of an implementation of a part recognition method provided by another embodiment of the present application;

[0089] Figure 7 is a flow chart of an implementation of a part recognition method provided by another embodiment of the present application;

[0090] Figure 8 is a schematic diagram of a cutting line provided by an embodiment of the present application;

[0091] Fig. 9 is a schematic diagram of a cutting line provided by another embodiment of the present application;

[0092] Fig.10 is a structural schematic diagram of a part identification device provided in one embodiment of the present application;

[0093] Fig.11It is a structural schematic diagram of an image processing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0094] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0095] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0096] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0097] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0098] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0099] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0100] In practical applications, an endoscope system is a detection instrument with an endoscope, an image processing device, a light source device, a display device, and other devices. It can enter the stomach through the mouth or enter the body through other natural orifices to detect the digestive tract condition of the patient. At present, when performing endoscopic examinations, in order to effectively reduce the missed detection rate of endoscopic examinations and improve the quality of endoscopic examinations, it is necessary to perform digestive tract site (hereinafter referred to as site) recognition processing on endoscopic images to determine the inspected sites and uninspected sites. However, in actual endoscopic examinations, the lens field of the endoscope is not completely standard, and the same endoscopic image may contain multiple sites at the same time. The current related technologies cannot accurately identify in this case. Specifically, the current related technologies will only identify the results of one site, so when there are multiple sites at the same time, it will only be identified as the site with the highest confidence. It can be seen that the current related technologies are inaccurate, unstable, and incomplete in the recognition results of multiple sites, resulting in reduced credibility of the recognition results.

[0101] Therefore, in order to improve the accuracy of identifying multiple parts, that is, to accurately determine whether there are multiple digestive tract parts in the collected image to be identified, and when it is determined that there are multiple digestive tract parts, the multiple parts can be accurately identified, in all the embodiments of the present application, the image processing device in the endoscope system can specifically realize accurate identification of multiple parts through the various method embodiments corresponding to the following figures, which will not be elaborated here.

[0102] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the structure of an endoscope system provided in one embodiment of the present application. Figure 1 As shown, the endoscope system 1 provided in the embodiment of the present application includes: an endoscope 10, an image processing device 20 and a display device 30. The endoscope 10 is connected to the image processing device 20 for communication, and the image processing device 20 is connected to the display device 30 for communication.

[0103] It should be noted that, in some possible embodiments, the image processing device 20 includes a light source device (not shown in the figure). The light source device is used to provide various lights to the endoscope 10.

[0104] In some other embodiments, the endoscope system 1 may further include a light source device (not shown in the figure). In this case, the light source device and the image processing device 20 are not the same device.

[0105] The image processing device 20 is used to perform part recognition on the to-be-recognized image of the digestive tract collected by the endoscope 10 in real time during the process of the endoscope 10 examining the digestive tract of the user, so as to determine the target part corresponding to the to-be-recognized image.

[0106] How the image processing device 20 specifically performs part recognition on the image to be recognized can be found in the description of the part recognition method shown in the following figure, and will not be elaborated here.

[0107] The display device 30 is used to display an endoscopic image of the user's digestive tract in real time while the endoscope 10 is examining the user's digestive tract.

[0108] In one embodiment of the present application, the display device 30 is further configured to receive the part recognition result sent by the image processing device 20 and display the part recognition result.

[0109] See also Figure 2 , Figure 2 This is a flowchart of a method for realizing part recognition provided by an embodiment of the present application. In the embodiment of the present application, the execution subject of the method for part recognition is an image processing device.

[0110] like Figure 2 As shown, the part recognition method provided in an embodiment of the present application may include S101 to S104, which are described in detail as follows:

[0111] In S101, an image to be identified is acquired from the digestive tract through an endoscope.

[0112] In actual applications, when an operator uses an endoscope system to inspect the user's digestive tract, in order to accurately understand the digestive tract site in the digestive tract image collected by the endoscope, the operator can trigger a site recognition instruction for the image processing device.

[0113] In an embodiment of the present application, the image processing device detecting the part recognition instruction may include: detecting that the operator performs a preset operation. The preset operation may be determined according to actual needs and is not limited here. Exemplarily, the preset operation may include: a preset control in the image processing device is triggered, or an endoscope is detected to enter the user's digestive tract. Based on this, when the image processing device detects that its own preset control is triggered, or detects that the endoscope enters the user's digestive tract, it indicates that a preset operation has been detected, that is, a part recognition instruction has been detected.

[0114] In the embodiment of the present application, after detecting the part recognition instruction, the image processing device can obtain the image to be recognized that is collected from the digestive tract through the endoscope.

[0115] In S102, the image to be identified is input into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to identify different digestive tract parts of the digestive tract.

[0116] In an embodiment of the present application, after obtaining an image to be identified, the image processing device may input the image to be identified into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified. The part recognition model is a neural network model pre-trained for the digestive tract part, and is used to identify different digestive tract parts of the digestive tract.

[0117] It should be noted that the attention heat map is used to represent the focus of the part recognition model. The darker the area in the attention heat map, the more attention the part recognition model pays to the area. The darker the area, the area where the digestive tract part exists in the image to be recognized.

[0118] In some possible embodiments, the initial site recognition result includes multiple confidence levels, and each confidence level is uniquely associated with a digestive tract site.

[0119] Based on this, in this embodiment, the part recognition model is specifically used to detect the confidence associated with each identifiable digestive tract part in the endoscopic image. Among them, the identifiable digestive tract parts can be set according to actual needs, and there is no limitation here. For example, when the part recognition model is applied to stomach recognition, the identifiable digestive tract parts include but are not limited to: the anterior wall, lesser curvature, posterior wall and greater curvature of the gastric body; the anterior wall, lesser curvature, posterior wall and greater curvature of the gastric antrum; the anterior angle, center and posterior angle of the gastric angle, etc. The part recognition model can be obtained by training a pre-constructed neural network model based on a preset sample set. Among them, each sample data in the preset sample set includes a sample endoscopic image and the confidence associated with each identifiable digestive tract part corresponding to the sample endoscopic image. When training the pre-built neural network model, the sample endoscopic image in each sample is used as the input of the neural network model, and the confidence associated with each identifiable digestive tract part corresponding to the sample endoscopic image in each sample is used as the output of the neural network model. Through training, the neural network model can learn the correspondence between all possible endoscopic images and the confidence associated with each identifiable digestive tract part, and the trained neural network model is used as a part recognition model.

[0120] In one embodiment of the present application, in order to improve the recognition accuracy and calculation speed of the part recognition model, before training the above-mentioned neural network model, the image processing device may first perform image preprocessing on the acquired sample endoscopic images, and use the image preprocessed sample endoscopic images to train the neural network model.

[0121] In another embodiment of the present application, since in actual applications, endoscopic images usually have black borders that record some invalid information that is useless for part identification, such as shooting time, shooting location, and patient age, and the black border accounts for a large proportion of the entire endoscopic image, therefore, in order to further improve the calculation speed and recognition accuracy of the part recognition model while reducing computing memory, the image processing device can crop the above-mentioned sample endoscopic image before training the above-mentioned neural network model, crop out all black borders in the sample endoscopic image, and use the cropped sample endoscopic image to train the neural network model.

[0122] In the embodiment of the present application, after the image processing device obtains the initial part recognition result, since the initial part recognition result includes multiple confidence levels, and each confidence level is uniquely associated with a digestive tract part, the image processing device can specifically perform the following steps: Figure 3 Step S201 shown is to detect whether there are multiple digestive tract parts in the image to be identified, as detailed below:

[0123] In S201, when multiple confidence levels include multiple target confidence levels whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidence levels belong to a preset digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified; wherein each digestive tract part included in the digestive tract part combination is a physiologically adjacent part.

[0124] In this embodiment, after obtaining the initial part recognition result, the image processing device can traverse and calculate the difference between each confidence level in the multiple confidence levels included in the initial part recognition result, and compare each difference level with the first threshold in turn. The first threshold can be set according to actual needs and is not limited here. For example, the first threshold can be 0.15.

[0125] In one embodiment of the present application, when the image processing device detects that each of the above differences is greater than or equal to the first threshold, it indicates that the differences between the confidences are large. Therefore, in order to improve the recognition accuracy of the part in the image to be recognized, the image processing device can determine the maximum confidence among the multiple confidences included in the initial part recognition result, and compare the maximum confidence with the second threshold. The second threshold can be set according to actual needs and is not limited here. Exemplarily, the second threshold can be set to 0.75.

[0126] In one implementation of this embodiment, when the image processing device detects that the above-mentioned maximum confidence is greater than or equal to the second threshold, it indicates that the initial part recognition result is credible. Therefore, the image processing device can determine that the digestive tract part in the image to be recognized is the digestive tract part associated with the maximum confidence.

[0127] In another implementation of this embodiment, when the image processing device detects that the above-mentioned maximum confidence level is less than the second threshold value, it indicates that the initial part recognition result is unreliable. Therefore, the image processing device can re-recognize the part of the image to be recognized, that is, re-input the image to be recognized into the part recognition model for further processing.

[0128] In another embodiment of the present application, when the image processing device detects that there is a difference less than the first threshold among the above-mentioned differences, it indicates that the differences between the multiple target confidences associated with the difference less than the first threshold are small, that is, there may be multiple digestive tract parts in the image to be identified, so the image processing device can detect whether the digestive tract parts associated with the above-mentioned multiple target confidences all belong to a preset digestive tract part combination. Among them, the digestive tract parts contained in a single preset digestive tract part combination are all physiologically adjacent parts.

[0129] For example, taking the digestive tract as the stomach, the digestive tract part combination includes but is not limited to: a combination of the anterior wall and the lesser curvature of the stomach body, a combination of the anterior angle and the center of the stomach angle, and the like.

[0130] In one implementation of this embodiment, when the image processing device detects that the digestive tract parts associated with the above-mentioned multiple target confidences all belong to a preset digestive tract part combination, the image processing device can determine that there are indeed multiple digestive tract parts in the image to be identified, and the number of digestive tract parts existing in the image to be identified can be determined according to the number of target confidences.

[0131] In another implementation of this embodiment, when the image processing device detects that the digestive tract parts associated with the above-mentioned multiple target confidences do not completely belong to the preset digestive tract part combination, the image processing device can determine that there are no multiple digestive tract parts in the image to be identified.

[0132] In another embodiment of the present application, the image processing device may also Figure 4 Steps S301 to S304 shown execute step S201, which are described in detail as follows:

[0133] In S301, a preset reference quantity is obtained.

[0134] In this embodiment, the preset reference number can be set according to actual needs and is not limited here. Exemplarily, the preset reference number can be any integer greater than or equal to 2 and less than or equal to 4, such as 2 or 4.

[0135] In S302, a confidence level with a maximum reference number is selected from a plurality of confidence levels.

[0136] In this embodiment, after obtaining the above-mentioned reference number, the image processing device can sort the multiple confidences included in the initial part recognition result according to a preset order, and select the confidence with the largest reference number from them. The preset order can be determined according to actual needs and is not limited here.

[0137] In some possible embodiments, the preset order may be from large to small, and therefore, the image processing device may directly determine the confidences of the first reference number as the confidences of the largest reference number.

[0138] In some other possible embodiments, the preset order may be from small to large, and therefore, the image processing device may directly determine the confidence of the latter reference number as the confidence of the largest reference number.

[0139] In this embodiment, after obtaining the reference number of maximum confidences, the image processing device may traverse and calculate the difference between each confidence among the reference number of maximum confidences, and compare each difference with the first threshold in turn.

[0140] In one embodiment of the present application, when the image processing device detects that the difference between each confidence level in the reference number of maximum confidence levels is less than a first threshold, step S303 may be executed.

[0141] In another embodiment of the present application, when the image processing device detects that the difference between the confidences in the reference number of maximum confidences is greater than or equal to the first threshold, the step S304 may be executed.

[0142] In S303, when the differences between the benchmark number of confidences are all less than the first threshold, the screened benchmark number of confidences are all used as target confidences, and when it is detected that the digestive tract parts associated with the benchmark number of confidences all belong to the digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified.

[0143] In this embodiment, when the image processing device detects that the difference between each confidence level in the maximum confidence level of the reference number is less than the first threshold, it indicates that the difference between the reference confidence levels screened is small, so the image processing device can determine the reference confidence levels as the target confidence levels. At this time, the image processing device can determine that there may be multiple digestive tract parts in the image to be identified. Therefore, in order to accurately determine whether there are multiple digestive tract parts in the image to be identified, the image processing device can detect whether the digestive tract parts associated with the multiple target confidence levels all belong to the preset digestive tract part combination.

[0144] In one implementation of this embodiment, when the image processing device detects that the digestive tract parts associated with the above-mentioned multiple target confidences all belong to a preset digestive tract part combination, the image processing device can determine that there are indeed multiple digestive tract parts in the image to be identified, and the number of parts existing in the image to be identified can be determined according to the number of target confidences.

[0145] In another implementation of this embodiment, when the image processing device detects that the digestive tract parts associated with the above-mentioned multiple target confidences do not completely belong to the preset digestive tract part combination, the image processing device can determine that there are no multiple digestive tract parts in the image to be identified.

[0146] In S304, when the difference between the confidence levels of the benchmark number is greater than or equal to the first threshold, the benchmark number is adjusted, and the process returns to the step of selecting the confidence level with the largest benchmark number from multiple confidence levels and subsequent steps until the adjusted benchmark number reaches a preset upper limit or lower limit.

[0147] In this embodiment, when the image processing device detects that the difference between each confidence level in the maximum confidence levels of the baseline number is greater than or equal to the first threshold, in order to further improve the accuracy of judging whether the image to be identified contains multiple parts, the image processing device may adjust the baseline number and return to execute the above steps S302 to S304 until the adjusted baseline number reaches the preset upper limit or lower limit.

[0148] It should be noted that the upper limit and the lower limit of the quantity can be set according to actual needs and are not limited here. For example, the upper limit of the quantity can be 4 and the lower limit of the quantity can be 2.

[0149] In some possible embodiments, the image processing device may randomly adjust the reference quantity.

[0150] In some other possible embodiments, in order to improve the accuracy of adjusting the reference number and improve the efficiency of judging whether the image to be identified contains multiple parts, the image processing device may adjust the reference number according to the following steps, which are described in detail as follows:

[0151] If the digestive tract is the stomach, the base quantity is adjusted in descending order until the adjusted base quantity reaches the lower limit; wherein the lower limit is 2;

[0152] If the digestive tract is not the stomach, the base quantity is adjusted in ascending order until the adjusted base quantity reaches the upper limit of the quantity.

[0153] In this embodiment, when the image processing device detects that the digestive tract is the stomach, since some parts of the stomach are distributed in three dimensions, multiple parts are likely to appear in one cross section. Therefore, in order to improve the efficiency of judging whether the image to be identified contains multiple parts, the image processing device can adjust the benchmark number in descending order until the adjusted benchmark number is the lower limit.

[0154] It should be noted that, when the image processing device detects that the digestive tract is the stomach, the image processing device may also adjust the reference number in order from small to large until the adjusted reference number reaches the upper limit.

[0155] In this embodiment, when the image processing device detects that the digestive tract is not the stomach, since the non-stomach part (such as the intestine) is usually linear, in order to improve the efficiency of judging whether the image to be identified contains multiple parts, the image processing device can adjust the benchmark number in order from small to large until the adjusted benchmark number reaches the upper limit.

[0156] In the embodiment of the present application, when the image processing device determines that there are multiple digestive tract parts in the image to be identified based on the initial part recognition result, steps S103 to S104 can be executed.

[0157] In S103, if it is determined according to the initial part recognition result that there are multiple digestive tract parts in the image to be recognized, the digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain multiple target images after segmentation.

[0158] In an embodiment of the present application, when the image processing device determines that there are multiple digestive tract parts in the image to be identified based on the initial part recognition result, in order to improve the recognition accuracy of each digestive tract part existing in the image to be identified, the image processing device can perform digestive tract part segmentation processing on the image to be identified based on the attention heat map to obtain multiple target images after segmentation.

[0159] Since the attention heat map is used to characterize the focus of the part recognition model, the darker the area in the attention heat map, the more the part recognition model pays attention to the area. Therefore, in one embodiment of the present application, the image processing device can determine each area in the attention heat map whose color depth is greater than the set depth as the target area that needs to be segmented. After that, the image processing device can segment the image to be recognized based on the determined target areas to obtain multiple segmented target images. Each target image contains only one target area.

[0160] The depth can be set according to actual needs and is not limited here.

[0161] In another embodiment of the present application, the image processing device can also Figure 5 The steps S401 to S403 shown execute step S103, which is described in detail as follows:

[0162] In S401, based on the attention heat map, a target area where each digestive tract part among a plurality of digestive tract parts is located in the image to be identified is determined.

[0163] In one implementation of the present embodiment, since the attention heat map is used to characterize the focus of the part recognition model, the darker the color of the area in the attention heat map, the more attention the part recognition model pays to the area. Therefore, in the present embodiment, the image processing device can determine each area in the attention heat map whose color depth is greater than the set depth as the target area of ​​each of the multiple digestive tract parts in the image to be recognized.

[0164] In another implementation of this embodiment, the image processing device may also determine the target area where each digestive tract part among the multiple digestive tract parts is located in the image to be identified according to the following steps, which are described in detail as follows:

[0165] Grayscale the attention heat map to obtain a grayscale image;

[0166] According to the number of parts in the grayscale image, the attention heat map is binarized to obtain the same number of target regions as the number of parts.

[0167] In practical applications, compared with color images, grayscale images take up less memory and run faster. At the same time, grayscale images can visually increase contrast and highlight core areas. Therefore, in this embodiment, in order to improve work efficiency and improve the accuracy of determining each target area, the image processing device can grayscale the attention heat map to obtain a grayscale image corresponding to the attention heat map.

[0168] It should be noted that the grayscale processing methods include but are not limited to: maximum value method, average value method and weighted average method.

[0169] In this embodiment, after obtaining the grayscale image, the image processing device can determine the number of parts in the grayscale image based on the number of target confidence levels among the multiple confidence levels included in the initial part recognition results, where the difference is less than a first threshold, and all digestive tract parts associated with the confidence levels associated with the difference less than the first threshold belong to a preset digestive tract part combination.

[0170] Afterwards, the image processing device can binarize the attention heat map according to the number of parts in the grayscale image to obtain the same number of target areas as the number of parts.

[0171] It should be noted that the binarization process in this embodiment is an adaptive binarization algorithm, that is, the image processing device can adaptively determine the binarization threshold according to the local features in the attention heat map.

[0172] Specifically, the image processing device can divide the attention heat map into multiple regions according to the color depth of the attention heat map, and the image processing device can determine the binarization threshold of each region according to the different color depths of different regions. Based on this, the image processing device can obtain the same number of target regions as the number of parts in the attention heat map.

[0173] Afterwards, the image processing device can determine the position area in the image to be identified that corresponds to each target area in the attention heat map as the target area where each digestive tract part among the multiple digestive tract parts is located in the image to be identified.

[0174] In S402, the first centroid of each target area is determined.

[0175] In S403, the image to be recognized is segmented according to each first centroid to obtain target images corresponding to each of the multiple parts.

[0176] In this embodiment, after determining the target area where each digestive tract part among the multiple digestive tract parts is located in the image to be identified, the image processing device can determine the first centroid of each target area.

[0177] In practical applications, the centroid refers to the geometric center of a figure.

[0178] It should be noted that, since the image to be identified is a physical object with uniform density, the first centroid of each target region is the centroid of the image region where the target region is located.

[0179] In this embodiment, the image processing device can perform image segmentation processing on the image to be identified according to each first centroid, so as to obtain target images corresponding to each of the multiple parts.

[0180] Specifically, the image processing device can determine the cutting lines for each target area based on each first centroid and the corresponding target area. Thereafter, the image processing device can perform image segmentation processing on the image to be identified based on the determined cutting lines to obtain target images corresponding to multiple parts.

[0181] In one embodiment of the present application, in order to reduce image noise and improve the segmentation accuracy of the image to be identified, the image processing device can first segment and eliminate other areas in the image to be identified except the target area to obtain an image to be identified that only contains each target area. After that, the image processing device performs image segmentation processing on the image to be identified that only contains each target area according to the determined cutting lines to obtain target images corresponding to multiple parts.

[0182] In another embodiment of the present application, in order to ensure that the size of the target image is the same as the size of the image to be identified, so as to improve the accuracy of part recognition of each target image, the image processing device can perform image filling processing on the multiple segmented images after segmentation processing to obtain multiple target images of the same size as the image to be identified.

[0183] In some possible embodiments, due to different numbers of target areas, the methods for determining the cutting lines of the target areas are different. Therefore, when the image processing device detects that the number of target areas is equal to 2, the image processing device may perform the following steps: Figure 6 In steps S501 to S502 shown, target images corresponding to multiple parts are obtained.

[0184] In some other possible embodiments, when the image processing device detects that the number of target areas is greater than 2, the image processing device may Figure 7 Steps S601 to S604 shown obtain target images corresponding to multiple parts.

[0185] In S104, part recognition is performed on each target image to determine the target part corresponding to each target image.

[0186] In some possible embodiments, after obtaining the multiple segmented target images, the image processing device may sequentially input the multiple target images into a trained part recognition model for processing to obtain various target part recognition results corresponding to various target images.

[0187] For any target image, the corresponding target part recognition result contains multiple confidence levels, and each confidence level is uniquely associated with a digestive tract part. Therefore, the image processing device can determine the maximum confidence level among the multiple confidence levels contained in the target part recognition result, and compare the maximum confidence level with the second threshold.

[0188] When the image processing device detects that the maximum confidence is greater than or equal to the second threshold, it indicates that the target part recognition result is credible. Therefore, the image processing device may regard the digestive tract part associated with the maximum confidence as the target part corresponding to the target image.

[0189] Based on this, the image processing device can determine the target part corresponding to each target image.

[0190] In some other possible embodiments, the image processing device pre-stores a digestive tract image set, wherein the digestive tract part in each digestive tract image in the digestive tract image set is different.

[0191] Therefore, in this embodiment, for any target image, the image processing device can compare the target image with each digestive tract region image in the above-mentioned digestive tract region image set one by one to obtain the similarity between the target image and each digestive tract region image in the above-mentioned digestive tract region image set.

[0192] Afterwards, the image processing device may determine the digestive tract portion corresponding to the digestive tract portion image with the greatest similarity as the target portion corresponding to the target image.

[0193] From the above, it can be seen that a part recognition method provided by an embodiment of the present application is obtained by acquiring an image to be recognized collected from the digestive tract through an endoscope; the image to be recognized is input into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be recognized; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to recognize different digestive tract parts of the digestive tract; if it is determined according to the initial part recognition result that there are multiple digestive tract parts in the image to be recognized, then the digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain multiple target images after segmentation; each target image is separately recognized to determine the target part corresponding to each target image. Compared with the prior art, the present method can accurately determine whether there are multiple digestive tract parts in the collected image to be recognized, and when it is determined that there are multiple digestive tract parts, the multiple parts can be accurately recognized, thereby improving the recognition accuracy of multiple parts.

[0194] See also Figure 6 , Figure 6 is a flowchart of a method for realizing part recognition provided by another embodiment of the present application. Figure 5 Corresponding embodiment, in this embodiment, step S403 may specifically include steps S501 to S502, which are described in detail as follows:

[0195] In S501, when the number of target regions is 2, the perpendicular midline of the line connecting the first centroids is determined as the cutting line for performing image segmentation processing on the image to be recognized.

[0196] In S502, the image to be recognized is segmented according to the cutting lines to obtain target images corresponding to the multiple parts.

[0197] In this embodiment, when the image processing device determines that the number of target areas is 2, it can connect the first centroids of the two target areas to obtain a first connecting line and make a perpendicular midline of the first connecting line. Finally, the image processing device can directly determine the perpendicular midline as a cutting line for image segmentation processing of the image to be recognized.

[0198] For example, see Figure 8 , Figure 8 Schematic diagram of a cutting line provided by an embodiment of the present application. Figure 8 As shown, T is the image to be identified, D1 is a target area, D2 is another target area, point P1 is the first centroid corresponding to the target area D1, point P2 is the first centroid corresponding to the target area D2, L1 is the line between the first centroid P1 and the first centroid P2, and L2 is the perpendicular bisector of the line L1, that is, the cutting line of the image to be identified.

[0199] Based on this, the image processing device can perform image segmentation processing on the image to be identified according to the cutting line to obtain target images corresponding to the multiple parts.

[0200] From the above, it can be seen that in the part recognition method provided by this embodiment, when the number of target areas is 2, the image processing device can directly determine the perpendicular bisector of the line connecting each first centroid as the cutting line, and perform image segmentation processing on the image to be recognized based on the cutting line, thereby improving the processing efficiency and segmentation accuracy of the image segmentation of the image to be recognized, so that different parts can be segmented into the same area as much as possible.

[0201] See also Figure 7 , Figure 7 FIG. 1 is a flowchart of a method for realizing part recognition provided by another embodiment of the present application. Figure 5 Corresponding embodiment, in this embodiment, step S403 may specifically include steps S601 to S604, which are described in detail as follows:

[0202] In S601, when the number of target areas is greater than 2, each first centroid is connected according to a preset order to obtain a centroid graph.

[0203] In S602, a second centroid of the centroid graphic is determined.

[0204] In S603, the lines connecting the second centroid and the midpoint of each side of the centroid figure are determined as a plurality of cutting lines for performing image segmentation processing on the image to be recognized.

[0205] In S604, the image to be identified is segmented according to the multiple cutting lines to obtain target images corresponding to the multiple parts.

[0206] In this embodiment, when the image processing device determines that the number of target areas is greater than 2, each first centroid may be connected according to a preset order to obtain a centroid graph, and determine the second centroid of the centroid graph. The preset order may be set according to actual needs and is not limited here. For example, the preset order may be a counterclockwise order.

[0207] After obtaining the centroid figure and the second centroid, the graphics processing device can determine the midpoint of each side length in the centroid figure, and connect the second centroid to each midpoint to obtain multiple lines. After that, the image processing device can extend each line in one direction to obtain multiple cutting lines for image segmentation processing of the image to be recognized.

[0208] It should be noted that the direction in which each connecting line is extended in one direction is the direction in which the second centroid points to the midpoint of each side length in the centroid figure corresponding to each connecting line.

[0209] For example, see Fig. 9 , Fig. 9 Schematic diagram of a cutting line provided by another embodiment of the present application. Fig. 9 As shown, T is the image to be recognized, A is the target area 1, B is the target area 2, C is the target area 3, D is the target area 4, S is the centroid figure, point a is the first centroid corresponding to the target area 1, point b is the first centroid corresponding to the target area 2, point c is the first centroid corresponding to the target area 3, point d is the first centroid corresponding to the target area 4, point s is the second centroid corresponding to the centroid figure, point k1 is the first midpoint of the first side ab of the centroid figure, point k2 is the second midpoint of the second side bc of the centroid figure, and point k3 is the third side c of the centroid figure d is the third midpoint of the image, point k4 is the fourth midpoint of the fourth side da of the centroid figure, La is the extension of the line between the second centroid s and the first midpoint k1, that is, a cutting line of the image to be identified, Lb is the extension of the line between the second centroid s and the second midpoint k2, that is, a cutting line of the image to be identified, Lc is the extension of the line between the second centroid s and the third midpoint k3, that is, a cutting line of the image to be identified, and Ld is the extension of the line between the second centroid s and the fourth midpoint k4, that is, a cutting line of the image to be identified.

[0210] Based on this, the image processing device can perform image segmentation processing on the image to be identified according to the cutting line to obtain target images corresponding to the multiple parts.

[0211] From the above, it can be seen that in the part recognition method provided by this embodiment, when the number of target areas is greater than 2, the image processing device can determine the cutting line for dividing each target area based on the centroid of the centroid figure formed by each first centroid and the length of each side, thereby improving the cutting accuracy of each target area, so that different parts can be divided into the same area as much as possible.

[0212] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0213] Corresponding to a part recognition method described in the above embodiment, Fig.10 The structure diagram of a part recognition device provided by an embodiment of the present application is shown. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Fig.10 The part recognition device 800 includes: a first acquisition unit 81, a first processing unit 82, a second processing unit 83 and a recognition unit 84. Among them:

[0214] The first acquisition unit 81 is used to acquire the image to be identified which is collected from the digestive tract through the endoscope.

[0215] The first processing unit 82 is used to input the image to be identified into the part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for the digestive tract part, and is used to identify different digestive tract parts of the digestive tract.

[0216] The second processing unit 83 is used to perform digestive tract segmentation processing on the image to be identified based on the attention heat map to obtain multiple target images after segmentation if it is determined that there are multiple digestive tract parts in the image to be identified according to the initial part identification result.

[0217] The recognition unit 84 is used to perform part recognition on each target image respectively, and determine the target part corresponding to each target image.

[0218] In one embodiment of the present application, the initial part recognition result includes multiple confidence levels, each confidence level is uniquely associated with a digestive tract part; the part recognition device 800 further includes:

[0219] The first determination unit is used to determine that there are multiple digestive tract parts in the image to be identified when multiple confidence levels include multiple target confidence levels whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidence levels belong to a preset digestive tract part combination; wherein each digestive tract part included in a single preset digestive tract part combination is a physiologically adjacent part.

[0220] In one embodiment of the present application, the first determining unit specifically includes:

[0221] The second acquisition unit is used to acquire a preset reference quantity.

[0222] The screening unit is used to screen out the confidences with the largest number of reference confidences from the multiple confidences.

[0223] The second determination unit is used to use the screened benchmark number of confidence levels as target confidence levels when the differences between the benchmark number of confidence levels are all less than the first threshold value, and to determine that there are multiple digestive tract parts in the image to be identified when it is detected that the digestive tract parts associated with the benchmark number of confidence levels all belong to the digestive tract part combination.

[0224] The first adjustment unit is used to adjust the baseline number when the difference between the baseline number of confidences is greater than or equal to a first threshold, and return to execute the step of screening out the baseline number of maximum confidences from multiple confidences and subsequent steps until the adjusted baseline number reaches a preset upper limit or lower limit.

[0225] In one embodiment of the present application, the first adjustment unit specifically includes:

[0226] The second adjustment unit is used to adjust the reference quantity in order from large to small if the digestive tract is the stomach, until the adjusted reference quantity reaches the lower limit of the quantity; wherein the lower limit of the quantity is 2.

[0227] The third adjustment unit is used for adjusting the reference quantity in ascending order if the digestive tract is not the stomach, until the adjusted reference quantity reaches the upper limit of the quantity.

[0228] In one embodiment of the present application, the second processing unit 83 specifically includes:

[0229] The third determination unit is used to determine the target area where each digestive tract part among the multiple digestive tract parts is located in the image to be identified based on the attention heat map.

[0230] The fourth determining unit is used to determine the first centroid of each target area.

[0231] The third processing unit is used for performing image segmentation processing on the image to be recognized according to each first centroid, so as to obtain target images corresponding to each of the multiple parts.

[0232] In one embodiment of the present application, the third determining unit specifically includes:

[0233] The fourth processing unit is used to perform grayscale processing on the attention heat map to obtain a grayscale image.

[0234] The fifth processing unit is used to perform binarization processing on the attention heat map according to the number of parts in the grayscale image to obtain target areas with the same number as the number of parts.

[0235] In one embodiment of the present application, the third processing unit specifically includes:

[0236] The fifth determining unit is used for determining the perpendicular bisector of the lines connecting the first centroids as the cutting line for performing image segmentation processing on the image to be recognized when the number of the target areas is 2.

[0237] The sixth processing unit is used to perform image segmentation processing on the image to be identified according to the cutting line to obtain target images corresponding to the multiple parts.

[0238] In one embodiment of the present application, the third processing unit specifically includes:

[0239] The connection unit is used to connect each first centroid according to a preset order to obtain a centroid graph when the number of target areas is greater than 2.

[0240] The sixth determining unit is used to determine the second centroid of the centroid figure.

[0241] The seventh determination unit is used to determine the lines connecting the second centroid and the midpoint of each side of the centroid figure as a plurality of cutting lines for performing image segmentation processing on the image to be recognized.

[0242] The seventh processing unit is used to perform image segmentation processing on the image to be identified according to the multiple cutting lines to obtain target images corresponding to the multiple parts.

[0243] In one embodiment of the present application, the second processing unit 83 specifically includes:

[0244] The eighth processing unit is used to perform image filling processing on the multiple segmented images after the segmentation processing, so as to obtain multiple target images with the same size as the image to be recognized.

[0245] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0246] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0247] Fig.11 This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. Fig.11 As shown, the image processing device 9 of this embodiment includes: at least one processor 90 ( Fig.11 Only one is shown in the figure) a processor, a memory 91, and a computer program 92 stored in the memory 91 and executable on the at least one processor 90, wherein the processor 90 implements the steps in any of the above-mentioned embodiments of the method for identifying a part when executing the computer program 92.

[0248] The image processing device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will appreciate that Fig.11 It is only an example of the image processing device 9 and does not constitute a limitation on the image processing device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0249] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0250] In some embodiments, the memory 91 may be an internal storage unit of the image processing device 9, such as the memory of the image processing device 9. In other embodiments, the memory 91 may also be an external storage device of the image processing device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the image processing device 9. Further, the memory 91 may also include both the internal storage unit of the image processing device 9 and the external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0251] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0252] An embodiment of the present application provides a computer program product. When the computer program product runs on an image processing device, the image processing device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0253] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the image processing device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0254] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0255] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A part recognition method, characterized in that: The method comprises: Acquire the image to be identified collected from the digestive tract through an endoscope; Inputting the image to be identified into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for digestive tract parts and is used to identify different digestive tract parts of the digestive tract; If it is determined according to the initial part recognition result that the image to be recognized has multiple digestive tract parts, then the image to be recognized is segmented into digestive tract parts based on the attention heat map to obtain multiple target images after segmentation; Part recognition is performed on each of the target images to determine the target part corresponding to each of the target images.

2. The part recognition method according to claim 1, characterized in that: The initial part recognition result includes a plurality of confidences, each confidence uniquely associated with a digestive tract part; if it is determined according to the initial part recognition result that the image to be recognized has a plurality of digestive tract parts, the digestive tract part segmentation processing is performed on the image to be recognized based on the attention heat map to obtain a plurality of segmented target images, the method further includes: When the multiple confidence levels include multiple target confidence levels whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidence levels belong to a preset digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified; wherein, each digestive tract part included in a single preset digestive tract part combination is a physiologically adjacent part.

3. The part recognition method according to claim 2, characterized in that: When the multiple confidences include multiple target confidences whose difference is less than a first threshold, and the digestive tract parts associated with all the target confidences belong to a preset digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified, including: Get the preset benchmark quantity; Filtering out the confidence levels with the largest number of the benchmark numbers from the multiple confidence levels; When the differences between the reference number of confidences are all smaller than the first threshold, the reference number of confidences screened out are all used as the target confidences, and when it is detected that the digestive tract parts respectively associated with the reference number of confidences all belong to the digestive tract part combination, it is determined that there are multiple digestive tract parts in the image to be identified; When the difference between the confidence levels of the benchmark number is greater than or equal to the first threshold, the benchmark number is adjusted, and the process returns to the step of selecting the confidence level with the largest number among the multiple confidence levels and subsequent steps until the adjusted benchmark number reaches a preset upper or lower limit.

4. The part recognition method according to claim 3, characterized in that: The adjusting the benchmark quantity comprises: If the digestive tract is the stomach, the reference quantity is adjusted in descending order until the adjusted reference quantity reaches the lower limit of the quantity; wherein the lower limit of the quantity is 2; If the digestive tract is not the stomach, the reference quantity is adjusted in order from small to large until the adjusted reference quantity reaches the upper limit of the quantity.

5. The part recognition method according to any one of claims 1 to 4, characterized in that: The digestive tract segmentation process is performed on the image to be identified based on the attention heat map to obtain a plurality of segmented target images, including: Based on the attention heat map, determining a target area where each of the multiple digestive tract parts is located in the image to be identified; Determining a first centroid of each of the target areas; The image to be identified is subjected to image segmentation processing according to each of the first centroids to obtain target images corresponding to each of the plurality of parts.

6. The part recognition method according to claim 5, characterized in that: The step of performing image segmentation processing on the image to be identified according to each of the first centroids to obtain target images corresponding to each of the plurality of parts includes: When the number of the target areas is 2, the perpendicular midline of the line connecting the first centroids is determined as a cutting line for performing image segmentation processing on the image to be identified; The image to be identified is segmented according to the cutting line to obtain target images corresponding to the multiple parts.

7. The part recognition method according to claim 5, characterized in that: The step of performing image segmentation processing on the image to be identified according to each of the first centroids to obtain target images corresponding to each of the plurality of parts includes: When the number of the target areas is greater than 2, connecting the first centroids according to a preset order to obtain a centroid graph; Determining a second centroid of the centroid figure; Determine each connecting line between the second centroid and the midpoint of each side length of the centroid figure as a plurality of cutting lines for performing image segmentation processing on the image to be recognized; The image to be identified is subjected to image segmentation processing according to the multiple cutting lines to obtain target images corresponding to the multiple parts respectively.

8. The part recognition method according to any one of claims 1 to 4, characterized in that: The digestive tract segmentation process is performed on the image to be identified based on the attention heat map to obtain a plurality of segmented target images, including: The multiple segmented images after the segmentation process are respectively subjected to image filling process to obtain multiple target images of the same size as the image to be identified.

9. A part recognition device, characterized in that: include A first acquisition unit is used to acquire an image to be identified collected from the digestive tract through an endoscope; A first processing unit is used to input the image to be identified into a part recognition model for processing to obtain an initial part recognition result and an attention heat map associated with the image to be identified; the part recognition model is a neural network model pre-trained for digestive tract parts, and is used to identify different digestive tract parts of the digestive tract; A second processing unit is configured to perform digestive tract segmentation processing on the image to be identified based on the attention heat map to obtain multiple target images after segmentation if it is determined that the image to be identified has multiple digestive tract parts according to the initial part identification result; The recognition unit is used to perform part recognition on each of the target images respectively to determine the target part corresponding to each of the target images.

10. An image processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the part recognition method according to any one of claims 1 to 8 is implemented.

11. An endoscope system, characterized in that: include: An endoscope, a display device, and an image processing device as claimed in claim 10.

Images