Image processing method and device, electronic equipment, and storage medium

Abstract

Embodiments of the present application provide an image processing method and device, electronic equipment and storage medium. The image processing method comprises: acquiring a target image; performing feature recognition on the target image to obtain wrinkle feature information corresponding to the target image; measuring height information of each wrinkle and spacing information of adjacent two wrinkles based on the wrinkle feature information; and identifying a stomach filling state in the target image by comparing the height information and the spacing information. The technical solution of the embodiments of the present application can automatically and in real time determine the stomach filling condition by image processing on the collected image, using the size of the wrinkle in the stomach and the spacing of adjacent wrinkles observed by the image, thereby effectively improving the detection efficiency and integrity in the stomach examination process.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer application, and in particular, to an image processing method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] An endoscope is inserted into the stomach through the oral cavity or other natural orifices, and can observe lesions that X-rays cannot display. Therefore, during the examination of the stomach, the endoscope can be used to observe ulcers or tumors in the stomach, such as by using a magnetic capsule endoscope. Currently, when a capsule endoscope is used to detect the stomach, the stomach of the person being detected needs to be in a full state. The current method usually requires the person being detected to drink a sufficient amount of water to expand the stomach to prevent folds from blocking and causing lesions to be missed. If the amount of water consumed is insufficient, the stomach cannot be fully expanded, which will result in incomplete detection by the capsule. If water is continuously introduced into the stomach after the stomach is fully expanded, the person being detected will feel uncomfortable due to excessive water consumption. Therefore, during the detection process, accurately determining whether the stomach is fully expanded is crucial to the completeness and comfort of the detection. However, in the current stomach examination process, whether the stomach is fully expanded is mainly determined by manual observation. This method not only requires high detection ability, but also is inefficient and ignores the comfort of the person being detected. SUMMARY

[0003] To solve the above technical problems, embodiments of the present application provide an image processing method and device, an electronic device, and a computer readable storage medium. Embodiments of the present application at least partially automatically determine whether the stomach is fully expanded, ensuring the completeness and comfort of the detection, reducing the requirement for the detection ability, and improving the efficiency.

[0004] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0005] According to an aspect of an embodiment of the present application, an image processing method is provided, including: acquiring a target image, the target image including a stomach image; performing feature recognition on the target image to obtain wrinkle feature information corresponding to the target image; measuring height information of each wrinkle and spacing information of adjacent two wrinkles based on the wrinkle feature information; and comparing the height information and the spacing information to identify a stomach filling condition in the target image.

[0006] According to an aspect of the embodiments of the present application, an image processing apparatus is provided, comprising: an image acquisition module configured to acquire a target image, the target image comprising a stomach image; a feature recognition module configured to perform feature recognition on the target image to obtain wrinkle feature information corresponding to the target image; a measurement module configured to measure height information of each wrinkle and spacing information of adjacent two wrinkles based on the wrinkle feature information; and a filling recognition module configured to compare the height information and the spacing information to recognize a stomach filling condition in the target image.

[0007] In the technical solutions provided by the embodiments of the present application, the feature recognition module comprises: an image segmentation unit configured to perform image segmentation on the target image to obtain a plurality of feature images; and a contour extraction unit configured to perform contour extraction on the feature images according to optical manifestations of the wrinkles and image features to measure the height information of each wrinkle and the spacing information of adjacent two wrinkles.

[0008] In the technical solutions provided by the embodiments of the present application, the measurement module is configured to: select a plurality of measurement points in the target image; calculate a ratio of the spacing information and the height information of the wrinkles in each measurement point to obtain a filling parameter; compare the filling parameter with a preset first threshold value, and count measurement points with a filling parameter greater than the first threshold value among all the measurement points to obtain a quantity statistical value; and the filling recognition module is configured to: compare the quantity statistical value with a preset second threshold value; when the quantity statistical value is greater than the second threshold value, it is determined that the stomach is in a filling state; and when the quantity statistical value is less than or equal to the second threshold value, it is determined that the stomach is in a non-filling state.

[0009] In the technical solutions provided by the embodiments of the present application, the filling recognition module comprises: a first processing unit configured to, when the stomach is in a filling state, determine whether the detection object corresponding to the target image needs water replenishment according to the quantity statistical value; and

[0010] comparing the quantity statistical value with a third threshold value to determine whether the detection object corresponding to the target image needs water replenishment, the third threshold value being greater than the second threshold value; if the quantity statistical value is greater than the third threshold value, it is determined that water replenishment is not needed; if the data statistical value is greater than the second threshold value and less than or equal to the third threshold value, it is determined that water replenishment is needed; and when it is determined that water replenishment is needed, a water replenishment amount that causes the stomach to be filled is acquired.

[0011] In the technical solutions provided by the embodiments of the present application, the filling recognition module further comprises: a second processing unit configured to, when the stomach is in a non-filling state, determine a water replenishment amount of the detection object corresponding to the target image according to the data statistical value; and

[0012] The quantity statistical value is compared with a fourth threshold value to determine the water supplement amount of the detection object corresponding to the target image, the fourth threshold value being smaller than the second threshold value; if the data statistical value is greater than the fourth threshold value, the water supplement amount for filling the stomach is obtained according to the average filling parameter; if the data statistical value is smaller than or equal to the fourth threshold value, the water supplement amount for filling the stomach is determined according to a preset water supplement amount range.

[0013] In the technical scheme provided in the embodiments of the present application, the filling identification module further comprises a first water supplement amount recommendation unit configured to obtain a mapping relationship between the filling parameter and the water supplement amount, the mapping relationship being determined based on the influence relationship of the water supplement amount on the stomach filling state, and the water supplement amount for filling the stomach in the target image is obtained according to the mapping relationship and the average filling parameter of the measurement point determined as needing water supplement.

[0014] In the technical scheme provided in the embodiments of the present application, the filling identification module further comprises a second water supplement amount recommendation unit, which comprises: a sample data subunit configured to collect sample image data of different stomach filling conditions; a labeling subunit configured to label the sample image data according to the water supplement amount level and the filling parameter to form a sample data set; a neural network model configured to establish a convolutional neural network model based on the sample data set and perform training; and an output subunit configured to input the filling parameter of the target image into the trained convolutional neural network model to obtain the water supplement amount for filling the stomach in the target image.

[0015] In the technical scheme provided in the embodiments of the present application, the contour extraction unit is configured to: obtain continuous frame target images, fuse the common wrinkle contours in the continuous frame target images to perform target tracking, and perform contour extraction on the fused wrinkle contours.

[0016] In the technical scheme provided in the embodiments of the present application, the contour extraction unit is configured to extract the wrinkle contour of a target image in one of the continuous frames and the wrinkle contour of the previous frame; perform corner point detection on the target image, and calculate the optical flow under multiple scales to obtain the displacement amount of the target wrinkle contour; and fuse the same wrinkle contour of adjacent two frames according to the displacement amount.

[0017] In the technical scheme provided in the embodiments of the present application, the image acquisition module comprises a capsule endoscope configured to acquire a stomach image, and a preprocessing unit configured to perform target identification on the acquired image to obtain a target image in which the included angle between the capsule endoscope and the stomach body is kept within a preset angle range.

[0018] In the technical scheme provided by the embodiment of the present application, the image acquisition module further comprises a detection unit configured to detect an included angle between the capsule gastroscope and the stomach body; and an adjusting mechanism configured to periodically adjust the position of the capsule gastroscope so as to keep the included angle between the capsule gastroscope and the stomach body within a preset angle range.

[0019] According to an aspect of the embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the image processing method as described above.

[0020] According to an aspect of the embodiment of the present application, a computer readable storage medium is provided, which stores computer readable instructions, when the computer readable instructions are executed by a processor of a computer, the computer executes the image processing method as described above.

[0021] In the technical scheme provided by the embodiment of the present application, by performing feature recognition on the obtained target image, corresponding wrinkle feature information is obtained, based on the wrinkle feature information, the height information of each wrinkle and the interval information of adjacent two wrinkles are measured respectively, and by comparing the height information and the interval information, the stomach filling state in the target image is recognized. It can be seen that the embodiment of the present application realizes automatic real-time judgment of the opening of the stomach, can replace manual inspection under the premise of guaranteeing the accuracy of the inspection, effectively improves the inspection efficiency, and based on the recognized stomach filling state, whether the detected person needs to supplement water can be determined more accurately, compared with the manual observation mode, the comfort of the detected person can be improved to a large extent.

[0022] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The drawings herein are incorporated into the specification and form part of the specification, show embodiments consistent with the present application, and together with the specification serve to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings:

[0024] Figure 1 is a schematic diagram of an exemplary system architecture shown by an exemplary embodiment of the present application;

[0025] Figure 2 is a flowchart of an image processing method shown by an exemplary embodiment of the present application;

[0026] Figure 3 is a flowchart of feature recognition shown in an example embodiment of the present application;

[0027] Figure 4 is a flowchart of gastric filling state recognition shown in an example embodiment of the present application;

[0028] Figure 5 is a flowchart of water replenishment determination shown in an example embodiment of the present application;

[0029] Figure 6 is a flowchart of water replenishment determination shown in an example embodiment of the present application;

[0030] Figure 7 is a flowchart of another feedback water replenishment amount method shown in an example embodiment of the present application;

[0031] Figure 8 is a flowchart of contour extraction shown in an example embodiment of the present application;

[0032] Figure 9 is a schematic diagram of a capsule endoscope and a gastric wall observation angle shown in an example embodiment of the present application;

[0033] Figure 10 is a schematic diagram of a fold measurement shown in an example embodiment of the present application;

[0034] Figure 11 is a schematic diagram of a multi-point fold measurement of an image I1 shown in an example embodiment of the present application;

[0035] Figure 12 is a schematic diagram of a multi-point fold measurement of an image I2 shown in an example embodiment of the present application;

[0036] Figure 13 is a schematic diagram before fold feature recognition shown in an example embodiment of the present application;

[0037] Figure 14 is a schematic diagram after fold feature recognition shown in an example embodiment of the present application;

[0038] Figure 15 shows a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application;

[0039] Figure 16 is a block diagram of an image processing method shown in an example embodiment of the present application;

[0040] Figure 17 shows a block diagram of an image processing apparatus suitable for implementing embodiments of the present application;

[0041] Figure 18 An exemplary measurement point detection numerical graph is shown;

[0042] Figure 19 Another exemplary measurement point detection numerical graph is shown;

[0043] Figure 20 Still another exemplary measurement point detection numerical graph is shown. DETAILED DESCRIPTION

[0044] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements throughout the description. The following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0045] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed depending on the actual situation.

[0047] In the present application, "a plurality of" means two or more. The "and / or" describes the association between the associated objects, indicating that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0048] First of all, it needs to be explained that the mucosal folds are the normal gastric mucosa forming many folds, called folds. The folds are tortuous and parallel to each other, from the upper part to the lower part, and become thin and disappear when reaching the antrum. The folds are most obvious at the greater curvature, and there are fewer folds on the anterior and posterior walls. At the lesser curvature, few folds can be seen. The folds at the fundus are thick and show obvious gyrus. After the stomach cavity is inflated, the mucosa expands. No longitudinal folds can be seen in the antrum. The characteristic of the folds is that there are "bumps" on the surface of the gastric body. Based on this, the use of feature extraction, AI recognition, etc. can determine whether there is fold information in the endoscope image.

[0049] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, tracking, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0050] An endoscope is a diagnostic instrument that integrates traditional optics, ergonomics, precision mechanics, modern electronics, mathematics, and software. It comprises an image sensor, optical lens, light source, and mechanical devices, and can be inserted into the stomach through the mouth or other natural orifices. Endoscopes can visualize lesions that are not visible to X-rays; for example, doctors can observe ulcers or tumors in the stomach with the aid of an endoscope. The embodiments in this application do not limit this scope.

[0051] The technical solutions of this application involve technologies such as machine learning and computer vision in artificial intelligence, and are specifically illustrated through the following embodiments:

[0052] Figure 1 This is a schematic diagram illustrating an exemplary system architecture as shown in an exemplary embodiment of this application.

[0053] Reference Figure 1 As shown, the system architecture may include an image acquisition device 101 and a computer device 102. The computer device 102 may be at least one of a desktop graphics processing unit (GPU) computer, a GPU computing cluster, or a neural network computer. Those skilled in the art can use the computer device 102 to process product images, identify target images for processing, and then measure the spacing and height of stomach folds to automatically determine the stomach's opening status. The image acquisition device 101 is used to acquire internal images of the stomach. In this embodiment, the image acquisition device 101 uses a capsule endoscope to acquire images and provides them to the computer device 102 for processing.

[0054] Illustratively, after the image acquisition device 101 acquires the target product image, the computer device 102 can perform feature recognition on the target image to obtain wrinkle feature information corresponding to the target image; based on the wrinkle feature information, the height information of each wrinkle and the spacing information of two adjacent wrinkles are measured respectively; and by comparing the height information and the spacing information, the stomach filling state in the target image is identified. It can be seen that the technical scheme of the embodiment of the present application can realize automatic real-time judgment of the stomach opening condition through image processing. The accuracy of the inspection result can be improved by avoiding human factor interference, and the inspection efficiency can also be effectively improved by replacing manual quality inspection.

[0055] It should be noted that the image processing method provided by the embodiment of the present application is generally executed by the computer device 102, and accordingly, the image processing apparatus is generally arranged in the computer device 102.

[0056] The implementation details of the technical scheme of the embodiment of the present application are described in detail as follows:

[0057] Figure 2 is a flowchart of an image processing method according to an example embodiment of the present application, which can be executed by a computing processing device, which can be the computer device 102 shown in Figure 1 . Referring to Figure 2 , the image processing method at least includes steps S210 to S240, which are described in detail as follows:

[0058] In step S210, a target image is acquired, and the target image includes a stomach image.

[0059] In an embodiment of the present application, the target image can be an image including wrinkle information obtained by processing a real-time acquired stomach image. The target image can also be acquired by the computing processing device 102 through other image acquisition devices connected thereto, and the target image can also be input into the computing processing device 102 by a relevant technical person.

[0060] In step S220, feature recognition is performed on the target image to obtain wrinkle feature information corresponding to the target image.

[0061] In an embodiment of the present application, Figure 13 and Figure 14The images before and after the wrinkle feature recognition are shown respectively. The image feature is the corresponding essential feature or characteristic of a certain class of objects that is different from other classes of objects, or a set of these features and characteristics. For an image, each image has its own features that can be distinguished from other classes of images, which can be extracted by measurement or processing, such as natural features that can be intuitively perceived, such as brightness, edge, texture, and color, etc.; or features that need to be obtained through transformation or processing, such as moments, histograms, and principal components, etc. In this embodiment, as shown in Figure 13 and Figure 14 The feature of the wrinkle is a "bulge" feature distributed on the surface of the body, so the corresponding wrinkle feature information can be obtained by performing feature recognition on the image with wrinkle information.

[0062] In an embodiment of the present application, the feature recognition on the image with wrinkle information can be performed according to the method flow in Figure 3 , and specifically refer to Figure 3 , which includes the following steps:

[0063] Step S310. The target image is segmented to obtain a plurality of feature images.

[0064] In an embodiment of the present application, the target image is divided into several specific regions with unique properties and the target of interest is proposed by image segmentation. Image segmentation is a key step from image processing to image analysis. From a mathematical point of view, image segmentation is a process of dividing a digital image into mutually disjoint regions. The subsequent wrinkle contour extraction is realized through image segmentation. Because different images have different gray scales, there are obvious edges at the boundaries, which can be used for image segmentation. It should be noted that the edge is not equal to the boundary between objects. The edge refers to the place where the value of the pixel in the image changes abruptly, while the boundary refers to the boundary between the bulge part of the wrinkle and other parts. The edge in the image is not the edge of the wrinkle in reality. The edge of the wrinkle in reality may not be reflected in the image. Because the object in the real world is three-dimensional, the target image in this embodiment only has two-dimensional information. The projection imaging from three-dimensional to two-dimensional inevitably loses part of the information. In this embodiment, the method of image segmentation can include threshold-based image segmentation, K-means clustering, histogram-based image segmentation, edge detection, etc.

[0065] In an embodiment, the target image can be segmented into foreground (including bulge features) and background by threshold-based image segmentation. The specified threshold divides the pixels into one of two levels to isolate the object. Thresholding converts a grayscale image into a binary image or distinguishes between lighter and darker pixels of a color image.

[0066] In one embodiment, by K-means clustering, the number of groups can be represented by a variable K, and each measurement point is assigned to one of the groups according to the feature similarity. The clustering can not be an analysis of predefined groups, but iteratively work so that the groups are formed organically.

[0067] In one embodiment, by histogram-based image segmentation, the measurement points can be grouped according to the gray scale by a histogram. The target image can be composed of an object and a background, and the background is a large entity with a gray scale. Therefore, a larger peak represents the background gray scale in the histogram. A smaller peak represents the raised feature of the wrinkle.

[0068] In one embodiment, by edge detection, the sharp change or discontinuity of brightness in the target image can be identified, and edge detection usually includes arranging discontinuous points into curve segments or edges, such as the boundary between the color of the raised feature of the wrinkle and the color of other parts.

[0069] Step S320. Contour extraction is performed on the feature image according to the optical appearance of the wrinkle and the image feature to measure the height information of each wrinkle and the spacing information between two adjacent wrinkles.

[0070] In one embodiment of the present application, when contour extraction is performed, the image feature contour extraction algorithm can be designed according to the optical appearance of the wrinkle and the image feature to accurately measure the spacing and size of the wrinkle.

[0071] Continuing to refer to Figure 2 As shown in FIG. 2B, in step S230, based on the wrinkle feature information, the height information of each wrinkle and the spacing information between two adjacent wrinkles are measured respectively.

[0072] In one embodiment of the present application, based on the feature recognition result in step S220, the wrinkle can be measured to measure the raised height H of each wrinkle and the spacing W between two adjacent wrinkles, as shown in FIG. 2C. Figure 10 As shown in FIG. 2C, the raised height of one wrinkle is 5 mm, and the spacing between two adjacent wrinkles is 22 mm.

[0073] In step S240, the height information and the spacing information are compared to identify the stomach filling state in the target image.

[0074] In one embodiment of the present application, when the spacing of most of the wrinkles in the target image is greater than the height of the wrinkle raised, it can be judged that the stomach has been opened and is suitable for stomach detection; otherwise, when the spacing of most of the wrinkles in the target image is less than the height of the wrinkle raised, it can be judged that the stomach has not been opened and is not suitable for further detection, and the person to be detected needs to drink water again to judge whether the stomach is opened.

[0075] In an embodiment of the present application, in step S240, the gastric filling state in the target image is identified by comparing the height information and the interval information. Figure 4 is a flowchart of the gastric filling state identification shown in an exemplary embodiment of the present application. The multiple-point measurement of the folds can be performed in the manner shown in Figure 4 , which includes the following steps:

[0076] S410. Selecting multiple measurement points in the target image.

[0077] In an embodiment of the present application, multiple measurement points are selected to perform the multiple-point measurement of the folds, as shown in Figure 11 , the measurement points include five, which are A, B, C, D, and E.

[0078] S420. Calculating the ratio of the interval information and the height information of the folds in each measurement point to obtain the filling parameter.

[0079] In an embodiment of the present application, as shown in Figure 11 , the height values of the five measurement points A, B, C, D, and E are H a = 4 mm, H b = 3 mm, H c = 3 mm, H d = 2 mm, and H e = 4 mm, respectively. The intervals of the adjacent folds in each measurement point are W a = 17 mm, W b = 26 mm, W c = 11 mm, W d = 14 mm, and W e = 12 mm, respectively. The ratio of the interval information and the height information of the folds in each measurement point, i.e., the filling parameter, is K a = 4.3, K b = 8.7, K c = 3.7, K d = 7.0, and K e = 3.0, respectively. It can be seen that the fold interval of all the measurement points in the target image is greater than the height of the fold protrusion, so it can be judged that the stomach has been opened and is suitable for gastric detection, as shown in Figure 18 .

[0080] It should be noted that the manner of selecting the measurement points in the target image can be arbitrary, for example, a predetermined number of points in the target image can be randomly selected as measurement points, or the target image can be divided into a plurality of image regions, and then a corresponding number of measurement points can be selected from each image region, or the measurement points can be selected according to image features related to the folds in the target image, for example, fewer measurement points can be selected in an image region with a larger fold spacing, and more measurement points can be selected in an image region with a smaller fold spacing, and the present embodiment is not limited in this regard.

[0081] S430. The filling parameter is compared with a preset first threshold value, and the measurement points with the filling parameter greater than the first threshold value in all the measurement points are counted to obtain a quantity statistical value;

[0082] In an embodiment of the present application, the first threshold value can be self-defined according to actual conditions, and by default, it can be 1. Based on this, the measurement points with the filling parameter greater than 1 in all the measurement points can be counted to obtain the quantity statistical value. In the present embodiment, the quantity statistical value can be the proportion of the measurement points with the filling parameter greater than the first threshold value in all the measurement points in a target image.

[0083] S440. The quantity statistical value is compared with a preset second threshold value, when the quantity statistical value is greater than the second threshold value, it is determined that the stomach is in a filled state, and when the quantity statistical value is less than or equal to the second threshold value, it is determined that the stomach is in an unfilled state;

[0084] In an embodiment of the present application, multi-point measurement can be used to determine whether the stomach is filled or not, and the second threshold value can be self-defined according to actual conditions, and can be set to any value between 75% and 85%, for example, the second threshold value can be set to 80% of the number of measurement points. By comparing the quantity statistical value with the preset second threshold value, it is determined whether the spacing of the folds at the plurality of points is greater than the height of the fold protrusion. If more than 80% of the measurement points have a fold spacing greater than the height of the fold protrusion, it is determined that the stomach is in a filled state, otherwise, if less than 80% of the measurement points have a fold spacing greater than the height of the fold protrusion, it is determined that the stomach is in an unfilled state. For n measurement points, the filling parameter and the average filling parameter can be obtained in the following manner:

[0085]

[0086]

[0087] wherein, K i is the filling parameter of the measurement point i, K is the average filling parameter, W i is the adjacent spacing information, H i is the height information.

[0088] As Figure 11 shown in FIG. 1 1, in the image I1, the number of measurement points is 5, and the filling parameters of each measurement point are K a = 4.3, K b = 8.7, K c = 3.7, K d = 7.0, and K e = 3.0, and the average filling parameter is 5.3.

[0089] In an embodiment of the present application, Figure 16 A block diagram of an image processing device suitable for implementing the embodiment of the present application is shown in FIG. 12. The device first determines whether a wrinkle exists in a target image, and then determines the filling state according to the recognition result, and further determines the water feedback according to the filling state. The water feedback is described in further detail below. After step S440, as shown in FIG. 13, when the stomach is determined to be in the filling state in S440, it is further determined whether the detection object corresponding to the target image needs to be supplemented with water, and the water supplement amount for filling the stomach is calculated when it is determined that the detection object needs to be supplemented with water; when the stomach is determined to be in the non-filling state in S440, it is determined that the detection object corresponding to the target image needs to be supplemented with water, and thus the water supplement amount for filling the stomach also needs to be calculated. Figure 16

[0090] In S440, when the stomach is determined to be in the filling state, the method shown in FIG. 14 can be used to further determine whether the detection object corresponding to the target image needs to be supplemented with water, and the process of calculating the corresponding water supplement amount when it is determined that the detection object needs to be supplemented with water, and the process specifically includes the following steps: determining whether the detection object corresponding to the target image needs to be supplemented with water according to the number statistical value; including: Figure 5

[0091] S510. Comparing the number statistical value with a third threshold value to determine whether the detection object corresponding to the target image needs to be supplemented with water, the third threshold value being greater than the second threshold value.

[0092] The number statistical value can reflect the filling state of the stomach to a large extent, and thus the embodiment can determine whether the detection object needs to be supplemented with water according to the number statistical value.

[0093] S520. If the number statistical value is greater than the third threshold value, it is determined that the detection object does not need to be supplemented with water.

[0094] When the number statistical value is greater than the third threshold value, it indicates that the stomach is highly filled, and thus it is determined that the detection object does not need to be supplemented with water. Exemplarily, the third threshold value can be any value in the range of 92% to 98%, for example, 95%.

[0095] S530. If the number statistical value is greater than the second threshold value and less than or equal to the third threshold value, it is determined that the detection object needs to be supplemented with water. ​​

[0096] When the quantity statistical value is greater than the second threshold value and less than or equal to the third threshold value, it indicates that the stomach is overall filled, but some of the folds are not fully opened, which can cause lesions to be missed due to fold obstruction, and thus it is determined that water needs to be supplemented.

[0097] S540. When it is determined that water needs to be supplemented, the water supplementing amount for filling the stomach is obtained.

[0098] As an exemplary embodiment, an average filling parameter is calculated according to the quantity of the measurement points and the filling parameters corresponding to the measurement points, and the water supplementing amount for filling the stomach is calculated according to the average filling parameter.

[0099] First, a sum value is calculated according to the filling parameters corresponding to the measurement points, and then an average value of the sum value is calculated according to the quantity of the measurement points, so that the average filling parameter is obtained. For example, for a target image I1, the specific detection values are as shown in Table 1. Figure 19 The filling parameters of each measurement point in the target image I1 are K a = 4.3, K b = 8.7, K c = 3.7, K d = 7.0, and K e = 3.0, and the average filling parameter is 5.3. In addition, the water supplementing amount for filling the stomach can be calculated according to the average filling parameter, and the formula is as follows:

[0100]

[0101] In the formula, L is the water supplementing amount, which is usually in units of ml (milliliter), and K is the average filling parameter of the current target image (in this embodiment, it is the target image I1).

[0102] For example, for a target image I2, as shown in Table 2, the average filling parameter of all the measurement points is 0.3, and it is determined that 200 mL of water needs to be supplemented, as shown in Table 3. Figure 12 Figure 20

[0103] In S440, when it is determined that the stomach is in an unfilled state, the process of further calculating the corresponding water supplementing amount can be performed according to the method in Table 4, and the water supplementing amount of the detection object corresponding to the target image is determined according to the quantity statistical value, and the specific steps include the following steps: Figure 6

[0104] S610. The quantity statistical value is compared with a fourth threshold value to determine the water supplementing amount of the detection object corresponding to the target image, and the fourth threshold value is less than the second threshold value.

[0105] ​​​As mentioned above, the quantity statistical value can reflect the filling state of the stomach to a large extent, and thus the embodiment determines the specific water supplement amount according to the quantity statistical value.

[0106] The fourth threshold value is less than the second threshold value, and the fourth threshold value can be any value between 30% and 50%, for example, 40% according to experience.

[0107] S620. If the quantity statistical value is greater than the fourth threshold value, the water supplement amount for filling the stomach is obtained according to the average filling parameter.

[0108] As an exemplary embodiment, the average filling parameter is calculated according to the number of measurement points and the filling parameter corresponding to each measurement point, and the water supplement amount for filling the stomach is calculated according to the average filling parameter and the filling standard value.

[0109] The filling standard value represents the average filling parameter of the target image corresponding to the case that the quantity statistical value of the measurement point is equal to the second threshold value in the image acquisition process. However, it is difficult to acquire the target image corresponding to the case that the quantity statistical value of the measurement point is equal to the second threshold value in actual operation, and thus it can be considered that the filling standard value represents the average filling parameter of the target image corresponding to the case that the quantity statistical value of the measurement point is closest to the second threshold value in the process of acquiring the stomach image of the detection object corresponding to the target image. It can be understood that the process of acquiring the stomach image by using the capsule endoscope is a dynamic process, in which the stomach image gradually changes from the filling state to the unfilling state, that is, multiple stomach images are acquired, and the average filling parameter corresponding to each stomach image is also obtained. The embodiment refers to the average filling parameter of the image corresponding to the case that the quantity statistical value of the measurement point is closest to the second threshold value in the acquired target image as the filling standard value.

[0110] The water supplement amount for filling the stomach can be calculated according to the following formula:

[0111]

[0112] Wherein, L is the water supplement amount, usually in units of ml (milliliter), K is the average filling parameter of the current target image, K' is the filling standard value, and X is the numerical part corresponding to the second threshold value, for example, when the second threshold value is 80%, X is 80.

[0113] S630. If the data statistical value is less than or equal to the fourth threshold value, the water supplement amount for filling the stomach is determined according to the preset water supplement amount range.

[0114] When the statistical value of the data of the measuring point is less than or equal to the fourth threshold value, it indicates that the stomach is in an unsuitable state for observation, and thus the water supplement amount for filling the stomach is determined according to a preset water supplement amount range. For example, the preset water supplement amount range is between 100 ml and 200 ml. It can be understood that the end values of the water supplement amount range can be determined according to corresponding object type information of the detected object, such as body type, gender, age, etc. Correspondingly, the water supplement amount that can fill the stomach of the detected object corresponding to the target image can be determined based on the preset water supplement amount range according to the corresponding type information, for example, if the detected object is a child, or has a thin body type, or is a female, a smaller water supplement amount can be determined from the preset water supplement amount range as the water supplement amount of the detected object. After the water supplement amount is supplemented, the filling state of the stomach can be re-determined after a preset period of time, which is not described herein.

[0115] As can be seen from the above, in one embodiment of the present application, after it is determined that water supplement is needed, on the one hand, if the water supplement amount is insufficient, the stomach cannot be fully opened, which can result in incomplete capsule detection area; on the other hand, if the stomach is fully opened, continuing to introduce additional water can cause discomfort of the detected person due to excessive water drinking. In the present embodiment, a feedback mechanism can be set according to the relationship between the water supplement amount and the change of the fold filling state, so as to realize feedback of the specific water supplement amount, so that the water supplement amount is not too small or too large, which can greatly improve the comfort of the tested person during the detection process.

[0116] It should be further noted that the above formulas (3) and (4) are obtained by obtaining a mapping relationship between the pre-set filling parameter and the water supplement amount, and according to the mapping relationship and the average filling parameter corresponding to the measuring point determined to need water supplement, so as to obtain the water supplement amount for filling the stomach in the target image. The mapping relationship is determined based on the influence relationship of the water supplement amount on the filling state of the stomach.

[0117] In one embodiment of the present application, the mapping relationship between the filling parameter and the water supplement amount can be set in the form of an empirical formula, for example, calculated by the formula in formula (3) or (4).

[0118] In one embodiment of the present application, Figure 7Another feedback water supplement amount method is shown in the flowchart, which realizes feedback of specific water supplement amount by combining a neural network model. Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. It is applied in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. Figure 7 The neural network model in machine learning is used to realize feedback of water supplement amount, so that the testee can introduce quantitative water supplement, or simply consider that the least amount of water can be introduced to form the required fullness state of the stomach to achieve the required morphological standard of stomach detection. As shown in Figure 7 The specific steps include the following:

[0119] S710. Collect sample image data of different stomach fullness states.

[0120] In an embodiment of the present application, sample image data can be collected in real time or selected by technicians from existing historical image data or other image data. The purpose of collecting sample image data is to train the subsequent deep learning. In the deep learning training, the original sample data can be divided into a training sample data set and a verification sample data set. The proportion of verification sample data can be pre-set, for example, the proportion is set to 20%. Then the training sample data set is used for deep learning training, and the verification sample data set is used to predict and evaluate the accuracy of the deep learning training result. In this embodiment, when independent sample image data is made, the training sample data set and the verification sample data set can be divided according to a certain strategy, for example, a random proportion is used. When the original sample image data is large and the sample standard is relatively small compared to the sample data, it can be processed by cropping and other methods to meet the size requirements.

[0121] S720. Label the sample image data according to the water supplement amount level and the fullness parameter to form a sample data set.

[0122] In an embodiment of the present application, the potential correlation or mapping relationship between the water filling amount and the visual feature space (i.e. the bulging feature of the wrinkle corresponding to the filling parameter) is automatically learned by using the images labeled with the water filling amount grades and the filling parameters or other available information, so as to add the filling parameters and the corresponding water filling amount to the unknown images. The image labeling in the embodiment can be manual labeling or automatic labeling. The automatic labeling refers to a process of automatically adding text feature information reflecting the content of the image to the image by means of machine learning with respect to the visual content of the image. After the image automatic labeling technology is processed, the image information problem can be converted into a relatively mature text information processing problem. For example, a supervised machine learning based classification image labeling manner or a semi-supervised model image automatic labeling manner can be used.

[0123] S730. A convolutional neural network model is established based on the sample data set and is trained.

[0124] In an embodiment of the present application, the convolutional neural network has representation learning ability and can perform shift-invariant classification on the input information according to its hierarchical structure. The input layer of the convolutional neural network can process multi-dimensional data. The input layer of the one-dimensional convolutional neural network receives one-dimensional or two-dimensional arrays. The hidden layer of the convolutional neural network includes three common structures of convolution layer, pooling layer and full connection layer. The input data can be feature extracted by the convolution layer. Each element of the convolution kernel corresponds to a weight coefficient and a bias amount, which is similar to a neuron of a feedforward neural network. Each neuron in the convolution layer is connected to multiple neurons in the region close to the position in the previous layer, and the size of the region depends on the size of the convolution kernel. The convolution kernel works by regularly scanning the input feature, performing matrix element multiplication and summation on the input feature in the receptive field and adding the bias amount. In the training process of the neural network model, the training data can be used for training, but the samples on the test set cannot be accessed. The neural network model can be verified by constructing a verification data set, and the overfitting can be prevented by means of regularization method, data augmentation, re-cleaning data, early stopping, dropout, etc.

[0125] S740. The filling parameter of the target image is input into the trained convolutional neural network model to obtain the water filling amount for filling the stomach in the target image.

[0126] In an embodiment of the present application, the correlation between the water filling amount and the wrinkle feature is automatically learned by using the trained convolutional neural network model, so as to output the water filling amount of the stomach in the target image. The upstream of the output layer in the convolutional neural network is usually a fully connected layer, which has the same structure and working principle as the output layer in the traditional feedforward neural network. For image classification problems, the output layer uses a logistic function or a normalized exponential function (softmax function) to output a classification label. In the recognition problem of the wrinkle feature, the output layer can be designed to output the center coordinates, size and classification of the object. In image semantic segmentation, the output layer directly outputs the water filling amount classification (grading or specific value) result of each measurement point. In the embodiment, the trained convolutional neural network model is used to realize the fine and quantitative detection of the water intake of the subject.

[0127] In an embodiment of the present application, for the contour extraction of the feature image in step S320, Figure 8 A flowchart of a contour extraction method is shown. As Figure 8 shown, the method specifically includes the following steps: step S810, acquiring the target image of the continuous frame and the corresponding wrinkle feature information; step S820, obtaining the same wrinkle contour in the target image of the continuous frame according to the wrinkle feature information; step S830, fusing the same wrinkle contour in the target image of the continuous frame to perform target tracking, and performing contour extraction on the fused wrinkle contour. Specifically, the wrinkle contour of a frame of the target image and the wrinkle contour of the previous frame are extracted; corner point detection is performed on the target image, and optical flow is calculated under multiple scales to obtain the displacement of the target wrinkle contour; then, the same wrinkle contour of the adjacent two frames is fused according to the displacement. In the adjacent two frames, when a point (x, y) has a displacement (u, v), the brightness of the two points before and after the movement is equal, and the displacement of each point in the adjacent two frames can be calculated according to the spatial correlation. By this target tracking method, a series of wrinkle contours in a frame of the internal image of the stomach and the wrinkle contour in the previous frame of the internal image of the stomach are extracted from a continuous image frame sequence, the same wrinkle contour in the above two frames of the internal image of the stomach is fused, and then the measurement of the wrinkle spacing and height is performed, so that the wrinkle spacing and the height of the protrusion can be more accurately measured, and the measurement accuracy can be effectively improved.

[0128] In an embodiment of the present application, real-time feedback can be provided for the results of image processing, for example, automatic real-time determination of the opening of the stomach, and a reminder message can be sent. The reminder can be in the form of text or animation displayed on a display device, and the required water supplement can be displayed simultaneously. Of course, the reminder can also be in the form of voice broadcast. When it is determined that water supplement is needed, the testee can be prompted to drink water while the stomach filling state is identified in real time, or the testee can be prompted to start and end water supplement automatically by voice, thereby improving the detection efficiency and integrity of the capsule endoscope.

[0129] In an embodiment of the present application, a capsule endoscope has a structure as shown in Figure 9 The capsule endoscope is one of endoscopes, and is a full name of a magnetic capsule endoscope system. The capsule endoscope can be controlled by a real-time and accurate external magnetic field to control the movement of the capsule in the stomach, change the posture of the capsule, and take photos of a lesion at a required angle, thereby accurately checking the stomach.

[0130] In an embodiment of the present application, binocular cameras can be used to collect images of the stomach. It should be noted that parallax refers to the change and difference in the position of an object in the field of view when the object is observed from two different positions. The angle between the two lines of sight when looking at the target from two observation points is called the parallax angle of the two points, and the distance between the two points is called the parallax baseline. If the same point is projected on the camera at different distances, it is impossible to determine the distance of the point by using a monocular camera. However, the binocular camera can determine the distance by using binocular stereo vision. By using the binocular camera, the collected target image can be more accurate for subsequent image processing and result generation.

[0131] In an embodiment of the present application, the binocular camera in the embodiment is different from the above-mentioned embodiments in that different specifications and types of cameras can be used in the embodiment, so that two different images can be obtained at the same time. For example, one camera can be a normal camera, and the other camera can be an infrared camera, such as an infrared thermal imager. The visible light image and the infrared light image of the same position can be obtained at the same time, and the two images can be fused to obtain a target image, thereby realizing target fusion detection based on visible light and infrared light. In the special application scenario of stomach examination, the advantages of three-dimensional information obtained by binocular stereo vision can be fully utilized to obtain three-dimensional information of a space target. The gray-scale information provided by visible light and the temperature information provided by the infrared thermal imager are used to extract the motion information and three-dimensional spatial information of the target, and the information is complementary for continuous tracking of the target in the complex environment of the stomach, so that the collected target image can be more accurate for subsequent image processing and result generation.

[0132] In the embodiment, for the complex problem of binocular system parameter calibration, due to the specific application scenario in the embodiment, the actual height information of the observation point or the measurement point is not important, therefore, the embodiment can directly randomly select four points in the common field of view in two images as reference points and the centers of the two cameras, and obtain the parameters required for calibration by measuring the related distances between the six points, in actual application, without the aid of a calibration template, only the images taken by the two cameras at the same time need to contain all the reference points. Using the calibrated binocular camera, three-dimensional measurement can be realized even if the relative position of the binoculars changes in rotation or translation in the scene of intragastric examination.

[0133] In an embodiment of the present application, different from the above-mentioned embodiments, the camera in the embodiment can be a multi-camera, which can be the same specification and type of camera, or different specifications and types of cameras.

[0134] In an embodiment of the present application, Figure 9 is a schematic diagram of the observation angle of a capsule endoscope and a stomach wall according to an exemplary embodiment of the present application, when target recognition is performed on the images obtained by the capsule gastroscope, as shown in Figure 9 the images obtained by the capsule gastroscope and the stomach body within a certain angle range θ (such as 0-20 degrees) are recognized, and the images with wrinkle information are extracted to ensure the accuracy of the measurement results. In an embodiment, the stomach detection images are collected by the capsule gastroscope, the position of the capsule gastroscope can be adjusted periodically to keep the included angle between the capsule gastroscope and the stomach body within the preset angle range, and the included angle between the capsule gastroscope and the stomach body can be obtained in real time by the acceleration sensor in the capsule gastroscope. In another embodiment, the software can be used to manually or automatically control the capsule to have a posture that meets the included angle θ at certain detection time intervals, so as to ensure that the images meeting the observation requirements are obtained at certain time intervals to judge the opening of the stomach.

[0135] It can be seen that the image processing method in the embodiment is not limited to receiving the target image collected by the single image collection device 101. In actual application scenarios, the computer device 102 can simultaneously perform data interaction with multiple image collection devices 101, and the multiple image collection devices 101 can be identified by identity identification. The image collection device 101 can perform data interaction with the computer device 102 in a wired manner, or perform data interaction with the computer device 102 in an infrared, Bluetooth, zigbee, NFC, or other near-field wireless technology, or perform communication with the computer device 102 in a GPRS (General packet radio service), 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), 5G (fifth generation mobile information technology), or other wireless network, and the present application is not limited in this regard.

[0136] The device embodiment of the present application is introduced below, which can be used to execute the image processing method in the above-mentioned embodiments of the present application. For details not disclosed in the device embodiment of the present application, please refer to the above-mentioned embodiments of the image processing method of the present application.

[0137] Figure 17 is a block diagram of an image processing device according to an exemplary embodiment of the present application. The device can be applied to the implementation environment shown in Figure 2 , and is specifically configured in the computer device 102. The device can also be applied to other exemplary implementation environments, and is specifically configured in other devices, and the present embodiment does not limit the implementation environment to which the device is applied.

[0138] As shown in Figure 17 , the exemplary image processing device includes an image collection unit 1401, a feature identification unit 1402, a measurement unit 1403, and a filling identification unit 1404.

[0139] The image collection unit 1401 is configured to obtain a target image; the feature identification unit 1402 is configured to perform feature identification on the target image to obtain wrinkle feature information corresponding to the target image; the measurement unit 1403 is configured to measure the height information of each wrinkle and the spacing information of adjacent two wrinkles based on the wrinkle feature information; and the filling identification unit 1404 is configured to identify the stomach filling condition in the target image by comparing the height information and the spacing information.

[0140] In another exemplary embodiment, the feature recognition unit 1402 comprises: an image segmentation subunit configured to perform image segmentation on the target image to obtain a plurality of feature images; and a contour extraction subunit configured to perform contour extraction on the feature images according to the optical appearance of the rugae and the image features to measure the height information of each ruga and the spacing information between two adjacent rugae.

[0141] In another exemplary embodiment, the measurement unit 1403 is configured to: select a plurality of measurement points in the target image; calculate the ratio of the spacing information and the height information of the rugae in each measurement point to obtain a filling parameter; compare the filling parameter with a preset first threshold value, and count the measurement points with the filling parameter greater than the first threshold value to obtain a quantity statistical value; and the filling recognition module is configured to: compare the quantity statistical value with a preset second threshold value; when the quantity statistical value is greater than the second threshold value, it is determined that the stomach is in a filled state; and when the quantity statistical value is less than or equal to the second threshold value, it is determined that the stomach is in an unfilled state.

[0142] In another exemplary embodiment, the filling recognition unit 1404 comprises: a first processing unit configured to compare the quantity statistical value with a third threshold value when the stomach is in the filled state to determine whether the detection object corresponding to the target image needs to be supplemented with water, the third threshold value being greater than the second threshold value; if the quantity statistical value is greater than the third threshold value, it is determined that the detection object corresponding to the target image does not need to be supplemented with water; if the quantity statistical value is greater than the second threshold value and less than or equal to the third threshold value, it is determined that the detection object corresponding to the target image needs to be supplemented with water; and when it is determined that the detection object needs to be supplemented with water, the amount of water to fill the stomach is obtained.

[0143] In another exemplary embodiment, the filling recognition unit 1404 further comprises: a second processing unit configured to compare the quantity statistical value with a fourth threshold value when the stomach is in the unfilled state to determine the amount of water to supplement the detection object corresponding to the target image, the fourth threshold value being less than the second threshold value; and specifically comprising: comparing the quantity statistical value with the fourth threshold value to determine the amount of water to supplement the detection object corresponding to the target image; if the quantity statistical value is greater than the fourth threshold value, the amount of water to fill the stomach is obtained according to the average filling parameter, the fourth threshold value being less than the second threshold value; and if the quantity statistical value is less than or equal to the fourth threshold value, the amount of water to fill the stomach is determined according to a preset water supplement range.

[0144] In another exemplary embodiment, the filling recognition unit 1404 comprises: a first water supplement amount recommendation subunit configured to pre-set a mapping relationship between the filling parameter and the water supplement amount based on the influence relationship of the water supplement amount on the filling state of the stomach, and obtain the amount of water to fill the stomach in the target image according to the mapping relationship and the average filling parameter of the measurement points determined to need to be supplemented with water.

[0145] In another example embodiment, the filling recognition unit 1404 comprises a second water supplement amount recommendation subunit, which comprises a sample data subunit configured to collect sample image data of different stomach filling conditions; a labeling subunit configured to label the sample image data according to water supplement amount levels and filling parameters to form a sample data set; a neural network model configured to establish a convolutional neural network model based on the sample data set and perform training; and an output subunit configured to input the filling parameters of a target image into the trained convolutional neural network model to obtain the water supplement amount for the stomach filling in the target image.

[0146] In another example embodiment, the contour extraction subunit is configured to obtain consecutive frame target images, fuse the common wrinkle contours in the consecutive frame target images to perform target tracking, and perform contour extraction on the fused wrinkle contours.

[0147] In another example embodiment, the contour extraction subunit is configured to extract the wrinkle contour of a target image in one of the consecutive frames and the wrinkle contour of the previous frame; perform corner point detection on the target image and calculate the optical flow under multiple scales to obtain the displacement amount of the target wrinkle contour; and fuse the same wrinkle contour of the adjacent two frames according to the displacement amount.

[0148] In another example embodiment, the image acquisition unit 1401 comprises a capsule endoscope configured to acquire a stomach detection image; and a preprocessing subunit configured to perform target recognition on the acquired image to obtain a target image with wrinkle information.

[0149] In another example embodiment, the image acquisition unit 1401 further comprises a detection subunit configured to detect the included angle between the capsule endoscope and the stomach body; and an adjustment mechanism configured to periodically adjust the position of the capsule endoscope to keep the included angle between the capsule endoscope and the stomach body within a preset angle range, so as to extract an image with wrinkle information.

[0150] In another example embodiment, the acquisition subunit can be configured as a binocular camera. It should be noted that the parallax refers to the change and difference in the position of an object in the field of view when the object is observed from two different positions. The included angle between the two lines of sight when looking at a target from two observation points is called the parallax angle of the two points, and the distance between the two points is called the parallax baseline. If a point is projected on the same position on the camera at different distances, it is impossible to determine the distance of the point through a monocular camera. However, a binocular camera can measure the distance through binocular stereo vision. Through the binocular camera, the target image collected can be more accurate in the subsequent image processing and the generated judgment result.

[0151] In another example embodiment, different from the above embodiment, in the present embodiment, the binocular camera can adopt different specifications and types of cameras, for example, one can adopt a normal camera, and the other can adopt an infrared camera, such as an infrared thermal imager. Through different specifications and types of cameras, two different images can be obtained at the same time, for example, a visible light image and an infrared light image of the same position can be obtained at the same time, and then the two images are fused to obtain a target image, and the visible light and thermal infrared video motion target fusion detection is performed. In the special application scenario of the in-gastric examination, by fully utilizing the advantages of the binocular stereo vision that can obtain three-dimensional information, the spatial target three-dimensional information is obtained, the gray information provided by the visible light and the temperature information provided by the infrared thermal imager are utilized, the motion information and three-dimensional spatial information of the target are extracted, the information complementation is realized for the continuous tracking of the target in the complex environment in the stomach, so that the target image collected can be more accurate for the subsequent image processing and the generated judgment result.

[0152] In another example embodiment, for the complex problem of binocular system parameter calibration, since the actual height information of the observation point or the measurement point is not important in the specific application scenario of the present embodiment, the present embodiment can directly randomly select four points in the common field of view in the two images as reference points and the centers of the two cameras, and obtain the required parameters for calibration by measuring the related distances between the six points. In actual application, without the aid of a calibration template, it is only required to ensure that all the reference points exist in the two cameras at the same time. By using the calibrated binocular camera, three-dimensional measurement can still be realized even if the relative position of the binoculars changes in rotation or translation in the in-gastric examination scenario.

[0153] In another example embodiment, different from the above embodiment, in the present embodiment, the camera can adopt a multi-view camera, which can adopt the same specifications and types of cameras or different specifications and types of cameras.

[0154] It should be noted that the image processing apparatus provided in the above embodiments and the image processing method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. In actual application, the above functions can be distributed to different functional modules to be completed, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions, and this is not limited herein.

[0155] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method provided in each of the above embodiments.

[0156] Figure 15 A structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 15 The computer system 1200 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0157] As Figure 15 shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1202 or programs loaded from a storage portion 1208 into a random access memory (RAM) 1203, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0158] The following components are connected to the I / O interface 1205: an input portion 1206 including a keyboard, a mouse, and the like; an output portion 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 1208 including a hard disk, and the like; and a communication portion 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as necessary. A removable recording medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed in the storage portion 1208 as necessary.

[0159] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing computer programs for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 1209, and / or installed from the detachable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0160] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave propagating in a baseband, in which the computer readable computer program is carried. Such a propagating data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit the program for use by or in connection with the instruction execution system, apparatus or device. The computer program contained on the computer readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0161] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowcharts, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0162] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may

[0163] Another aspect of the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the image processing method as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately from the electronic device.

[0164] It should be noted that although several modules or units for performing actions are mentioned in the above detailed description, the division into such modules or units is not mandatory. In fact, according to the embodiments of the present application, features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0165] Another aspect of the present application provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method provided in the above embodiments.

[0166] Those skilled in the art can clearly understand the example embodiments described herein through the above description of the example embodiments, and the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to perform the method according to the embodiments of the present application.

[0167] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the present application cover any and all variations of the present application that come within the scope of the claims and that the claims be prevailing over any prior art. It is intended that the specification and examples be considered exemplary only, with the true scope of the application being indicated by the following claims.

[0168] It should be understood that the above description is only the preferred example embodiments of the present application, and is not intended to limit the embodiments of the present application. Those skilled in the art can easily make corresponding modifications or changes according to the main concept and spirit of the present application, and the protection scope of the present application should be subject to the protection scope claimed in the claims.

Claims

1. An image processing method, characterized in that, include: Acquire a target image, which includes a stomach image; The target image is subjected to feature recognition to obtain the fold feature information corresponding to the target image; Based on the fold feature information, the height information of each fold and the distance information between two adjacent folds are measured respectively. The height information and spacing information are compared to identify the stomach fullness state in the target image; The step of comparing the height information and the spacing information to identify the stomach fullness state in the target image includes: Multiple measurement points are selected in the target image; Calculate the ratio of the spacing information to the height information of the folds at each measurement point to obtain the filling parameter; The filling parameter is compared with a preset first threshold, and the number of measurement points with a filling parameter greater than the first threshold is counted to obtain a statistical value. The statistical value of the quantity is compared with a preset second threshold; When the statistical value of the quantity is greater than the second threshold, the stomach is determined to be in a full state; When the statistical value of the quantity is less than or equal to the second threshold, the stomach is determined to be in an unfilled state.

2. The image processing method according to claim 1, characterized in that, Feature recognition of the target image includes: The target image is segmented to obtain multiple feature images; Based on the optical characteristics and image features of the folds, contour extraction is performed on the feature image to measure the height information of each fold and the distance information between two adjacent folds.

3. The image processing method according to claim 1, characterized in that, After identifying the state of stomach fullness in the target image, the process further includes: When the stomach is full, determine whether the detection object corresponding to the target image needs hydration based on the statistical values; including: The statistical value is compared with the third threshold to determine whether the detection object corresponding to the target image needs water replenishment. The third threshold is greater than the second threshold. If the statistical value of the quantity is greater than the third threshold, it is determined that no water replenishment is needed; If the statistical value of the quantity is greater than the second threshold and less than or equal to the third threshold, it is determined that water needs to be added; When it is determined that water needs to be replenished, obtain an amount of water to fill the stomach.

4. The image processing method according to claim 1, characterized in that, After identifying the state of stomach fullness in the target image, the process further includes: When the stomach is not full, the amount of water replenishment for the detection object corresponding to the target image is determined based on the statistical values; including: The statistical value is compared with the fourth threshold to determine the amount of water replenishment for the target image corresponding to the detection object. The fourth threshold is less than the second threshold. If the statistical value of the quantity is greater than the fourth threshold, the amount of water replenishment required to fill the stomach is obtained based on the average filling parameter. If the statistical value is less than or equal to the fourth threshold, the amount of water to fill the stomach is determined according to the preset water replenishment range.

5. The image processing method according to claim 3 or 4, characterized in that, The process of obtaining the amount of water needed to fill the stomach includes: Obtain the mapping relationship between the pre-set filling parameters and the water replenishment amount, which is determined based on the influence of the water replenishment amount on the stomach filling state; Based on the mapping relationship and the average filling parameters corresponding to the measurement points that are determined to require water replenishment, the amount of water replenishment required to fill the stomach in the target image is obtained.

6. The image processing method according to claim 3 or 4, characterized in that, The process of obtaining the amount of water needed to fill the stomach includes: Collect sample image data at different gastric fullness states; The sample image data were labeled according to the water replenishment level and filling parameters to form a sample dataset; A convolutional neural network model was built based on the aforementioned sample dataset and trained. The filling parameters of the target image are input into the trained convolutional neural network model to obtain the amount of water added to fill the stomach in the target image.

7. The image processing method according to claim 2, characterized in that, Contour extraction of the feature image includes: Acquire the target image of consecutive frames, and the fold feature information corresponding to the target image of the consecutive frames; Based on the wrinkle feature information, the same wrinkle contour in the target image of the consecutive frames is obtained; The same fold contour in the target image of the consecutive frames is fused, and the fused fold contour is extracted.

8. The image processing method according to claim 7, characterized in that, Fusing the same fold contour in the consecutive frames of the target image includes: Extract the fold contour of a target image in a consecutive frame and the fold contour of the previous frame. Corner detection is performed on the target image, and optical flow is calculated at multiple scales to obtain the displacement of the target fold contour. Based on the displacement, the same fold contour of two adjacent frames is merged.

9. The image processing method according to claim 1, characterized in that, Acquire the target image, including: The stomach image acquired by the capsule endoscope is obtained, and target recognition is performed on the stomach image to obtain a target image in which the angle between the capsule endoscope and the stomach body is kept within a preset angle range.

10. The image processing method according to claim 9, characterized in that, Before acquiring the target image, the following steps are also included: The angle between the capsule endoscope and the stomach body is detected, and the position of the capsule endoscope is periodically adjusted to keep the angle between the capsule endoscope and the stomach body within a preset range.

11. An image processing apparatus, characterized in that, include: An image acquisition module is used to acquire a target image, including a stomach image; The feature recognition module is used to perform feature recognition on the target image to obtain the fold feature information corresponding to the target image; The measurement module is used to measure the height information of each fold and the distance information between two adjacent folds based on the fold feature information. A fullness recognition module is used to compare the height information and the spacing information to identify the stomach fullness in the target image; The fullness recognition module is further configured to perform the following steps: Multiple measurement points are selected in the target image; Calculate the ratio of the spacing information to the height information of the folds at each measurement point to obtain the filling parameter; The filling parameter is compared with a preset first threshold, and the number of measurement points with a filling parameter greater than the first threshold is counted to obtain a statistical value. The statistical value of the quantity is compared with a preset second threshold; When the statistical value of the quantity is greater than the second threshold, the stomach is determined to be in a full state; When the statistical value of the quantity is less than or equal to the second threshold, the stomach is determined to be in an unfilled state.

12. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the processor of a computer, cause the computer to perform the image processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, Includes computer instructions, which are stored in a computer-readable storage medium; The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image processing method according to any one of claims 1 to 10.

Images