Esophagus information detection method and system, processing method and equipment and storage medium

Abstract

The invention discloses an esophagus information detection method and system, a processing method and device and a storage medium. The esophagus information detection method comprises the steps that first feature information shown by an accessory when a sampling device is located at the cardia part is obtained; obtaining esophageal sampling data detected by the sampling device between the cardia part and the esophageal starting part; obtaining second feature information shown by the accessory when the sampling device is located at the starting part of the esophagus; determining scale information according to the first feature information and the second feature information; determining esophageal state information relative to a world coordinate system according to the scale information and the esophageal sampling data. According to the esophageal information detection method provided by the invention, esophageal state information relative to a world coordinate system can be constructed as intermediate data and can be used for recovering an absolute size and providing accurate reference for subsequent diagnosis and treatment.

Description

Technical Field

[0001] This invention relates to the field of information detection and processing technology, and in particular to a method, system, processing method, device and storage medium for esophageal information detection. Background Technology

[0002] Due to changes in diet and lifestyle, the esophagus of most people is in a sub-healthy state. Detecting information inside the esophagus as intermediate data to understand the current health status of the human body in order to assist in subsequent diagnosis and treatment has gradually become a necessity in related fields. In addition, using information inside esophageal models to conduct training and teaching for medical personnel is also a powerful means to promote modern education and teaching.

[0003] The methods for detecting esophageal information, especially for achieving three-dimensional reconstruction of the esophagus, are mainly achieved through multi-slice spiral CT (Computed Tomography) three-dimensional reconstruction and MRI reconstruction in one existing technology. However, this technical solution can have a certain impact on the human body and cannot directly observe the severity of lesions.

[0004] Another existing technology provides a technical solution for information detection through capsule endoscopy. However, the information obtained by capsule endoscopy is based on the coordinate system of the capsule endoscopy itself, which can only restore the three-dimensional shape of the esophagus and cannot restore the absolute size of the esophagus. As a result, it is impossible to determine the size of the lesion, and medical staff will find it difficult to carry out subsequent diagnosis and treatment based on this. Summary of the Invention

[0005] One of the objectives of this invention is to provide a method for detecting esophageal information, in order to solve the technical problem that existing esophageal detection methods cannot recover absolute dimensions and cannot provide accurate references for judging the state of the esophagus.

[0006] One of the objectives of this invention is to provide a method for processing esophageal information.

[0007] One of the objectives of this invention is to provide an esophageal information detection system.

[0008] One of the objectives of this invention is to provide an electronic device.

[0009] One of the objectives of this invention is to provide a computer-readable storage medium.

[0010] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides an esophageal information detection method. The esophageal information detection method is applied to an esophageal information detection system, which includes a sampling device and an accessory associated with the sampling device. When the sampling device collects esophageal information, the accessory uses its characteristic information to indicate the positional relationship between the sampling device and the esophagus relative to a world coordinate system. The esophageal information detection method includes: obtaining first characteristic information indicated by the accessory when the sampling device is located at the cardia; obtaining esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus; obtaining second characteristic information indicated by the accessory when the sampling device is located at the beginning of the esophagus; determining scale information based on the first and second characteristic information; and determining esophageal state information relative to a world coordinate system based on the scale information and the esophageal sampling data.

[0011] As a further improvement of one embodiment of the present invention, the accessory feature information includes the positional relationship between the first accessory part and the first reference point, wherein the first reference point is stationary relative to the esophagus, and the relative position between the first accessory part and the first reference point changes with the position of the sampling device.

[0012] As a further improvement of one embodiment of the present invention, the first feature information includes the first accessory position information of the accessory at the incisor position when the sampling device is located at the cardia, and the second feature information includes the second accessory position information of the accessory at the incisor position when the sampling device is located at the beginning of the esophagus.

[0013] As a further improvement of one embodiment of the present invention, the step of determining the scale information based on the first feature information and the second feature information includes: determining the scale information based on the difference between the first attachment position information and the second attachment position information.

[0014] As a further improvement of one embodiment of the present invention, the esophageal information detection method further includes: obtaining the difference in device position information between the sampling device at the cardia and at the beginning of the esophagus; the step of determining scale information based on the first feature information and the second feature information includes: determining the scale information based on the difference between the first feature information and the second feature information, and the difference in device position information.

[0015] As a further improvement of one embodiment of the present invention, obtaining the difference in device position information between the sampling device at the cardia and at the beginning of the esophagus includes: obtaining a first sampling image of the sampling device at the cardia; determining first device position information of the sampling device based on the first sampling image; obtaining a second sampling image of the sampling device at the beginning of the esophagus; determining second device position information of the sampling device based on the second sampling image; and determining the moving distance of the sampling device to represent the difference in device position information based on the first device position information and the second device position information.

[0016] As a further improvement of one embodiment of the present invention, the step of determining the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data includes: updating the device position information based on the scale information; the device position information corresponds to the sampling process of the sampling device; updating the sparse point cloud data based on the scale information; the sparse point cloud data corresponds to the sampling process of the sampling device; and performing three-dimensional reconstruction based on the updated device position information and the updated sparse point cloud data to determine the three-dimensional esophageal information relative to the world coordinate system, which is used to indicate the current esophageal state.

[0017] As a further improvement of one embodiment of the present invention, the device location information and the sparse point cloud data correspond to each other; the three-dimensional reconstruction includes: depth reconstruction, dense reconstruction, point cloud fusion, mesh reconstruction and texture reconstruction.

[0018] As a further improvement of one embodiment of the present invention, obtaining the esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus includes: obtaining a plurality of esophageal sampling images detected by the sampling device between the cardia and the beginning of the esophagus; calculating feature point descriptors based on the esophageal sampling images; determining the initial pose information of the sampling device corresponding to the esophageal sampling images based on the matching of feature point descriptors between the esophageal sampling images; and performing global optimization based on the feature point descriptors corresponding to the esophageal sampling images and the initial pose information to obtain sparse point cloud data and device pose information; the device pose information includes device position information and device posture information.

[0019] As a further improvement of one embodiment of the present invention, the esophageal information detection further includes: obtaining camera parameters of the sampling device; performing distortion correction processing on the esophageal sampling image according to the camera parameters; and calculating feature point descriptors based on the esophageal sampling image, which includes: calculating feature point descriptors based on the distortion-corrected esophageal sampling image.

[0020] As a further improvement of one embodiment of the present invention, the step of performing global optimization based on the feature point descriptor corresponding to the esophageal sampling image and the initial pose information to obtain sparse point cloud data and device pose information includes: determining the sparse point cloud data and the reprojection error of the feature points based on the feature point descriptor and the initial pose information; and performing bundled adjustment optimization with the goal of minimizing the reprojection error to adjust and determine the sparse point cloud data and device pose information.

[0021] As a further improvement of one embodiment of the present invention, the feature point descriptor is a SURF feature point descriptor.

[0022] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides an esophageal information processing method, comprising: determining esophageal state information according to the esophageal information detection method described in any of the above technical solutions; obtaining sparse point cloud data corresponding to lesions in the esophageal state information; determining the area of ​​several triangular facets corresponding to the lesions by using triangular facet segmentation based on the sparse point cloud data of the lesions; and determining the total area of ​​the lesions based on the area of ​​the triangular facets.

[0023] To achieve one of the aforementioned objectives, one embodiment of the present invention provides an esophageal information detection system, comprising: a capsule endoscope for detecting esophageal sampling data; a traction wire, one end of which is fixed to the capsule endoscope, and the other end of which indicates the positional relationship between the capsule endoscope and the esophagus relative to a world coordinate system at its position at the incisor region; a data acquisition module for obtaining first feature information indicated by the traction wire when the capsule endoscope is located at the cardia, for obtaining second feature information indicated by the traction wire when the capsule endoscope is located at the esophageal initiation region, and for obtaining esophageal sampling data detected by the capsule endoscope between the cardia and the esophageal initiation region; and a scale recovery module for determining scale information based on the first feature information and the second feature information, and for determining esophageal state information relative to a world coordinate system based on the scale information and the esophageal sampling data.

[0024] As a further improvement of one embodiment of the present invention, the scale recovery module is further configured to update the device position information of the capsule endoscope according to the scale information, and update the sparse point cloud data according to the scale information; wherein, the device position information corresponds to the sampling process of the capsule endoscope, and the sparse point cloud data corresponds to the sampling process of the capsule endoscope; the esophageal information detection system further includes: a three-dimensional reconstruction module, configured to perform three-dimensional reconstruction according to the updated device position information and the updated sparse point cloud data, to determine a three-dimensional esophageal model relative to the world coordinate system, for displaying the current esophageal state; and a display module, configured to output and display the three-dimensional esophageal model.

[0025] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory storing a computer program executable on the processor, characterized in that the processor executes the program to perform the steps of the esophageal information detection method as described in any of the above technical solutions.

[0026] To achieve one of the above-mentioned objectives, one embodiment of the present invention provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it performs the steps of the esophageal information detection method as described in any of the above technical solutions.

[0027] Compared with existing technologies, the esophageal information detection method provided by this invention determines the feature information of the sampling device at the cardia and at the beginning of the esophagus. The attachments can show the positional relationship between the sampling device and the esophagus relative to the world coordinate system, and construct the relationship between the esophageal information collected by the sampling device and the world coordinate system. Therefore, based on the scale information determined by the feature information, the esophageal sampling data collected by the sampling device can be restored to the world coordinate system. The size, position, distance and other information contained in the esophageal sampling data can be restored to its actual size, position, distance and other information relative to the world coordinate system according to the "scale" defined by the scale information. Medical staff can use this actual information as intermediate data to ensure the accuracy of subsequent treatment, diagnosis, teaching and other work. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the esophageal information detection system according to one embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the structure of an electronic device according to one embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram of the steps of an esophageal information detection method according to an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram of the steps of an esophageal information detection method in one embodiment of the present invention.

[0032] Figure 5 This is a schematic diagram of the steps of the esophageal information detection method in another embodiment of the present invention.

[0033] Figure 6 This is a schematic diagram of step S40 in a specific embodiment of another embodiment of the present invention.

[0034] Figure 7 This is a schematic diagram of step S5 in another embodiment of the present invention.

[0035] Figure 8 This is a schematic diagram of step S2 in another embodiment of the present invention.

[0036] Figure 9 This is a schematic diagram of step M13 in a specific embodiment of another embodiment of the present invention.

[0037] Figure 10 This is a schematic diagram of step S2 in another specific embodiment of the present invention.

[0038] Figure 11 This is a schematic diagram of the steps of an esophageal information processing method according to an embodiment of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0040] It should be noted that the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] like Figure 1 As shown, one embodiment of the present invention provides an esophageal information detection system 100.

[0042] In one embodiment, the esophageal information detection system 100 determines the esophageal state information relative to the world coordinate system according to an esophageal information detection method, so as to obtain the absolute data of the internal position of the esophagus relative to the world coordinate system, and assist medical staff in judging the actual health status or the stage of disease development.

[0043] In one specific embodiment, the esophageal information detection method can be implemented with reference to any of the technical solutions provided below.

[0044] The esophageal information detection system 100 includes a capsule endoscope 11. The capsule endoscope 11 is used to detect and obtain esophageal sampling data.

[0045] In one embodiment, the esophageal sampling data may be a sampled image; the capsule endoscope 11 may include a camera or other image acquisition module. In other embodiments, the esophageal sampling data may also be three-dimensional point cloud data; the capsule endoscope 11 may include a lidar, a structured light sensor, a time-of-flight (ToF) camera, or a stereo vision system. The type of esophageal sampling data matches the structural configuration of the capsule endoscope 11.

[0046] The esophageal information detection system 100 includes a traction cable 12. The traction cable 12 includes a first end and a second end. The first end of the traction cable 12 is fixed to the capsule endoscope 11.

[0047] In one embodiment, the second end of the traction line 12 is a free end. In another embodiment, the second end of the traction line 12 may be fixed with a handle for adjusting the length and / or extension direction of the traction line 12.

[0048] The traction line 12 is used to show the positional relationship between the capsule endoscope 11 and the esophagus relative to the world coordinate system using its characteristic information.

[0049] In one embodiment, the traction wire 12 indicates the positional relationship between the capsule endoscope 11 and the esophagus with its second end not fixed to the capsule endoscope 11. In other embodiments, the traction wire 12 indicates the positional relationship between the capsule endoscope 11 and the esophagus with the length of the portion of the traction wire exposed outside the esophagus.

[0050] In one specific embodiment, the positional relationship is indicated by the position of the traction line 12 at the incisor region, showing the positional relationship between the capsule endoscope 11 and the esophagus. The position of the traction line 12 at the incisor region can be represented by the distance between the end of the traction line 12 and the incisor region, or by the length of the traction line 12 on a first side at the incisor region; the first side can be the side away from the cardia region.

[0051] In one embodiment, the second end of the traction line 12 indicates the positional relationship between the capsule endoscope 11 and the esophagus relative to the world coordinate system by its position at the incisor portion.

[0052] The esophageal detection system 100 includes a data acquisition module 13.

[0053] In one embodiment, the data acquisition module 13 is used to obtain data from one side of the capsule endoscope 11.

[0054] In one specific embodiment, the data acquisition module 13 is used to obtain at least a portion of the esophagus sampling data from the capsule endoscope 11.

[0055] In one specific example, the data acquisition module 13 is used to obtain esophageal sampling data detected by the capsule endoscope 11 between the cardia and the beginning of the esophagus.

[0056] In one embodiment, the data acquisition module 13 is used to obtain data from one side of the traction line 12.

[0057] In one specific embodiment, the data acquisition module 13 is used to obtain feature information shown by the traction line 12 when the capsule endoscope 11 is located at different parts of the esophagus.

[0058] In one specific example, the data acquisition module 13 is used to obtain first feature information shown by the traction line 12 when the capsule endoscope 11 is located at the cardia. In another specific example, the data acquisition module 13 is used to obtain second feature information shown by the traction line 12 when the capsule endoscope 11 is located at the beginning of the esophagus.

[0059] In other specific examples, the data acquisition module 13 can be used to obtain the feature information shown by the traction line 12 when the capsule endoscope 11 is located at other locations; specifically, it can be used to obtain the feature information shown by the traction line 12 when the capsule endoscope 11 is located at the detection start point and the detection end point.

[0060] The first feature information can be the position of the traction line 12 at the incisor region when the capsule endoscope 11 is located at the cardia. The second feature information can be the position of the traction line 12 at the incisor region when the capsule endoscope 11 is located at the beginning of the esophagus. Specifically, the position of the traction line 12 at the incisor region can be the length of the portion of the traction line 12 exposed outside the incisor region and the esophagus.

[0061] The esophageal detection system 100 includes a scale recovery module 15.

[0062] In one embodiment, the scale recovery module 15 is used to determine scale information based on the first feature information and the second feature information.

[0063] In one specific embodiment, the scale recovery module 15 can determine the scale information based on the position of the capsule endoscope 11 relative to the world coordinate system indicated by the feature information.

[0064] The scale information represents the scale relationship between the position of the feature information acquired by the capsule endoscope 11 relative to the world coordinate system and its position relative to the coordinates of the capsule endoscope 11 itself.

[0065] In one embodiment, the scale recovery module 15 is used to determine the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data.

[0066] The scale restoration module 15 is used to restore the esophageal sampling data collected by the capsule endoscope 11 to the world coordinate system based on the scale information and determine the esophageal state information accordingly. In this way, the generated esophageal state information can accurately record the actual dimensions of various parts of the esophagus, including the actual volume of lesions or other tissue parts, the actual distance between lesions or other parts, and other information, so that medical personnel can judge the actual state of the esophagus or the stage of disease development.

[0067] The esophageal state information can be a two-dimensional image of the entire esophagus. In this embodiment, the esophageal sampling data used to determine the esophageal state information can be a two-dimensional image of a local area of ​​the esophagus or other data information that can be used to determine the overall image.

[0068] The esophageal state information can be a three-dimensional model of the entire esophagus. In this embodiment, the esophageal sampling data used to determine the esophageal state information can be a two-dimensional image of a local part of the esophagus, point cloud data, or other data information that can be used to determine the three-dimensional model. In one specific embodiment, three-dimensional reconstruction can be performed based on the esophageal sampling data in the form of a two-dimensional image, and a three-dimensional model can be constructed based on the point cloud data after it is determined.

[0069] In one embodiment, the scale recovery module 15 is used to update the device position information of the capsule endoscope 11 based on the scale information. The device position information records the position of the capsule endoscope 11, which may specifically be position information determined based on the coordinate system of the capsule endoscope 11 itself.

[0070] The device location information corresponds to the sampling process of the capsule endoscope 11.

[0071] In one embodiment, the scale recovery module 15 is used to update the sparse point cloud data based on scale information. The sparse point cloud data is determined based on the esophageal sampling data, or the esophageal sampling data is the sparse point cloud data itself. The sparse point cloud data records the characteristics of tissue sites in the esophagus.

[0072] The sparse point cloud data corresponds to the sampling process of the capsule endoscope 11.

[0073] In one embodiment, the esophageal information detection system 100 further includes a three-dimensional reconstruction module 14.

[0074] The 3D reconstruction module 14 is used to perform 3D reconstruction, determine the 3D esophageal model as esophageal state information, and show the current esophageal state.

[0075] In one specific embodiment, 3D reconstruction can be achieved using the SFM (Structure from Motion) algorithm. Specifically, features of esophageal sampling images can be extracted using the SFM algorithm, feature point matching can be performed, the F matrix and the essential matrix can be calculated, and the camera motion trajectory can be obtained. Based on this, a sparse point cloud set can be reconstructed to construct a 3D esophageal model.

[0076] The 3D reconstruction module 14 is used to perform 3D reconstruction based on the updated device position information to determine the 3D esophageal model relative to the world coordinate system.

[0077] The 3D reconstruction module 14 is used to perform 3D reconstruction based on the updated sparse point cloud data to determine the 3D esophageal model relative to the world coordinate system.

[0078] The 3D esophageal model can be determined simultaneously based on the updated device location information and the updated sparse point cloud data. The sparse point cloud data can determine the features of the corresponding esophageal sampling image, and the device location information corresponding to the esophageal sampling image can determine the position of the sparse point cloud in the 3D model, thus constructing the 3D esophageal model.

[0079] In one embodiment, the three-dimensional reconstruction module 14 is further configured to perform distortion correction processing on the esophageal sampling image before performing three-dimensional reconstruction based on the esophageal sampling image.

[0080] In one specific embodiment, the three-dimensional reconstruction module 14 or other control device is used to control the capsule endoscope 11 to take pictures of the checkerboard pattern, obtain the checkerboard pattern image, and calibrate the camera parameters cameraParams of the capsule endoscope 11 accordingly. The esophageal sampling image is then subjected to distortion correction processing based on the camera parameters cameraParams.

[0081] In one embodiment, the esophageal information detection system 100 further includes a display module 16.

[0082] Display module 16 is used to output and display the three-dimensional esophageal model. Display module 16 can be a display screen or the like.

[0083] In one specific embodiment, the display module 16 can also output and display lesion information in the three-dimensional esophageal model. Specifically, it can highlight or change the color of the edge or the entire lesion in the three-dimensional esophageal model, and / or, in response to input calculation or evaluation instructions, determine and output information such as the location, area, and volume of the lesion.

[0084] In one embodiment, the 3D reconstruction module 14 outputs sparse point cloud data and the device pose information of the capsule endoscope 11. The scale recovery module 15 updates the sparse point cloud data and the device pose information based on scale information. The display module 16 processes the updated sparse point cloud data and device pose information to reconstruct a 3D esophageal model. In this embodiment, the 3D reconstruction module 14 reconstructs 3D data (i.e., sparse point cloud data) from a 2D image, and the display module 16 reconstructs a 3D model based on the 3D data.

[0085] Specifically, in this embodiment, the display module 16 is used to perform depth reconstruction, dense reconstruction, mesh reconstruction and texture reconstruction based on the updated sparse point cloud data and device pose information to obtain a three-dimensional esophageal model.

[0086] The esophageal information detection system 100 provided by the present invention may also include other functional configurations or functional modules. Specifically, the esophageal information detection method or esophageal information processing method provided later can be adapted for configuration, which will not be elaborated here.

[0087] The relationship between the modules and functions provided above in this invention is not absolutely fixed. In some embodiments, modules can be combined and integrated so that one module can perform the functions of multiple modules; in some embodiments, some functions can be separated and implemented by several additional modules.

[0088] For example, the data acquisition module 13, the scale recovery module 15, the three-dimensional reconstruction module 14, and the display module 16 can be integrated into a control module, which carries all or part of the functions of the above modules.

[0089] In one embodiment, the esophageal information detection system 100 may include only the aforementioned data acquisition module 12 and scale recovery module 15, using two modules with corresponding functions, or a total module integrating the functions of both modules, as the minimum selling unit. In other words, in this embodiment, the esophageal information detection system 100 may not include the capsule endoscope 11 and traction wire 12, and may only perform the function of data processing.

[0090] like Figure 2 As shown, one embodiment of the present invention provides an electronic device 200.

[0091] Electronic device 200 may specifically be a computer device, which may be a terminal device or a server.

[0092] The electronic device 200 includes at least one processor. The esophageal information detection method provided by the present invention can be applied to or implemented by the processor. Specifically, the processor may be a central processing unit (CPU) 21.

[0093] Electronic device 200 includes a memory. The memory is used to store various types of data to support the operation of electronic device 200. Examples of such data include any computer program used to operate on a computer device. The memory may be a read-only memory (ROM) 22, a random access memory (RAM) 23, or other storage portion 28. The storage portion 28 may be located within or outside electronic device 200.

[0094] In one embodiment, when the processor executes the computer program stored in the memory, the steps of the esophageal information detection method of any technical solution of the present invention are performed.

[0095] In another embodiment, when the processor executes the computer program stored in the memory, the steps of the esophageal information processing method of any technical solution of the present invention are performed.

[0096] In one embodiment, the electronic device 200 includes a central processing unit 21, which can perform various appropriate actions and processes based on a program stored in a read-only memory 22 or a program loaded from a storage section 28 into a random access memory 23. The random access memory 23 also stores various programs and data required for system operation. The central processing unit 21, the read-only memory 22, and the random access memory 23 are interconnected via a bus 24. An input / output interface (I / O interface) 25 is also connected to the bus 24.

[0097] The following components are connected to the input / output interface 25: an input section 26 including a keyboard, mouse, etc.; an output section 27 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 28 including a hard disk, etc.; and a communication section 29 including a network interface card such as a local area network card, modem, etc. The communication section 29 performs communication processing via a network such as the Internet. A drive 210 is also connected to the input / output interface 25 as needed. A removable medium 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 210 as needed so that computer programs read from it can be installed into the storage section 28 as needed.

[0098] One embodiment of the present invention provides a computer-readable storage medium.

[0099] In one embodiment, a computer-readable storage medium stores a computer program executed by the processor mentioned above, or the esophageal information detection method in any of the following technical solutions.

[0100] In another embodiment, the computer-readable storage medium stores a computer program executed by the processor mentioned above, or the esophageal information processing method in any of the following technical solutions.

[0101] When the processor executes the computer program, it can perform the descriptions of the esophageal information detection method or esophageal information processing method in any of the following technical solutions; therefore, they will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0102] The computer-readable storage medium may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0103] like Figure 3 As shown, one embodiment of the present invention provides a method for detecting esophageal information.

[0104] The esophageal information detection method is applied to an esophageal information detection system.

[0105] In one embodiment, the esophageal information detection system can be as follows: Figure 1 The configuration described above is applied, and the corresponding technical solutions are referenced in the detection method provided by this invention. Specifically, in this embodiment, using... Figure 1 The capsule endoscope 11 shown serves as a sampling device, as described later. Figure 1The traction line 12 shown is attached as an appendix below.

[0106] Of course, the esophageal detection system used in the esophageal detection method provided by the present invention is not limited to this structural configuration. The sampling device mentioned later can be any device capable of collecting esophageal information, and the appendix mentioned later can be any component capable of showing the positional relationship between the sampling device and the esophagus.

[0107] In this embodiment, the esophageal information detection system includes a sampling device. The sampling device is used to collect esophageal information. In one embodiment, the esophageal information includes information characterizing the current condition of tissue sites within the esophagus, specifically including information characterizing the current condition of the cervical esophagus, the thoracic esophagus, and the abdominal esophagus. In another embodiment, the esophageal information includes information characterizing the current condition of lesions within the esophagus.

[0108] The esophageal information is represented as esophageal sampling data. This esophageal sampling data can be in the form of numerical values, images, 3D point clouds, etc. The structure of the sampling device is configured based on the form of the esophageal sampling data; for example, when the esophageal sampling data is an image, the sampling device may include a camera for detecting and generating the image.

[0109] The esophageal information detection system also includes an accessory. The accessory is associated with the sampling device; the association can be a physical structural association, for example, at least part of the accessory is fixed to the sampling device or always remains relatively stationary with the sampling device; the association can also be an informational coupling, for example, when the sampling device is activated, the feature information on one side of the accessory also changes.

[0110] When the sampling device collects esophageal information, the attachment uses its characteristic information to show the positional relationship between the sampling device and the esophagus relative to the world coordinate system.

[0111] In embodiments that establish a physical structural association between the accessory and the sampling device, the accessory indicates the positional relationship between the sampling device and the esophagus through its physical characteristics. For example, when the sampling device moves relative to the esophagus, the accessory indicates the positional change of the sampling device through features such as its exposed length and distance from the esophagus or other fixed locations.

[0112] In embodiments where information coupling is established between the accessory and the sampling device, the accessory indicates the positional relationship between the sampling device and the esophagus through information determined by the coupling. For example, when the sampling device moves relative to the esophagus, the accessory obtains information indicating the positional change of the sampling device through information exchange with the sampling device and / or the esophagus.

[0113] In one embodiment, the sampling device is capsule-shaped. In a specific embodiment, the sampling device is a capsule endoscope.

[0114] In one embodiment, the accessory includes a traction line. In a specific embodiment, one end of the traction line is fixed to the sampling device. In a specific embodiment, the traction line, at its other end, indicates the positional relationship between the sampling device and the esophagus.

[0115] In one specific embodiment, the traction line, through its position at the incisor region, illustrates the positional relationship between the sampling device and the esophagus relative to the world coordinate system.

[0116] In one specific embodiment, the feature information of the accessory is the position of the other end of the traction line at the incisor portion.

[0117] like Figure 3 As shown, an esophageal information detection method provided in one embodiment of the present invention includes the following steps.

[0118] Step S1: Obtain the first feature information shown by the accessory when the sampling device is located at the cardia.

[0119] Step S2: Obtain esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus.

[0120] Step S3: Obtain the second feature information shown by the attachment when the sampling device is located at the beginning of the esophagus.

[0121] Step S4: Determine scale information based on the first feature information and the second feature information.

[0122] Step S5: Determine the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data.

[0123] In this way, at least the characteristic information corresponding to the sampling device at the cardia and the beginning of the esophagus can be determined, forming a positional representation of the sampling device relative to the world coordinate system at these two locations. This positional representation can then be used to construct scale information that maps the sampling device's own coordinate system to the world coordinate system. When precise determination of the esophageal condition is required, the esophageal sampling data can be reconstructed using this scale information to form esophageal state information in the world coordinate system. Therefore, medical personnel can accurately determine the dimensions of various parts or lesions within the esophagus based on this esophageal state information.

[0124] The first feature information corresponds to the cardia (near the end of the esophagus), and the second feature information corresponds to the beginning of the esophagus. When the esophageal information detection method provided by this invention is applied to other scenarios to achieve scale-recoverable state information detection, the first feature information may specifically be the feature information shown in the appendix when the sampling device is located at the end of the detection object, and the second feature information may specifically be the feature information shown in the appendix when the sampling device is located at the beginning of the detection object.

[0125] The steps described above do not necessarily have a specific order indicated by their numbers. For example, step S2 may precede step S1, follow step S3, or follow step S4.

[0126] In the first embodiment, the sampling device can be placed at the cardia to obtain first feature information (state one), and then the sampling device can be moved from the cardia to the beginning of the esophagus to obtain esophageal sampling data between the two locations, until the sampling device is located at the beginning of the esophagus to obtain second feature information (state two).

[0127] In the second embodiment, the sampling device can be placed at the beginning of the esophagus to obtain the second feature information (state two), and then the sampling device can be moved from the beginning of the esophagus to the cardia to obtain esophageal sampling data between the two parts, until the sampling device is located at the cardia to obtain the first feature information (state one).

[0128] The esophageal sampling data includes sampling data from the cardia, sampling data from the beginning of the esophagus, and sampling data from at least one location between the cardia and the beginning of the esophagus.

[0129] In one embodiment, the sampling device collects esophageal information at a frequency of n frames per second to obtain esophageal sampling data. In a specific embodiment, the sampling device is used to collect image information of various parts within the esophagus to obtain esophageal sampling images, and several esophageal sampling images constitute an image dataset. .

[0130] In one embodiment, an index value corresponding to the esophageal sampling image is also obtained simultaneously. The index value is used to indicate the sequential order of the esophageal sampling images.

[0131] In one embodiment, the sampling time corresponding to the esophageal sampling image is also obtained at the same time as the esophageal sampling image.

[0132] In one specific embodiment, the index value is defined as , No. Zhang's esophageal sampling image is , No. The sampling time for the esophageal sampling image is The total number of esophageal sampling images is The image dataset is then represented as: .

[0133] Among them, the total number of esophageal sampling images It is determined based on the sampling frequency n and the total sampling time.

[0134] In one specific example, the first esophageal sampling image in the image dataset is a sampling image taken by the sampling device at the cardia, and the last esophageal sampling image in the image dataset is a sampling image taken by the sampling device at the beginning of the esophagus.

[0135] In one specific example, the index value of the sampled image at the cardia region is... ,satisfy The index value of the sampled image at the beginning of the esophagus is... ,satisfy .

[0136] In one embodiment, the process corresponding to state one may involve controlling the sampling device to continuously collect data until the esophageal sampling data indicates that the sampling device is located at the cardia. Then, the sampling device is controlled to maintain its position, and the esophageal sampling image and index value at the current position are sampled and obtained. The feature information currently shown in the appendix is ​​obtained as the first feature information.

[0137] In the first embodiment described above, the data collected during this process can be discarded; in the second embodiment described above, the data collected during this process is retained as esophageal sampling data.

[0138] In one embodiment, the process corresponding to state two may involve controlling the sampling device to continuously collect data until the esophageal sampling data indicates that the sampling device is located at the beginning of the esophagus. Then, the sampling device is controlled to maintain its position, and the esophageal sampling image and index value at the current position are sampled and obtained. The feature information currently shown in the appendix is ​​then obtained as the second feature information.

[0139] In the first embodiment described above, the data collected during this process is retained as esophageal sampling data; in the first embodiment described above, the data collected during this process can be discarded.

[0140] In one embodiment, the index value is defined as It can also be based on feature information esophageal sampling images Sampling time Building an information dataset , can be represented as: .

[0141] The first feature information is The second feature information is .

[0142] In one embodiment, esophageal state information is determined based on scale information. Specifically, the esophageal sampling data is mapped using the scale information as a proportion, and the esophageal state information is determined based on the data generated after mapping.

[0143] In one embodiment, the attachment feature information (e.g., the first feature information and the second feature information) includes the positional relationship between the first attachment portion and the first reference point. The first attachment portion refers to at least one portion on the attachment.

[0144] The first reference point is stationary relative to the esophagus. The first reference point can be a point at a certain location on the esophagus, or it can be another point outside the esophagus that is stationary relative to the esophagus.

[0145] The relative position of the first attachment portion and the first reference point changes with the position of the sampling device; specifically, the relative positional relationship between the two changes with the relative position of the sampling device and the esophagus. Based on this, the attachment, through the relative position of its first attachment portion and the first reference point, illustrates the positional relationship of the sampling device and the esophagus relative to the world coordinate system.

[0146] In one embodiment, the sampling device moves a first distance within the esophagus relative to a world coordinate system, and the relative position of the first accessory portion and the first reference point changes by a second distance. The first distance is equal to the second distance, or there is a fixed relationship between the first distance and the second distance; this fixed relationship can be a known or preset proportional relationship or difference relationship.

[0147] In one embodiment, the first reference point is located at the incisor region. The attachment, by its position at the incisor region, indicates the positional relationship between the sampling device and the esophagus relative to the world coordinate system. In a specific embodiment, one end of the attachment is fixed to the sampling device, the first attachment portion is the other end of the attachment, and the distance between the other end of the attachment not fixed to the sampling device and the incisor region indicates the positional relationship between the sampling device and the esophagus relative to the world coordinate system. In a specific embodiment, the first attachment portion is the part of the attachment exposed outside the incisor region and the esophagus, and the length or other dimensions of the portion of the attachment exposed outside the incisor region indicate the positional relationship between the sampling device and the esophagus relative to the world coordinate system.

[0148] For example, in an embodiment where the attachment is linear and one end is fixed to the sampling device, if the capsule endoscope enters the middle of the esophagus from the incisor region and the beginning of the esophagus, and if the attachment feature information is defined as the length of the part exposed outside the incisor region or the distance between the free end and the incisor region, then in the world coordinate system, the attachment feature information is directly proportional to the distance between the sampling device and the cardia region, inversely proportional to the distance between the sampling device and the beginning of the esophagus, and inversely proportional to the depth of the sampling device in the esophagus.

[0149] In one embodiment, the first feature information includes the position information of the attachment at the incisor region when the sampling device is located at the cardia region. The first attachment position information can be expressed by distance or length.

[0150] In one embodiment, the second feature information includes the position information of the second accessory at the incisor region when the sampling device is located at the beginning of the esophagus. The position information of the second accessory can be expressed by distance or length.

[0151] like Figure 4 As shown, in one embodiment, the esophageal information detection method provided by the present invention includes the following steps.

[0152] Step S1: Obtain the first feature information shown by the accessory when the sampling device is located at the cardia.

[0153] Step S2: Obtain esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus.

[0154] Step S3: Obtain the second feature information shown by the attachment when the sampling device is located at the beginning of the esophagus.

[0155] Step S41: Determine the scale information based on the difference between the first attachment location information and the second attachment location information.

[0156] Step S5: Determine the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data.

[0157] Step S41 can be interpreted as a variation of step S4, or as included in step S4.

[0158] Esophageal sampling data is used to describe the sampling action of the sampling device from the cardia to the beginning of the esophagus in its own device coordinate system. Therefore, the difference between the first feature information and the second feature information is used to describe the positional change of the sampling device from the cardia to the beginning of the esophagus in the world coordinate system to construct scale information, which can correspond with the esophageal sampling data, so as to achieve the effect of recovering the scale of the entire esophageal sampling data with only two feature information.

[0159] When the location information of the first attachment is expressed as length or distance and... The location information in the second attachment is expressed as length or distance and is indicated by... When representing the location information of the two attachments, the difference can specifically be the difference between the two, that is, based on the difference. Or its absolute value determines the scale information.

[0160] like Figure 5 As shown, in one embodiment, the esophageal information detection method provided by the present invention includes the following steps.

[0161] Step S1: Obtain the first feature information shown by the accessory when the sampling device is located at the cardia.

[0162] Step S2: Obtain esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus.

[0163] Step S3: Obtain the second feature information shown by the attachment when the sampling device is located at the beginning of the esophagus.

[0164] Step S40: Determine the scale information based on the difference between the first attachment location information and the second attachment location information.

[0165] Step S42: Determine the scale information based on the difference between the first feature information and the second feature information, as well as the difference in the device position information.

[0166] Step S5: Determine the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data.

[0167] Thus, the difference in device location information can determine the positional difference between the two parts in the device coordinate system, and the difference in feature information can determine the positional difference between the two parts in the world coordinate system. The scale information constructed thereby can correlate the positional differences in the two coordinate systems to realize the mapping of esophageal sampling data between the two coordinate systems, so as to confirm the current state of the esophagus.

[0168] If the difference in device location information is defined as The difference between the first feature information and the second feature information is defined as follows: Then the scale information It can satisfy: .

[0169] This helps restore the scale of esophageal sampling information to the world coordinate system.

[0170] Figure 5The dashed lines in the diagram illustrate the sequential relationship between steps S40 and S42 and other steps, indicating that the positions of steps S40 and S42 can be adjusted. Step S40 discloses obtaining the difference in device position information at two locations and can be placed anywhere before step S42, such as between steps S2 and S3. Step S42 discloses determining scale information based on the two differences and can therefore be placed anywhere after step S40 and before step S5.

[0171] In one embodiment, the difference in device location information disclosed in step S40 is determined based on esophageal sampling data, so step S40 can be set at any position after step S2 and before step S42.

[0172] like Figure 6 As shown, in one specific embodiment, step S40 of the present invention may include the following steps.

[0173] Step S401: Obtain the first sampling image of the cardia area by the sampling device.

[0174] Step S402: Determine the first device position information of the sampling device based on the first sampled image.

[0175] Step S403: Obtain a second sampling image of the sampling device at the beginning of the esophagus.

[0176] Step S404: Determine the second device position information of the sampling device based on the second sampled image.

[0177] Step S405: Based on the first device position information and the second device position information, determine the moving distance of the sampling device to represent the difference in the device position information.

[0178] In this way, the current position of the device can be determined based on the sampled image, and the position information of the device at the beginning and end (cardiac and esophageal initiation) can be determined. The difference in device position information is constructed, simplifying the steps of determining the difference without the need to introduce other devices or methods.

[0179] In an embodiment where the esophageal sampling data is an esophageal sampling image, the first sampling image and the second sampling image may be included in the esophageal sampling data. The esophageal sampling data includes a first sampling image of the cardia, several other sampling images between the cardia and the beginning of the esophagus, and a second sampling image of the beginning of the esophagus. It features an extremely simplified operation method and computational logic, thereby improving operational and computational efficiency.

[0180] In this embodiment, steps S401 and S403 may be included in step S2, and are not necessarily set after step S2. Similarly, in this embodiment, steps S402 and S404 may be set in or after step S2, and step S405 may be set in or after step S2, and before step S42.

[0181] The order of steps S401 to S404 can be adjusted. In other specific embodiments, steps S401 and S403 can be executed first, followed by steps S402 and S404. Simply place step S402 after step S401 and step S404 after step S403.

[0182] In one specific example, the device location information is in the form of coordinates, based on which the distance the sampling device moves can be the Euclidean distance between the coordinates indicating the location information of the two devices.

[0183] For example, esophageal sampling data may be in the form of an image dataset, where the first sampled image from the cardia is the first image in the dataset. In this case, the first sampled image is defined as... The location information of the first device determined accordingly is ,in, The sampling device is located at the cardia in three dimensions; in one embodiment, the position information of the first device may be included in the pose information of the first device, and the pose information of the first device is... ,in, The first device attitude information includes the pitch angle information when the sampling device is located at the cardia. Yaw angle information and roll angle information .

[0184] For example, if the second sampled image of the esophageal initiation is the last image in the image dataset (e.g., the Mth image), then the second sampled image is defined as... The location information of the second device determined accordingly is ,in, The sampling device is positioned at the beginning of the esophagus, providing its three-dimensional coordinates. In one embodiment, the position information of the second device may be included in the pose information of the second device, which is: ,in, The second device's attitude information includes the pitch angle information when the sampling device is located at the beginning of the esophagus. Yaw angle information and roll angle information .

[0185] The sampling device movement distance used to represent the difference in device position information can be determined by calculating the Euclidean distance between the first device position information and the second device position information. At this time, the difference in device position information... satisfy: .

[0186] Thus, in the preferred embodiment, the device location information differs. Describe the relative movement distance of the sampling device between the cardia and the beginning of the esophagus based on its own coordinate system, and the difference between the first feature information and the second feature information. The description describes the actual distance the sampling device moves between the cardia and the beginning of the esophagus, based on the world coordinate system.

[0187] In one embodiment, the esophageal sampling data may be device location information and sparse point cloud data, or data such as esophageal sampling images that can be analyzed to determine the device location information and sparse point cloud data.

[0188] like Figure 7 As shown, in one embodiment, step S5 of the present invention may include the following steps.

[0189] Step S51: Update the device position information according to the scale information; the device position information corresponds to the sampling process of the sampling device.

[0190] Step S52: Update the sparse point cloud data according to the scale information; the sparse point cloud data corresponds to the sampling process of the sampling device.

[0191] Step S53: Based on the updated device location information and the updated sparse point cloud data, perform three-dimensional reconstruction to determine the three-dimensional esophageal information relative to the world coordinate system, which is used to show the current esophageal state.

[0192] In this way, by utilizing scale information, device location information and sparse point cloud data can be unified in scale and mapped to the world coordinate system. Based on this, three-dimensional reconstruction can be performed, directly determining the esophageal state information relative to the world coordinate system to provide a clear and accurate illustration. In this embodiment, the esophageal state information includes three-dimensional esophageal information, which can be in the form of a three-dimensional model, allowing medical personnel to use it as intermediate data. Based on the advantage of being able to restore the actual size of the esophagus and lesions, diagnosis and treatment can be implemented.

[0193] The device location information corresponds to the sampling process of the sampling device. Specifically, the device location information corresponds to the esophageal sampling data obtained by the sampling device during the sampling process. For example, each esophageal sampling image corresponds to a set of device location information.

[0194] The device location information can have three-dimensional coordinates. Based on this, the updated device location information can satisfy the following: .

[0195] In an embodiment where device position information is included within device pose information, on one hand, each esophageal sampling image corresponds to a set of device pose information; the device pose information includes device position information and device orientation information. On the other hand, the device pose information may have a six-dimensional array. The form corresponds to the device pose information of all esophageal sampling data to form a pose dataset. Based on the above scale update, the updated pose dataset satisfies: .

[0196] The sparse point cloud data corresponds to the sampling process of the sampling device. Specifically, the sparse point cloud data corresponds to the esophageal sampling data obtained by the sampling device during the sampling process. For example, each esophageal sampling image corresponds to a set of sparse point cloud data.

[0197] The location of sparse point clouds in the sparse point cloud data can have three-dimensional coordinates. Based on this, the location of the updated sparse point cloud can satisfy the following form: .

[0198] in, This represents the total number of sparse point clouds used to generate esophageal state information.

[0199] In one embodiment, the total number of sparse point clouds obtained by parsing esophageal sampling data is In one specific embodiment, the sparse point cloud data obtained from esophageal sampling data parsing is filtered based on error to obtain sparse point cloud data for esophageal state information, thus resulting in... .

[0200] A point cloud dataset is formed from sparse point cloud data corresponding to all esophageal sampling data. Based on the above scale update, the updated point cloud dataset satisfies: .

[0201] In one specific embodiment, the device location information and the sparse point cloud data correspond to each other. This correspondence is used to construct a three-dimensional esophageal model. In another specific embodiment, both the device location information and the sparse point cloud data are generated based on esophageal sampling data, or the esophageal sampling data includes both device location information and sparse point cloud data.

[0202] In one specific embodiment, the 3D reconstruction includes depth reconstruction. Specifically, this includes obtaining a depth map based on updated location information and updated sparse point cloud data. This can be achieved through multi-view stereo matching; in other embodiments, it can also be achieved using techniques such as structured light or time-of-flight. When implementing these other techniques, the structural configuration within the sampling device can be adjusted accordingly.

[0203] In one specific embodiment, the 3D reconstruction includes dense reconstruction. Specifically, this includes interpolating sparse point cloud data to determine dense point cloud data. The interpolation method can be Poisson reconstruction, a smoothing algorithm, or interpolation based on neighborhood information, etc.

[0204] In one specific embodiment, the 3D reconstruction includes point cloud fusion. Specifically, this includes: registering overlapping regions between different point clouds to determine a global point cloud representation.

[0205] In one specific embodiment, the 3D reconstruction includes initial mesh reconstruction. Specifically, this includes converting point cloud data into a surface model in the form of a triangular mesh. This can be achieved through Delaunay triangulation or moving least squares methods, etc. The initial mesh reconstruction is included in the overall mesh reconstruction process.

[0206] In one specific embodiment, the 3D reconstruction includes mesh optimization. Specifically, it includes: based on the smoothness or shape of the surface model determined during initialization, improving the topology and optimizing the shape of the resulting surface model. The improvement can be aimed at reducing the number of unnecessary triangles, optimizing the shape, etc. The mesh optimization is included in the overall mesh reconstruction process.

[0207] In one specific embodiment, the 3D reconstruction includes texture reconstruction. Specifically, this includes projecting esophageal sampling data (raw data, specifically, the original esophageal sampling image) onto a surface model. This includes steps such as texture coordinate calculation and texture mapping.

[0208] In this embodiment, the input to the 3D reconstruction step includes not only the updated device location information and sparse point cloud data, but also esophageal sampling images.

[0209] In one specific example, the 3D reconstruction includes: depth reconstruction, density reconstruction, point cloud fusion, mesh reconstruction, and texture reconstruction. Preferably, the above steps are performed sequentially.

[0210] In one embodiment, the esophageal sampling data directly includes sparse point cloud data and / or device location information. In a specific embodiment, the sparse point cloud data and device location information are determined based on the esophageal sampling image; therefore, steps S20, M11 to M13 provided below are all included in step S2.

[0211] In one embodiment, the esophageal sampling data is in image form (i.e., esophageal sampling image), and the aforementioned sparse point cloud data and device location information are obtained based on the parsing of the esophageal sampling image. In this embodiment, step S20, provided below, is included in step S2, and steps M11 to M13 are set at any position between steps S2 and S5.

[0212] The following description, using esophageal sampling data that directly includes sparse point cloud data and / or device location information, will be elaborated upon as an example. Figure 8 As shown, this does not mean that the invention excludes the other embodiment described above.

[0213] like Figure 8 As shown, in this embodiment, step S2 of the present invention may include the following steps.

[0214] Step S20: Obtain several esophageal sampling images detected by the sampling device between the cardia and the beginning of the esophagus.

[0215] Step M11: Calculate feature point descriptors based on the esophageal sampled images.

[0216] Step M12: Determine the initial pose information of the sampling device corresponding to the esophageal sampling image based on the matching of feature point descriptors between esophageal sampling images.

[0217] Step M13: Based on the feature point descriptors corresponding to the esophageal sampling image and the initial pose information, perform global optimization to obtain sparse point cloud data and device pose information; the device pose information includes device position information and device attitude information.

[0218] In this way, complex three-dimensional features can be determined based on simple image data, simplifying the sampling process and accelerating the overall process of esophageal state information detection.

[0219] The above process describes a method for recovering the motion trajectory of a sampling device based on image features, which can be implemented using the SFM algorithm or other algorithms. In an embodiment using the SFM algorithm to obtain the motion trajectory, after performing feature point descriptor matching, steps such as calculating the F matrix and the essential matrix are also included.

[0220] Step M11 may specifically include determining feature points of the esophageal sampling image and further determining feature point descriptors based on the feature points.

[0221] In one specific embodiment, the feature point is a SURF (Speeded-Up Robust Features) feature point, and the feature point descriptor is a SURF feature point descriptor. Of course, in other specific embodiments, the feature point descriptor can also be a SIFT (Scale-Invariant Feature Transform) feature point descriptor, an ORB (Oriented Fast and Rotated BRIEF) feature point descriptor, an AKAZE (Accelerated-KAZE) feature point descriptor, or a BRISK (Binary Robust Invariant Scalable Keypoints) feature point descriptor.

[0222] In one specific embodiment, step M11 includes: extracting the corresponding to the first... Zhang's esophageal sampling image The feature points, corresponding to the first Feature point set of Zhang's esophageal sampling image .in, Indicates the corresponding to the first Number of feature points in the esophageal sampling image Indicates the corresponding to the first The first esophageal sampling image The location of each feature point in the image.

[0223] The step of extracting feature points can be performed directly on the original esophageal sampling image, or the original esophageal sampling image can be processed first to obtain a processed esophageal sampling image, and then feature point descriptors can be calculated based on this.

[0224] In one specific embodiment, step M11 includes: calculating the value corresponding to the first... Zhang's esophageal sampling image The descriptor of the i-th feature point is obtained, corresponding to the i-th feature point. Feature point descriptor subset of Zhang's esophageal sampling image In this specific embodiment, each feature point descriptor is represented by a 64-dimensional vector. Indicates the corresponding number The first esophageal sampling image A descriptor for a feature point; in other specific embodiments, the form of the feature point descriptor can be adjusted, thereby producing other forms of feature point descriptor sets.

[0225] In one specific embodiment, the matching of feature point descriptors between esophageal sampling images specifically includes the matching of feature point descriptors of the current esophageal sampling image with the feature point descriptors of the previous esophageal sampling image. In this way, the matching of feature point descriptors can be determined step by step to determine the pose information of the sampling device.

[0226] In this specific embodiment, the current esophageal sampling image and the previous esophageal sampling image are considered as a pair of images. Of course, in other specific embodiments, feature point descriptor matching can be performed at intervals of several esophageal sampling images, in which case the image pairs have different definitions.

[0227] For ease of description, the following text will use the current esophageal sampling image and the previous esophageal sampling image as a pair of images as examples.

[0228] In one specific example, step M12 includes: performing feature matching on the feature point descriptors to determine feature point matching pairs (indexpairs). Feature matching specifically includes determining which indexpairs belong to the first index pair. Features of esophageal sampling images were analyzed to identify their characteristics in the first... Nearest neighbor (nearest neighbor in a subspace) features in a sampled image of the esophagus. For example, the nearest neighbor can be efficiently computed using the kd number.

[0229] In one specific example, step M12 includes: retrieving corresponding feature points based on feature point matching pairs indexpairs, estimating the difference between the field of view of the current esophageal sampling image and the field of view of the previous esophageal sampling image, and determining the relative position information relativeLoc and relative pose information relativeOri between the field of view of the current esophageal sampling image and the field of view of the previous esophageal sampling image.

[0230] In one specific example, step M12 includes: determining the actual position information orientation and actual posture information location of the current esophageal sampling image based on the actual position information prevLoc and actual orientation information prevOri of the previous esophageal sampling image, as well as the relative position information relativeLoc and relative orientation information relativeOri.

[0231] The actual position and attitude information mentioned above can be relative to the world coordinate system or the device coordinate system. Since the relevant information corresponding to each esophageal sampling image is determined based on the relevant information corresponding to the previous esophageal sampling image, the relevant information corresponding to the first esophageal sampling image can be set by preset, initialized to a default value, or directly represented by its corresponding attachment feature information.

[0232] For example, the actual orientation information of the current esophageal sampling image satisfies: orientation = relativeOri * prevOri.

[0233] For example, the actual pose information (location) of the current esophageal sampling image satisfies: location = prevLoc + relativeLoc * prevOri.

[0234] Step M13, through global optimization, maintains the uniformity of the predicted sampling device location information, facilitating the filtering of obviously unreasonable sparse feature points and their corresponding device location information, thereby improving data accuracy. Specifically, global optimization includes steps such as adjusting device location information, adjusting device attitude information, adjusting the position of the sparse point cloud, and adjusting the positional error of the same feature point in different image pairs / feature point matching pairs.

[0235] like Figure 9 As shown, in one specific embodiment, step M13 of the present invention may include the following steps.

[0236] Step M131: Determine the sparse point cloud data and the reprojection error of the feature points based on the feature point descriptor and the initial pose information.

[0237] Step M132 involves performing bundled adjustment and optimization with the goal of minimizing the reprojection error, adjusting and determining the sparse point cloud data and device pose information.

[0238] The reprojection error represents the positional error of the same feature point in different image pairs / feature point matching pairs. The bundle adjustment optimization is used to minimize the reprojection error; the bundle adjustment optimization involves a nonlinear least squares problem, which can be solved by iterative optimization algorithms such as the Levenberg-Marquardt algorithm.

[0239] Based on step M13, the device pose data in the form of a pose dataset corresponding to the M images can finally be determined. .

[0240] Based on step M13, the sparse point cloud data in the form of a point cloud dataset corresponding to the M images and the reprojection error can be determined. Combinations: .

[0241] in, This refers to the total number of sparse point cloud data sets in the point cloud dataset at this stage, determined based on the actual matching of feature points, and is numerically less than or equal to... .

[0242] In one specific example, the adjustment and determination of sparse point cloud data and device pose information in step M132 further includes: adjusting the sparse point cloud data according to an acceptable reprojection error threshold. The feature points with reprojection errors less than an acceptable threshold (reprojectionErrors) are selected and retained to determine the final sparse point cloud data. .

[0243] in, This is the total number of sparse point cloud data sets in the point cloud dataset at this stage, based on the feature point reprojection error. Determined by comparing with the acceptable threshold reprojectionErrors, numerically less than or equal to The acceptable threshold for reprojectionErrors is determined based on actual detection requirements, especially the accuracy requirements.

[0244] like Figure 10 As shown, in a specific embodiment of this invention, step S2 may include the following steps.

[0245] Step S20: Obtain several esophageal sampling images detected by the sampling device between the cardia and the beginning of the esophagus.

[0246] Step M01: Obtain the camera parameters of the sampling device.

[0247] Step M02: Perform distortion correction processing on the esophageal sampled image according to the camera parameters.

[0248] Step M11': Calculate feature point descriptors based on the esophageal sampled image after distortion correction.

[0249] Step M12: Determine the initial pose information of the sampling device corresponding to the esophageal sampling image based on the matching of feature point descriptors between esophageal sampling images.

[0250] Step M13: Based on the feature point descriptors corresponding to the esophageal sampling image and the initial pose information, perform global optimization to obtain sparse point cloud data and device pose information; the device pose information includes device position information and device attitude information.

[0251] The step M11' is included in the aforementioned step M11.

[0252] This can improve the accuracy of detection methods, especially making the processes of feature point extraction and pose information determination more precise.

[0253] In one specific example, the camera parameters cameraParams include focal length, image size, radial distortion, tangential distortion, image center point, etc.

[0254] In one specific example, the camera parameters cameraParams can be determined by controlling the sampling device to capture m0 images from different angles of a black and white checkerboard image, and then calibrating them. Where 20 ≤ m0 ≤ 30.

[0255] In one specific example, the camera configured in the sampling device is a fisheye camera.

[0256] After performing step M02, the distortion-corrected image dataset can be obtained: .

[0257] in, Indicates by Distortion-corrected esophageal sampling images Thus, step M11' yields the processed esophageal sampling image. For each object, compute feature point descriptors. For example, compute SURF feature point descriptors.

[0258] like Figure 11 As shown, one embodiment of the present invention provides an esophageal information processing method, including the following steps.

[0259] Step P1: Determine the esophageal status information according to the esophageal information detection method.

[0260] Step P2: Obtain sparse point cloud data corresponding to the lesion from the esophageal status information.

[0261] Step P3: Based on the sparse point cloud data of the lesion, use triangular patch segmentation to determine the area of ​​several triangular patches corresponding to the lesion.

[0262] Step P4: Determine the total area of ​​the lesion based on the area of ​​the triangular facet.

[0263] In this way, the actual area of ​​the lesion can be determined based on the sparse point cloud data in the esophageal status information, reflecting the size of the lesion, and serving as intermediate information to reflect the current development status of the lesion for reference by medical staff.

[0264] The esophageal information detection method in step P1 can be any of the esophageal information detection methods provided by the present invention, thereby obtaining esophageal state information relative to the world coordinate system, so that the total area of ​​the lesion determined accordingly is also the absolute area relative to the world coordinate system, which facilitates medical staff to assess the actual size of the lesion.

[0265] In one embodiment, the esophageal state information is in the form of a three-dimensional esophageal model.

[0266] In one embodiment, the esophageal state information includes sparse point cloud data.

[0267] Step P2 is based on the premise that the esophageal state information indicates the presence of a lesion in the corresponding esophagus. The esophageal state information includes at least sparse point cloud data corresponding to the lesion.

[0268] In one embodiment, before step P2, a step of determining whether there is a lesion within the esophageal state information is included. When it is determined that there is a lesion within the esophagus indicated by the esophageal state information, the edge points of the lesion are screened and determined to obtain a set of lesion boundary information. .

[0269] in, This represents the total number of lesion boundary points. Indicates the first The three-dimensional coordinates of the boundary points.

[0270] Then, based on the set of lesion boundary information, all feature points enclosed by the lesion boundary formed by the edge points of the lesion are determined, and the sparse point cloud data corresponding to the lesion is fixed: .

[0271] in, This indicates the total number of points in the lesion area. Indicates the first The three-dimensional coordinates of each lesion point.

[0272] This process may also include steps to highlight relevant feature points. For example, after determining the edge points of the lesion, the boundary of the lesion can be highlighted by changing its color or other means. Similarly, after determining feature points within the lesion area, the entire lesion can be highlighted by changing its color or other means.

[0273] Step P3 calculates the total area of ​​the three-dimensional lesion by dividing it into triangular facets and calculating the area of ​​each facet. After triangular facet division, a set of triangular facets is formed corresponding to the lesion. .

[0274] in, This indicates the total number of triangular facets. Indicates the first The numbering of the three vertices of each triangular facet.

[0275] For example, lesion point cloud dataset midpoint If a triangle is formed, then the triangle dataset is... In the middle, there is point (1, 3, 6).

[0276] Definition point Having three-dimensional coordinates ,point Having three-dimensional coordinates ,point Having three-dimensional coordinates Then the first The area of ​​each triangular facet is: .

[0277] in, ; ; in, This indicates a cross product.

[0278] For example: .

[0279] in, This indicates the search for the L2 norm.

[0280] For example, .

[0281] Step P4 specifically involves summing the areas of all triangular facets to determine the total area of ​​the lesion. Specifically, it satisfies: .

[0282] Based on this, medical staff determine the size of the lesion to assess relevant treatment options and better assist the patient.

[0283] In summary, the esophageal information detection method, system, processing method, device, and storage medium provided by this invention determine the characteristic information of the sampling device at the cardia and at the beginning of the esophagus. Using these attachments, the positional relationship between the sampling device and the esophagus relative to the world coordinate system can be shown. This constructs the relationship between the esophageal information collected by the sampling device and the world coordinate system. Therefore, based on the scale information determined by the characteristic information, the esophageal sampling data collected by the sampling device can be restored to the world coordinate system. This allows the size, position, distance, and other information contained in the esophageal sampling data to be restored to their actual size, position, and distance relative to the world coordinate system according to the "scale" defined by the scale information. Medical personnel can use this actual information as intermediate data to ensure the accuracy of subsequent treatment, diagnosis, teaching, and other work.

[0284] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0285] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting esophageal information, characterized in that, The esophageal information detection method is applied to an esophageal information detection system, which includes a sampling device and an accessory associated with the sampling device. When the sampling device collects esophageal information, the accessory uses its characteristic information to show the positional relationship between the sampling device and the esophagus relative to the world coordinate system. The esophageal information detection method includes: Obtain first feature information shown by the accessory when the sampling device is located at the cardia; The sampling device detects esophageal sampling data between the cardia and the beginning of the esophagus. Obtain second feature information shown by the accessory when the sampling device is located at the beginning of the esophagus; Scale information is determined based on the first feature information and the second feature information; Based on the scale information and the esophageal sampling data, the esophageal state information relative to the world coordinate system is determined.

2. The esophageal information detection method according to claim 1, characterized in that, The accessory feature information includes the positional relationship between the first accessory part and the first reference point, the first reference point being stationary relative to the esophagus, and the relative position between the first accessory part and the first reference point changing with the position of the sampling device.

3. The esophageal information detection method according to claim 1, characterized in that, The first feature information includes the first accessory position information of the accessory at the incisor position when the sampling device is located at the cardia, and the second feature information includes the second accessory position information of the accessory at the incisor position when the sampling device is located at the beginning of the esophagus.

4. The esophageal information detection method according to claim 3, characterized in that, Determining the scale information based on the first feature information and the second feature information includes: The scale information is determined based on the difference between the first attachment location information and the second attachment location information.

5. The esophageal information detection method according to claim 1, characterized in that, The esophageal information detection method also includes: The difference in device position information between the sampling device at the cardia and at the beginning of the esophagus was obtained; Determining the scale information based on the first feature information and the second feature information includes: The scale information is determined based on the differences between the first feature information and the second feature information, as well as the differences in the device position information.

6. The esophageal information detection method according to claim 5, characterized in that, The difference in device position information obtained between the sampling device at the cardia and at the beginning of the esophagus includes: Obtain the first sampled image at the cardia using the sampling device; Based on the first sampled image, determine the first device location information of the sampling device; A second sampling image of the sampling device at the beginning of the esophagus is obtained; Based on the second sampled image, determine the second device position information of the sampling device; Based on the first device location information and the second device location information, the moving distance of the sampling device is determined to represent the difference in the device location information.

7. The esophageal information detection method according to claim 1, characterized in that, The step of determining the esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data includes: The device position information is updated based on the scale information; the device position information corresponds to the sampling process of the sampling device. The sparse point cloud data is updated based on the scale information; the sparse point cloud data corresponds to the sampling process of the sampling device. Based on the updated device location information and the updated sparse point cloud data, a 3D reconstruction is performed to determine the 3D esophageal information relative to the world coordinate system, which is used to show the current esophageal state.

8. The esophageal information detection method according to claim 7, characterized in that, The device location information and the sparse point cloud data correspond to each other; The 3D reconstruction includes: depth reconstruction, density reconstruction, point cloud fusion, mesh reconstruction, and texture reconstruction.

9. The esophageal information detection method according to claim 1, characterized in that, The acquisition of esophageal sampling data detected by the sampling device between the cardia and the beginning of the esophagus includes: A number of esophageal sampling images were obtained by the sampling device between the cardia and the beginning of the esophagus; Calculate feature point descriptors based on esophageal sampling images; Based on the matching of feature point descriptors between esophageal sampling images, the initial pose information of the sampling device corresponding to the esophageal sampling image is determined; Based on the feature point descriptors corresponding to the esophageal sampling images and the initial pose information, global optimization is performed to obtain sparse point cloud data and device pose information; the device pose information includes device position information and device attitude information.

10. The esophageal information detection method according to claim 9, characterized in that, The esophageal information detection also includes: Obtain the camera parameters of the sampling device; The esophageal sampled images are subjected to distortion correction processing based on the camera parameters. The step of calculating feature point descriptors based on esophageal sampled images includes: Based on the distortion-corrected esophageal sample image, feature point descriptors are calculated.

11. The esophageal information detection method according to claim 9, characterized in that, The step of performing global optimization based on feature point descriptors corresponding to the esophageal sampled image and initial pose information to obtain sparse point cloud data and device pose information includes: Based on the feature point descriptors and initial pose information, determine the sparse point cloud data and the reprojection error of the feature points; With the goal of minimizing the reprojection error, a bundled adjustment and optimization is performed to adjust and determine the sparse point cloud data and device pose information.

12. The esophageal information detection method according to claim 9, characterized in that, The feature point descriptor is a SURF feature point descriptor.

13. A method for processing esophageal information, characterized in that, include: The esophageal information detection method according to any one of claims 1 to 12 determines esophageal state information; Obtain sparse point cloud data corresponding to lesions from esophageal status information; Based on the sparse point cloud data of the lesions, triangular patch segmentation is used to determine the area of ​​several triangular patches corresponding to the lesions. The total area of ​​the lesion is determined based on the area of ​​the triangular facet.

14. An esophageal information detection system, characterized in that, include: Capsule endoscopy is used to collect data from esophageal samples. The traction wire has one end fixed to the capsule endoscope, and the other end, with its position at the incisor region, indicates the positional relationship between the capsule endoscope and the esophagus relative to the world coordinate system. The data acquisition module is used to obtain first feature information shown by the traction line when the capsule endoscope is located at the cardia, to obtain second feature information shown by the traction line when the capsule endoscope is located at the beginning of the esophagus, and to obtain esophageal sampling data detected by the capsule endoscope between the cardia and the beginning of the esophagus. The scale recovery module is used to determine scale information based on the first feature information and the second feature information, and to determine esophageal state information relative to the world coordinate system based on the scale information and the esophageal sampling data.

15. The esophageal information detection system according to claim 14, characterized in that, The scale recovery module is further configured to update the device position information of the capsule endoscope according to the scale information, and update the sparse point cloud data according to the scale information; wherein the device position information corresponds to the sampling process of the capsule endoscope, and the sparse point cloud data corresponds to the sampling process of the capsule endoscope. The esophageal information detection system also includes: The 3D reconstruction module is used to perform 3D reconstruction based on the updated device location information and the updated sparse point cloud data, and determine the 3D esophageal model relative to the world coordinate system to show the current esophageal state. The display module is used to output and display the three-dimensional esophageal model.

16. An electronic device comprising: At least one processor; A memory storing a computer program executable on the processor, characterized in that the processor executes the program to perform the steps of the esophageal information detection method as described in any one of claims 1 to 12.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it performs the steps of the esophageal information detection method as described in any one of claims 1 to 12.

Images