Method for reconstructing a three-dimensional model and endoscope system

Abstract

Embodiments of the present application provide a three-dimensional model reconstruction method and an endoscope system. The method comprises: acquiring a plurality of images of a subject collected by an endoscope; constructing an actual three-dimensional model of the subject based on the plurality of images; registering the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and mapping actual color information to the standard three-dimensional model based on the spatial mapping relationship to obtain an expected three-dimensional model of the subject, wherein the actual color information is color information extracted from the plurality of images. When performing endoscopic examination on the subject using this scheme, the doctor can quickly locate the lesion position and output accurate examination results, thereby greatly improving the examination efficiency and the accuracy of the examination results. In addition, the visual effect of the expected three-dimensional model is also good, so the user experience is also better.

Description

Technical Field

[0001] This application relates to the field of three-dimensional model reconstruction technology, and more specifically to a method for reconstructing a three-dimensional model, an endoscope system, an electronic device, and a storage medium. Background Technology

[0002] In recent years, electronic endoscopy technology has been widely used in the medical field. Doctors can use endoscopic systems to examine various parts of a patient's body, thereby enabling them to promptly detect lesions such as tumors, polyps, and obstructions based on endoscopic imaging, and to initiate corresponding treatments in a timely manner.

[0003] Existing endoscopic systems only provide two-dimensional images of the examined area. The presentation of two-dimensional images cannot intuitively show the actual three-dimensional shape of the examined area, nor can it accurately present the detailed color information of the examined area. Therefore, it greatly affects the doctor's intuitive judgment, resulting in low examination efficiency and low accuracy of the output examination results. Summary of the Invention

[0004] Embodiments of this application are proposed in view of the above-mentioned problems. Embodiments of this application provide a method for reconstructing a three-dimensional model, an endoscope system, an electronic device, and a storage medium.

[0005] According to one aspect of this application, a method for reconstructing a three-dimensional model is provided, comprising: acquiring multiple images of a subject captured using an endoscope; constructing an actual three-dimensional model of the subject based on the multiple images; registering the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and mapping actual color information onto the standard three-dimensional model based on the spatial mapping relationship to obtain a desired three-dimensional model of the subject, wherein the actual color information is color information extracted from the multiple images.

[0006] For example, registering an actual 3D model to a standard 3D model of a photographed object includes: performing feature point matching between feature points on the actual 3D model and feature points on the standard 3D model to determine a set of matching points between the actual 3D model and the standard 3D model; and determining a rotation and translation matrix for registering the actual 3D model to the standard 3D model based on the positional correspondence of the matching points in the set of matching points, wherein the rotation and translation matrix is ​​used to represent the spatial mapping relationship.

[0007] For example, before mapping the actual color information onto the standard 3D model, the method further includes: downsampling the standard 3D model based on the positional correspondence of the matching points to obtain a sparse standard 3D model, wherein each feature point on the sparse standard 3D model has a corresponding matching point on the actual 3D model; mapping the actual color information onto the standard 3D model includes: mapping the actual color information onto the sparse standard 3D model based on the positional correspondence between the feature points on the sparse standard 3D model and the matching points on the actual 3D model.

[0008] For example, mapping actual color information onto a sparse standard 3D model includes: for each feature point on the sparse standard 3D model, determining a first matching point on the actual 3D model corresponding to the position of the feature point based on the positional correspondence between the coordinates of the feature point and the matching point on the sparse standard 3D model; determining the pixel corresponding to the first matching point in multiple images; and determining the color information of the feature point on the sparse standard 3D model based on the determined pixel.

[0009] For example, determining the color information of the feature point on the sparse standard 3D model based on the determined pixel includes: filtering the spatial neighborhood of the determined pixel to obtain pixel value information of the pixels in the spatial neighborhood when the first matching point corresponds to multiple pixels; and determining the color information of the feature point on the sparse standard 3D model based on the pixel value information of the pixels in the spatial neighborhood.

[0010] For example, mapping actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model further includes: filling the spatial neighborhood of the feature point on the sparse standard 3D model according to the pixel value information of the pixels in the determined spatial neighborhood of the pixel.

[0011] For example, the method further includes: identifying abnormal parts of the subject in multiple images respectively, and marking the abnormal location regions of the abnormal parts in the multiple images; and mapping the abnormal location regions onto an actual three-dimensional model to determine the abnormal location regions on the actual three-dimensional model; wherein, based on the positional correspondence between feature points on the sparse standard three-dimensional model and matching points on the actual three-dimensional model, mapping the actual color information onto the sparse standard three-dimensional model further includes: for a first feature point on the sparse standard three-dimensional model, filling the spatial neighborhood of the first feature point on the sparse standard three-dimensional model, wherein the matching point of the first feature point on the actual three-dimensional model is located within the abnormal location region on the actual three-dimensional model.

[0012] For example, acquiring multiple images of a subject captured using an endoscope includes: acquiring images while the endoscope is capturing images of the subject; wherein the steps from constructing an actual three-dimensional model of the subject to mapping actual color information onto a standard three-dimensional model are all performed based on the currently acquired images; the method further includes: displaying a desired three-dimensional model in real time, wherein regions in the desired three-dimensional model that have not yet been mapped with actual color information are displayed or not displayed in a first preset style.

[0013] For example, before displaying the desired 3D model in real time, the method further includes: mapping the abnormal location region onto the desired 3D model based on the positional correspondence of the matching points, so as to determine the abnormal location region on the desired 3D model; wherein, when displaying the desired 3D model in real time, the abnormal location region in the desired 3D model is displayed in a second preset style.

[0014] For example, the method further includes: providing a human-computer interaction interface to the user; responding to the user's operation on the human-computer interaction interface for a position point on a desired 3D model, determining a first position point on the actual 3D model corresponding to the position point based on a spatial mapping relationship, determining an image among multiple images for extracting color information of the first position point, and displaying the determined image.

[0015] For example, the method further includes: displaying the desired 3D model while displaying the determined image, and displaying the mapping relationship between the determined image and the corresponding points in the desired 3D model.

[0016] According to another aspect of this application, an endoscope system is provided, comprising: a light source device for emitting illumination light toward a subject; a camera device for receiving light signals returned from the subject and generating image signals based on the light signals; and an image processing device for generating multiple images of the subject based on the image signals; constructing an actual three-dimensional model of the subject based on the multiple images; registering the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and mapping actual color information onto the standard three-dimensional model based on the spatial mapping relationship to obtain a desired three-dimensional model of the subject, wherein the actual color information is color information extracted from the multiple images.

[0017] According to another aspect of this application, an electronic device is provided, including a processor and a memory, wherein computer program instructions are stored in the memory, and the computer program instructions are executed by the processor to perform the above-described method for reconstructing a three-dimensional model.

[0018] According to another aspect of this application, a storage medium is provided on which program instructions are stored, which are used to execute the above-described method for reconstructing the three-dimensional model when the program instructions are run.

[0019] According to the above scheme, an actual 3D model of the subject is constructed based on multiple images of the subject acquired by endoscopy. Then, based on the spatial mapping relationship between the acquired actual 3D model and a standard 3D model, the actual color information is mapped onto the standard 3D model to obtain the desired 3D model of the subject. The desired 3D model of the subject obtained by this scheme can present the subject both intuitively and accurately. Therefore, when using this scheme to perform endoscopic examinations on the subject, it can greatly facilitate doctors in accurately obtaining the detailed state of the patient's examined organs, thereby helping doctors quickly locate lesions and output accurate examination results, thus greatly improving examination efficiency and the accuracy of examination results. In addition, the visualization effect of this desired 3D model is also good, resulting in a better user experience. Attached Figure Description

[0020] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 A schematic flowchart illustrating a method for reconstructing a three-dimensional model according to an embodiment of this application is shown.

[0022] Figure 2a A schematic diagram showing an inspection report of a first preset template according to an embodiment of this application;

[0023] Figure 2b A schematic diagram showing an inspection report of a second preset template according to an embodiment of this application;

[0024] Figure 3 A schematic flowchart illustrating a method for reconstructing a three-dimensional model according to another embodiment of this application is shown.

[0025] Figure 4 A schematic block diagram of an endoscope system according to one embodiment of this application is shown; and

[0026] Figure 5 A schematic block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0028] To at least partially solve the aforementioned technical problems, according to one aspect of this application, a method for reconstructing a three-dimensional model is provided. In this reconstruction method, an actual three-dimensional model of the subject is first reconstructed based on an endoscopic image of the subject. It can be understood that this actual three-dimensional model presents the true appearance of the subject. For example, the actual three-dimensional model of a stomach in a hungry state is smaller and has more folds. The actual three-dimensional model of a stomach in a full state is larger and fuller. According to an embodiment of this application, this actual three-dimensional model is mapped onto a standard three-dimensional model of the subject, thereby obtaining its desired three-dimensional model. Thus, the shape of the desired three-dimensional model obtained according to the embodiment of this application is the standard shape of the subject, and its detailed colors are determined based on the colors of the corresponding parts of the endoscopic image. This provides the user with a more realistic three-dimensional model.

[0029] Figure 1 A schematic flowchart illustrating a method 100 for reconstructing a three-dimensional model according to an embodiment of this application is shown. Figure 1 As shown, the reconstruction method 100 includes steps S120, S140, S160 and S180.

[0030] Step S120: Acquire multiple images of the subject using an endoscope.

[0031] According to embodiments of this application, the multiple images acquired in this step can be obtained using any existing or future-developed endoscope. Exemplarily, the endoscope may include various types of endoscopic cameras, such as monocular cameras, binocular cameras, and multi-view cameras. Images of the subject acquired using these cameras can be used for the reconstruction of a three-dimensional model. The subject can be any suitable part of the human body or animal to be examined, such as cavities or organs like the upper digestive tract, lower digestive tract, esophagus, stomach, intestines, nasopharynx, bronchi, and ureters.

[0032] According to embodiments of this application, the plurality of images are two-dimensional color images, such as two-dimensional RGB images. Alternatively, the plurality of images can also be two-dimensional grayscale images. Exemplarily, the plurality of images can be static images of the subject acquired via an endoscope, or multiple frames from a series of dynamic video frames of the subject acquired via an endoscope. The plurality of images can be raw images directly acquired by the endoscope, or images obtained after preprocessing the raw images. The preprocessing operations can include all operations to improve the visual effect of the image, enhance image clarity, or highlight certain features in the image to facilitate 3D modeling. For example, the preprocessing operations can include noise reduction operations such as filtering, adjustments to image parameters such as grayscale, contrast, and brightness, and operations such as image cropping or scaling.

[0033] In one example, the multiple images acquired in this step can be images of the subject captured in real time by the endoscope. For instance, they could be images captured in real time as the endoscope moves through the patient's stomach during an endoscopy. These multiple images could be images acquired from the moment the endoscope began acquiring data to the current moment.

[0034] In another example, the multiple images acquired in this step can also be pre-captured images of the subject. For example, step S120 can be performed after acquiring images of the human stomach using an endoscope.

[0035] Step S140: Based on the multiple images obtained in step S120, construct an actual three-dimensional model of the photographed object.

[0036] According to the embodiments of this application, this step can be implemented using any existing or future suitable 3D reconstruction algorithm. Exemplarily, and not limitingly, an actual 3D model of the photographed object can be constructed based on multiple acquired images using any of the following methods: time-of-flight algorithm, structure-of-motion reconstruction algorithm, simultaneous localization and mapping algorithm, binocular stereo matching algorithm, etc.

[0037] It is understood that the actual 3D model can be a real 3D model of the object examined by the endoscope, which can fully and realistically reflect the true shape of the object. According to embodiments of this application, the actual 3D model can include both the shape information of the object and the color information of each part of the object. It is understood that in embodiments of this application, if the acquired image is a color image, the color information can include color information, such as the data values ​​of the red, green, and blue channels. If the acquired image is a grayscale image, the color information can include grayscale information. Alternatively, the actual 3D model can also only include the shape information of the object without color information. The actual 3D model can be a 3D model of the stomach composed of multiple triangular meshes. The vertex of each triangular mesh can be a feature point of the actual 3D model. The shape information of the actual 3D model can include the position information of the 3D coordinates of each feature point, and its color information can include the color information corresponding to each feature point. It is understood that the color information corresponding to each feature point can be determined based on the pixel value information of the corresponding pixels in multiple images acquired by the endoscope.

[0038] According to the embodiments of this application, the actual 3D model constructed in step S140 can be a complete 3D model of the subject or a partial 3D model of the subject. As mentioned above, the multiple images acquired in step S120 can be images of the subject acquired in real time by the endoscope. It can be understood that, in this case, the completeness of the actual 3D model depends on the progress of the acquisition. In one example, if the overall acquisition of the subject has been completed, the actual 3D model constructed in this step can be a complete 3D model of the subject. In another example, if the overall acquisition of the subject has not been completed, the actual 3D model constructed in this step can be a partial 3D model of the subject, that is, the model may have some missing parts.

[0039] Step S160: Register the actual 3D model to the standard 3D model of the photographed object to obtain the spatial mapping relationship between the actual 3D model and the standard 3D model.

[0040] A standard 3D model can be a 3D model of the subject that meets preset morphological standards. Exemplarily, and not limitingly, a standard 3D model can be a relatively regular, idealized 3D model. In a specific example, if the subject is the upper digestive tract (esophagus + stomach), the standard 3D model could be a beautified, standard upper digestive tract 3D model. See also... Figure 2aThe left side shows a standard 3D model of the upper digestive tract. As mentioned earlier, the standard 3D model presents the standard form of the subject, while the actual 3D model presents the real form of the subject. It is understandable that there can be certain morphological differences between the standard 3D model and the actual 3D model.

[0041] The standard 3D model can be any suitable form of 3D model. For example, it can be a meshed 3D model composed of multiple triangular facets, or a point cloud 3D model composed of multiple points. According to an embodiment of this application, the standard 3D model includes standard shape information of the subject. This shape information can be the positional information of multiple feature points constituting the standard 3D model; for example, each feature point can correspond to a vertex of a triangular facet. The number of feature points in the standard 3D model can be the same as or different from the number of feature points in the actual 3D model. In one example, the standard 3D model can be a high-resolution 3D model, whose number and distribution density of feature points can be greater than that of the actual 3D model. According to an embodiment of this application, the standard 3D model can include standard color information of the subject, or it can exclude its color information. Optionally, the standard 3D model can be a 3D model that does not include color information.

[0042] It is understandable that during endoscopic examinations, the subject may present various forms depending on the animal's physical condition. For example, the human stomach may appear shrunken and wrinkled. The visual effect of such a three-dimensional model of the subject is poor, and if presented directly to the observer, it may cause visual discomfort or cognitive confusion, thus affecting their experience. Furthermore, in procedures such as those performed by a doctor, it may interfere with the doctor's assessment of certain lesions.

[0043] According to the embodiments of this application, the constructed actual 3D model of the subject can be registered with a standard 3D model to obtain the spatial mapping relationship between the actual 3D model and the standard 3D model. This step can be implemented using any existing or future suitable 3D model registration method, including but not limited to any one of various 3D model registration algorithms such as template matching, feature matching, and nearest neighbor matching.

[0044] The spatial mapping relationship between the actual 3D model and the standard 3D model can be any suitable spatial mapping relationship. Exemplarily, but not limitingly, this spatial mapping relationship can include the correspondence between at least a portion of feature points in the actual 3D model and at least a portion of feature points in the standard 3D model, and the mapping path between two corresponding feature points. This correspondence of feature points can be a correspondence of all feature points or a correspondence of only a portion of feature points. Optionally, if the number and distribution density of feature points in the standard 3D model are greater than those in the actual 3D model, the correspondence of feature points can be a correspondence between all feature points in the actual 3D model and a portion of feature points in the standard 3D model. Alternatively, if the number and distribution density of feature points in the standard 3D model are less than those in the actual 3D model, the correspondence of feature points can include a correspondence between a portion of feature points in the actual 3D model and all feature points in the standard 3D model. Specifically, the correspondence of feature points can include a correspondence between all feature points in the actual 3D model and all feature points in the standard 3D model.

[0045] According to embodiments of this application, after registration, the actual 3D model and the standard 3D model can be represented based on the same coordinate system. For example, in this coordinate system, the centers of the two 3D models can have the same coordinate values. Since there are morphological differences between the actual 3D model and the standard 3D model, at least some feature points of the two models in the same coordinate system after registration can have a certain distance between them.

[0046] It should be noted that in this step, only the spatial mapping relationship between the two models needs to be acquired and stored. This spatial mapping relationship can be expressed in any suitable form; for example, a mapping function representing the spatial mapping relationship can be stored.

[0047] Step S180: Based on the spatial mapping relationship, the actual color information is mapped onto a standard 3D model to obtain the desired 3D model of the subject. The actual color information is extracted from multiple images.

[0048] In one example, the actual color information can be color information extracted from multiple images and mapped to the actual 3D model. As mentioned earlier, the actual 3D model constructed in step S160 can include color information. It is understood that the source of the color information of the actual 3D model is multiple images. In this case, based on the spatial mapping relationship between the actual 3D model and the standard 3D model, the color information of the actual 3D model can be directly mapped to the standard 3D model. As mentioned earlier, the spatial mapping relationship between the actual 3D model and the standard 3D model obtained in step S160 can include the correspondence between feature points of the actual 3D model and feature points of the standard 3D model. In this step, the color information of feature points in the actual 3D model can be mapped to the corresponding feature points in the standard 3D model, so that the corresponding feature points also have the same color information. For example, if there is a correspondence between feature point a1 in the actual 3D model and feature point b1 in the standard 3D model, i.e., there is a spatial mapping relationship between the two, then the color information corresponding to feature point a1 can be assigned to feature point b1 in the standard 3D model.

[0049] In another example, the actual color information can also be extracted directly from multiple images. Based on the correspondence between pixels in multiple images and points in the actual 3D model, as well as the spatial mapping relationship between the actual 3D model and the standard 3D model, for any point on the standard 3D model, the corresponding pixels in multiple images can be determined. Furthermore, the color information of points on the standard 3D model can be determined directly based on the color information of these corresponding pixels.

[0050] First, as mentioned earlier, for any point on the standard 3D model, based on the spatial mapping relationship between the actual 3D model and the standard 3D model, one or more points corresponding to that point can be determined in the actual 3D model. Furthermore, during the construction of the actual 3D model, for each point in the actual 3D model, there exists a corresponding pixel. According to the embodiments of this application, since the actual 3D model is constructed based on multiple color images obtained in step S120, each point on the actual 3D model can find a corresponding pixel in multiple images. For each point on the actual 3D model, the pixel corresponding to that point can be at least one pixel in at least one of the multiple images acquired using an endoscope. Optionally, the pixel corresponding to that point can be one pixel in one image, or multiple pixels in one image. Alternatively, the pixel corresponding to a point on the actual 3D model can also be pixels located in multiple images respectively. It can be understood that each point on the actual 3D model corresponds to a certain location point on the object being photographed. This location point may appear in multiple images when the endoscope acquires images. Therefore, a point on the actual 3D model can correspond to a specific pixel in these images. It's understandable that if an endoscope acquires images while in motion, then a point on the actual 3D model corresponds to pixels in different images, each located at a different position within the image. For example, a point on the actual 3D model can correspond to pixels in adjacent video frames across multiple frames. Specifically, this point might correspond to pixel A in the first frame, pixel B in the second frame, and pixel C in the third frame. In short, for any point on the actual 3D model, the corresponding pixel can be determined across multiple images. Therefore, based on the mapping path from pixels in multiple images to points on the standard 3D model, it's possible to assign color information from multiple images to feature points on the standard 3D model.

[0051] This step S180 can be implemented using any existing or future texture mapping (or texture mapping) method. It can be understood that texture mapping is the process of coloring a standard 3D model. For examples where the standard 3D model itself does not contain color information, this step can be a process of adding color information to the standard 3D model. For examples where the standard 3D model itself includes color information, this step can be a process of adjusting the color information of the standard 3D model.

[0052] It is understandable that after mapping each feature point in the standard 3D model to obtain its corresponding color information, a colored standard 3D model can be obtained. As mentioned earlier, the spatial mapping relationship between the actual 3D model and the standard 3D model may only include the positional correspondence of some feature points. If the feature point correspondence in the spatial mapping relationship includes the correspondence of each feature point in the standard 3D model, then in this step, the actual color information can be directly mapped onto the standard 3D model based on the spatial mapping relationship, thereby obtaining a standard 3D model of the subject with color information, i.e., the desired 3D model. However, if some feature points in the standard 3D model are not involved in the above correspondence, for example, if the number and density of feature points in the standard 3D model are greater than those in the actual 3D model, then the standard 3D model can first be downsampled. Then, based on the spatial mapping relationship, the actual color information can be mapped onto the downsampled standard 3D model to obtain the desired 3D model of the subject.

[0053] According to the above scheme, an actual 3D model of the subject is constructed based on multiple images of the subject acquired through endoscopy. Then, based on the spatial mapping relationship between the acquired actual 3D model and a standard 3D model, the actual color information extracted from the multiple images is mapped onto the standard 3D model to obtain the desired 3D model of the subject. The desired 3D model of the subject obtained by this scheme can intuitively and accurately present the actual shape of the subject. Therefore, when using this scheme to perform endoscopic examinations on the subject, it can greatly facilitate doctors in accurately obtaining the detailed state of the patient's subject, thereby helping doctors quickly locate lesions and output accurate examination results, thus greatly improving examination efficiency and the accuracy of examination results. In addition, the visualization effect of this desired 3D model is also good, resulting in a better user experience.

[0054] For example, step S160 registers the actual 3D model to the standard 3D model of the subject, including steps S161 and S162. Step S161 involves matching feature points on the actual 3D model with feature points on the standard 3D model to determine a set of matching points between the two models. Step S162 determines the rotation and translation matrix for registering the actual 3D model to the standard 3D model based on the positional correspondence of the matching points in the set. The rotation and translation matrix represents the spatial mapping relationship.

[0055] According to the embodiments of the present application, any existing or future developed feature point matching method can be used to implement step S161. By way of example and not limitation, the Iterative Closest Point (ICP for short) algorithm can be used for fine matching of feature points. Before matching with the iterative closest point algorithm, other suitable coarse registration methods can also be used to first obtain a coarse matched point set between the actual three-dimensional model and the standard three-dimensional model. For example, a coarse matched point set can be preliminarily determined based on the normal vectors of feature points and Fast Point Feature Histograms (FPFH for short) features, and fine registration such as by the ICP algorithm is performed based on the preliminarily determined matched point set, which can reduce the amount of calculation.

[0056] Then, based on the ICP algorithm, a matched point set composed of feature point p in the actual three-dimensional model j and feature point q in the standard three-dimensional model j can be determined. It can be understood that feature point p j and feature point q j can be referred to as a group of corresponding matching point pairs. According to the respective position coordinates of each group of matching point pairs, a rotation-translation relation formula for registration transformation of feature points in the actual three-dimensional model to corresponding feature points in the standard three-dimensional model can be determined:

[0057]

[0058] wherein 0<j<the total number of reference structure points +1, R represents a rotation matrix, and T represents a translation matrix. The rotation matrix and the translation matrix may be collectively referred to as a rotation-translation matrix. Based on the solved rotation matrix and translation matrix, the spatial mapping relationship between the feature points of the actual three-dimensional model and the standard three-dimensional model, that is, the mapping path, can be obtained. Illustratively, the actual three-dimensional model can also be subjected to coordinate transformation based on the rotation matrix and the translation matrix, so as to unify the actual three-dimensional model and the standard three-dimensional model into the same coordinate system.

[0059] According to the above solution, the feature point matching method can be used to determine the matched point set between the actual three-dimensional model and the standard three-dimensional model, and the rotation-translation matrix for registering the actual three-dimensional model to the standard three-dimensional model. This solution has high registration efficiency and accuracy. Furthermore, the above technical solution can obtain a more realistic expected three-dimensional model of the photographed object.

[0060] Illustratively, as described above, the standard three-dimensional model can also be a three-dimensional model with higher resolution, and the number and density of its feature points can both be greater than those of the actual three-dimensional model. For example, feature point p in the actual three-dimensional model determined in step S161 j and feature point q of the standard three-dimensional modelj In the set of matching points between them, q j These can be some feature points from a standard 3D model. For redundant feature points in the standard 3D model that are not in the matching point set, they can be deleted to reduce the computational load in subsequent steps.

[0061] Before mapping the actual color information onto the standard 3D model in step S180, the method 100 further includes step S170. In step S170, the standard 3D model is downsampled based on the positional correspondence of the matching points to obtain a sparse standard 3D model. Each feature point in the sparse standard 3D model has a corresponding matching point in the actual 3D model. Based on the coordinates of each pair of matching points in the matching point set, feature points in the standard 3D model that are not in that matching point set can be deleted to obtain the sparse standard 3D model. For example, if the standard 3D model includes 1000 feature points, and 500 of these feature points fall within the matching point set, then the other 500 feature points can be deleted, leaving only the feature points that fall within the matching point set as the sparse standard 3D model. Thus, for each feature point in the sparse standard 3D model, a matching point in the actual 3D model corresponding to the position of that feature point can be found. Exemplarily, each feature point in the actual 3D model can be a matching point falling within the matching point set, or only some feature points in the actual 3D model can be matching points falling within the matching point set.

[0062] Step S180 maps the actual color information onto the standard 3D model, including step S181. Step S181 maps the actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model. Since each feature point on the downsampled sparse standard 3D model can find a corresponding matching point on the actual 3D model, for each feature point on the sparse standard 3D model, the color information corresponding to that feature point's position can be mapped onto that feature point as its corresponding color information. As mentioned earlier, the color information corresponding to that feature point's position can originate from matching points on the actual 3D model, or from pixels in multiple images corresponding to that matching point. Thus, the color information corresponding to each feature point on the sparse standard 3D model can be determined, thereby obtaining the color standard 3D model.

[0063] According to the above scheme, the standard 3D model can be downsampled based on the positional correspondence of matching points in the matching point set between the actual 3D model and the standard 3D model to remove redundant feature points. This allows the acquisition of color information from the downsampled standard 3D model. This scheme reduces the computational burden of redundant feature points, significantly improving processing speed and saving storage space.

[0064] For example, step S181 maps the actual color information onto the sparse standard 3D model based on the positional correspondence between the feature points on the sparse standard 3D model and the matching points on the actual 3D model, including: for each feature point on the sparse standard 3D model, executing steps S181.1, S181.2 and S181.3.

[0065] Step S181.1: Based on the correspondence between the coordinates of the feature point on the sparse standard 3D model and the positions of the matching points in the matching point set, determine the first matching point on the actual 3D model corresponding to the position of the feature point. For example, for each feature point on the sparse standard 3D model... By substituting the coordinates of the feature point into the aforementioned rotation and translation matrix formula, the feature point on the actual 3D model corresponding to that feature point can be obtained. The position coordinates of this feature point This is called the first matching point.

[0066] Step S181.2 involves determining the pixel corresponding to the first matching point in multiple images. As mentioned earlier, for each point on the actual 3D model, at least one corresponding pixel can be matched in multiple images. This corresponding pixel can be one or more. If there are multiple corresponding pixels, these pixels can be located in one image or in different images. Based on the foregoing description, those skilled in the art can understand the specific implementation of step S181.2, and for the sake of brevity, it will not be elaborated further here.

[0067] Step S181.3: Based on the determined pixels, determine the color information of the feature point on the sparse standard 3D model. It can be understood that for each feature point on the actual 3D model, its corresponding color information can be determined by the pixel information of the corresponding pixel. There is also a correspondence between each feature point on the sparse standard 3D model and its matching points on the actual 3D model. Therefore, a mapping relationship between each feature point on the sparse standard 3D model and pixels in multiple images can be indirectly constructed. That is, the color texture information corresponding to each feature point on the sparse standard 3D model can be obtained. For example, the coordinate names of each feature point on the sparse standard 3D model and the correspondence with the corresponding pixels can be stored, and point-by-point color mapping can be performed on each feature point on the sparse standard 3D model using, for example, a coordinate index of the same-name point.

[0068] It is understandable that in existing methods for 3D reconstruction based on multiple 2D images, the different shooting conditions of adjacent images, such as changes in ambient light when acquiring adjacent frames, can lead to different pixel values ​​at the same location in the overlapping area of ​​the two images. This can result in ghosting artifacts in the corresponding overlapping areas of the colored 3D model during color information mapping. Such ghosting artifacts have poor visual effects and affect the doctor's judgment of the examination results. The point-by-point color mapping method used in this application embodiment can effectively avoid this type of ghosting artifact problem.

[0069] In the above scheme, the correspondence between feature points of the actual 3D model and pixels of multiple images can be established based on the correspondence between feature points of the actual 3D model and the positional correspondence between matching points of the actual 3D model and the sparse standard 3D model. This allows for point-by-point color information mapping, transferring the color information of the images onto the sparse standard 3D model. This color information mapping method is relatively simple and computationally inefficient. Furthermore, it effectively avoids ghosting artifacts in overlapping areas of adjacent image blocks, resulting in a better and more accurate color standard 3D model with superior visual effects. This leads to more accurate inspection results and a better user experience.

[0070] For example, step S181.3, which determines the color information of the feature point on the sparse standard 3D model based on the determined pixels, includes the following steps: First, if the first matching point corresponds to multiple pixels, the spatial neighborhood of the determined pixel is filtered to obtain the pixel value information of the pixels in the spatial neighborhood. Then, the color information of the feature point on the sparse standard 3D model is determined based on the pixel value information of the pixels in the spatial neighborhood.

[0071] As mentioned earlier, the first matching point may correspond to multiple pixels, and the feature point on the sparse standard 3D model corresponding to the first matching point also corresponds to these multiple pixels. These multiple pixels can exist in the same image or in different images. When multiple pixels exist in the same image, to accurately determine the color information corresponding to the feature point on the sparse standard 3D model, the spatial neighborhood of these multiple pixels in the same image can be filtered. The size of this spatial neighborhood can be arbitrarily set according to actual needs. For example, the spatial neighborhood can be an 8px*8px pixel area or a 10px*10px pixel area centered on these multiple pixels. Any suitable filtering algorithm can be used to filter the determined spatial neighborhood of the pixels. For example, a smooth linear filtering method can be used to filter, thereby determining the pixel value information of the pixels within the smoothed filtering spatial neighborhood. Then, based on, for example, the pixel value information of the pixels within the smoothed filtering spatial neighborhood, the color information of the feature point on the sparse standard 3D model can be determined. For example, if the pixel values ​​of the multiple filtered pixels are the same, these identical pixel values ​​can be directly determined as the pixel values ​​of the corresponding feature point on the sparse standard 3D model. When the pixel values ​​of the filtered pixels are different, a method such as weighted averaging can be used to determine the average pixel value of the pixels, and then the color information of the corresponding feature point can be determined. Alternatively, a preset evaluation and ranking method can be used to sort the pixels, and the color information of the corresponding feature point can be determined based on the pixel value of the pixel with the highest final score or the first pixel in the ranking.

[0072] Furthermore, as mentioned earlier, the first matching point may correspond to multiple pixels, and the feature points on the sparse standard 3D model corresponding to this matching point also correspond to these multiple pixels. These multiple pixels can exist in different images, for example, corresponding to pixels at different pixel positions in adjacent frames. In this approach, pixels at different pixel positions can be directly evaluated and sorted according to a preset evaluation and sorting rule. Finally, the color information of the corresponding feature point of the sparse standard 3D model can be determined based on the pixel value of the pixel with the highest score. Alternatively, the spatial neighborhood of the pixel in each image of the adjacent frames can be filtered first, and the color information of the corresponding feature point of the sparse standard 3D model can be determined based on the pixel value information of the pixels in the filtered adjacent frame images.

[0073] In the above scheme, when the first matching point corresponds to multiple pixels, spatial domain filtering can be applied to these pixels, and the color information of the corresponding feature points on the sparse standard 3D model can be determined based on the filtering results. This scheme has a low computational cost and the determined color information is relatively accurate, ensuring the accuracy of the color information of the final desired 3D model.

[0074] For example, step S181 maps the actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model, and also includes step S181.4. Step S181.4 fills the spatial neighborhood of the feature point on the sparse standard 3D model according to the pixel value information of the pixels in the determined spatial neighborhood of the pixel to obtain the desired 3D model.

[0075] According to embodiments of this application, pixels corresponding to each feature point on a sparse standard 3D model can be determined using a method such as the aforementioned corresponding point coordinate index, and color information of the feature points can be determined based on the corresponding pixels. Optionally, color information of key parts of the subject can be filled in according to actual needs to improve the detail effect of some key parts. For example, color information of key parts can be filled in to make them present a high-resolution visual effect. Specifically, feature points on the sparse standard 3D model related to key parts can be determined, and then color information of the spatial neighborhood of these feature points on the sparse standard 3D model can be filled in. After determining the pixel corresponding to each feature point, the pixel value information of the pixels in the spatial neighborhood of the pixel in its image can be determined. This spatial neighborhood can also be arbitrarily set according to actual needs. For example, it can be a 5px*5px pixel field centered on the pixel, and the pixel value information of each pixel in this pixel field can be determined, and then the pixel values ​​of these pixels can be mapped to the spatial neighborhood of the feature point on the sparse standard 3D model. In specific implementation, feature points can also be expanded in the spatial neighborhood near at least some feature points. For example, a predetermined number of extended feature points are added at equal intervals between each adjacent feature point. Then, according to a certain positional correspondence, the pixel values ​​of multiple pixels in the spatial neighborhood of the pixel corresponding to the feature point can be mapped to the spatial neighborhood of these extended feature points to obtain the filled color standard 3D model.

[0076] Based on the above scheme, the spatial neighborhood of a feature point in a sparse standard 3D model can also be filled with pixels near the corresponding pixels of that feature point to obtain the desired 3D model. This scheme requires less computation and can produce a high-resolution desired 3D model that accurately represents detailed features.

[0077] For example, the method 100 further includes steps S130 and S150.

[0078] Step S130 involves identifying abnormal parts of the subject in multiple images and marking the abnormal location regions of these abnormal parts in the multiple images. After acquiring multiple images of the subject using an endoscope, abnormal parts of the subject can be identified based on a certain recognition algorithm. For example, abnormal parts such as tumors, polyps, and ulcers in the stomach can be identified. It is understood that abnormal parts have different pixel features than normal parts, and these pixel features can be used to identify abnormal parts of the subject. For example, an abnormal part recognition model can be pre-trained, and multiple images can be input into the abnormal recognition model, which will output the abnormal location regions of the abnormal parts in the multiple images for marking. Any suitable region marking method can be used for marking. For example, the abnormal location regions can be marked in the form of rectangular boxes.

[0079] Step S150: Map the abnormal location region onto the actual 3D model to determine the abnormal location region on the actual 3D model. As mentioned earlier, each feature point of the actual 3D model corresponding to the abnormal location region in the image can be determined based on the positional correspondence between each feature point on the actual 3D model and pixels in the image. The corresponding location region of the actual 3D model covered by the corresponding feature point can be determined as the abnormal location region on the actual 3D model. For example, if the feature point of the actual 3D model corresponding to the abnormal location region in the image is three adjacent feature points, the region enclosed by these three feature points can be considered the abnormal location region.

[0080] The color information of the abnormal location region in the actual 3D model can be determined based on the pixel value information of each pixel in the abnormal location region. That is, in this step, the pixel value information of the abnormal location region in the image can be mapped to the abnormal location region in the actual 3D model.

[0081] In an optional embodiment, the marking can also be presented in the actual 3D model in a suitable manner. For example, the area of ​​the actual 3D model can be highlighted, or abnormal information of the abnormal location area can be displayed in response to the user's human-computer interaction. The abnormal information may be, for example, lesion attribute information and lesion size information of the area. For example, in response to clicking on the abnormal area, a message such as "ulcer, size 1.2*3" can be displayed.

[0082] According to one embodiment of this application, step S181 maps the actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model. Step S181.5 may further include step S181.5. In step S181.5, for a first feature point on the sparse standard 3D model, the spatial neighborhood of the first feature point on the sparse standard 3D model is filled to obtain the desired 3D model. The matching point of the first feature point on the actual 3D model is located within an abnormal location region on the actual 3D model.

[0083] In one example, the color information of multiple images can be mapped onto an actual 3D model first. Then, the spatial neighborhood of the corresponding feature point on the sparse standard 3D model is filled according to the color information of the first feature point on the actual 3D model to obtain the desired 3D model. That is, step S181.5 may include steps S181.51 and S181.52.

[0084] Step S181.51: Based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model, the color information of the feature points of the actual 3D model is mapped to the feature points of the sparse standard 3D model. As mentioned above, according to the embodiments of this application, the color information of the feature points of the actual 3D model can be directly mapped to the corresponding feature points of the sparse standard 3D model. For example, feature point A on the actual 3D model corresponds to feature point B in the sparse standard 3D model. The color information corresponding to feature point A on the actual 3D model is, for example, the pixel value (255, 240, 245). In this step, the color information corresponding to feature point B in the sparse standard 3D model can be directly determined as the pixel value (255, 240, 245).

[0085] Step S181.52: Based on the color information of the abnormal location regions on the actual 3D model, the corresponding regions of the sparse standard 3D model are filled to obtain the desired 3D model. As mentioned earlier, the color information of the abnormal location regions in the actual 3D model can be filled high-resolution pixel information. In this step, the color information of the abnormal location regions on the actual 3D model can be directly filled into the sparse standard 3D model. Optionally, based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model, linear interpolation can be used to fill the color information of the abnormal location regions on the actual 3D model into the sparse standard 3D model. Alternatively, based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model, methods such as multi-view solid geometry, feature point expansion, and region growing can be used to densify and expand the corresponding regions of the sparse standard 3D model.

[0086] In another example, pixels corresponding to the first feature point on the sparse standard 3D model in multiple images can be determined first, and then the spatial neighborhood of the first feature point on the sparse standard 3D model can be filled based on the pixel value information of the pixels in the spatial neighborhood of the pixel in the multiple images. Step S181.5 may also include step S181.53. In the previous step S181.2, pixels corresponding to the feature point on the sparse standard 3D model were determined in multiple images. Therefore, in step S181.53, the spatial neighborhood of the first feature point on the sparse standard 3D model can be filled based on the pixel value information of the pixels in the spatial neighborhood of the determined pixel. Here, the matching point of the first feature point on the actual 3D model is located in an abnormal location region on the actual 3D model. Unlike the embodiment including steps S181.51 and S181.52, in step S181.53, for the first feature point on the sparse standard 3D model, the spatial neighborhood of the first feature point is filled based on the pixel value of the pixels in the spatial neighborhood of the corresponding pixel in the multiple images. Those skilled in the art can understand the specific implementation of step S181.53 by reading the above description of step S181.4, and for the sake of brevity, it will not be described again here.

[0087] In existing technologies, doctors typically use an endoscope to examine patients, simultaneously acquiring two-dimensional images and storing the results for report output. Finally, a suitable image is selected from the stored images for report output. This approach requires a certain level of skill from the doctor. For doctors unfamiliar with the procedure, errors such as hand tremors can easily lead to blurry images when capturing target lesions, resulting in a high misdiagnosis rate. Furthermore, this method of storing images while examining increases the doctor's workload and reduces examination efficiency. However, in the embodiments described in this application, abnormal areas can be marked based on the acquired two-dimensional images, and the relevant lesions can be displayed at high resolution within a sparse standard three-dimensional model of the subject, accurately and clearly presenting the lesions. Moreover, no doctor intervention is required. Therefore, this improves examination efficiency and facilitates doctor verification and output after the examination.

[0088] According to the above scheme, based on multiple images of the acquired object, high-resolution color information can be presented for corresponding abnormal areas in a sparse standard 3D model. This scheme can clearly display the detailed color information of important abnormal areas while saving computational resources, thus presenting and accurately outputting examination results with better visual effects. It reduces operational difficulty, greatly improves the efficiency of doctors' examinations and result output, and ensures the accuracy of the output examination. Therefore, the user experience is better.

[0089] For example, step S120 acquires multiple images of the subject captured by an endoscope, including: acquiring images while the endoscope is capturing images of the subject. Steps S140, constructing the actual 3D model of the subject, to S180, mapping the actual color information onto the standard 3D model, are all performed based on the currently acquired images.

[0090] As mentioned above, step S120 acquires multiple images of the subject using an endoscope, which can be real-time images. For example, these multiple images can be images acquired in real-time during an endoscopic examination of the patient's subject, i.e., images acquired from the start of the examination to the present as the endoscope is advanced. Based on the multiple real-time acquired images, an actual 3D model of the subject can be constructed in real-time, and the spatial mapping relationship between the actual 3D model and the standard 3D model can be acquired in real-time, as well as the actual color information can be mapped onto the standard 3D model in real-time.

[0091] The method 100 further includes step S191. In step S191, the desired 3D model is displayed in real time. Regions in the desired 3D model that have not yet been mapped with actual color information are displayed in a first preset style. It is understood that, for example, in cases where the examination of the patient's subject has not yet been completed, the multiple images acquired in step S120 only include partial images of the subject. Therefore, the actual 3D model constructed in step S140 can be a partial 3D model of the subject. Furthermore, as the endoscope advances, other partial 2D images of the subject are gradually added, thus the acquired actual 3D model can be gradually improved. Before the examination ends, for the examined portions of the subject, the color information of those portions can be presented in real time in the desired 3D model acquired in step S180. For the unexamined portions, regions in the desired 3D model that have not yet been mapped with color information are displayed in the first preset style. This first preset style can be any suitable display style, for example, it can be displayed in a form different from the actual color of the subject. Exemplarily, and not limitingly, the first preset style can be a display style filled with a solid color such as gray or white, or the portion may not be displayed. The first preset style should be easy for users to distinguish between the inspected and uninspected parts.

[0092] In existing technologies, most endoscopic examination systems provide two-dimensional images of the subject, indicating the examination progress by highlighting specific areas. This method of highlighting specific areas cannot intuitively represent the actual condition of the patient's organs, nor can it accurately reflect the progress of examinations, especially for details. Consequently, the rate of missed diagnoses is relatively high. After the examination is completed, if the doctor finds the image results unsatisfactory, a second examination is required. Thus, the examination efficiency is also low.

[0093] In the solution of this application embodiment, areas that have not yet been mapped with color information can be displayed or not displayed in a preset display style within the desired 3D model displayed in real time. This allows for the differentiation between inspected and uninspected portions, providing a clear and accurate presentation of the inspection progress within the 3D model. This solution significantly improves the accuracy and efficiency of inspections, and also enhances the user experience.

[0094] For example, before displaying the desired 3D model in real time in step S191, method 100 further includes: mapping abnormal location regions onto the desired 3D model based on the positional correspondence of matching points, so as to determine abnormal location regions on the desired 3D model. Wherein, when displaying the desired 3D model in real time, the abnormal location regions in the desired 3D model are displayed in a second preset style.

[0095] By mapping the abnormal location region onto the actual 3D model through the aforementioned step S150, the abnormal location region on the actual 3D model is determined. After step S150, the abnormal location region can be further mapped onto the desired 3D model based on the positional correspondence between feature points in the actual 3D model and the sparse standard 3D model, thereby determining the abnormal location region on the desired 3D model.

[0096] The identified abnormal location region in the desired 3D model can be displayed using a second preset style. This second preset style can be any suitable display style, as long as it clearly distinguishes the abnormal location region. For example, the entire abnormal location region in the desired 3D model can be highlighted, or the border of the abnormal location region can be highlighted with a high-contrast color. Furthermore, abnormal information about the abnormal location region can be displayed in response to user interaction. This abnormal information may include lesion attribute information and lesion size information, such as displaying "ulcer, size 1.2*3" in response to a user's click on the abnormal region in the desired 3D model. In a specific example, during an endoscopy examination of a patient's upper gastrointestinal tract, a standard 3D model matching the patient's actual gastrointestinal tract can be displayed in real time. The automatically identified abnormal location regions of the patient's upper gastrointestinal tract can be highlighted within this standard 3D model, and lesion information about the abnormal location region can be displayed simultaneously when the doctor moves the cursor to the highlighted area.

[0097] In the above solution, abnormal location areas in the desired 3D model can also be displayed on the interface. This allows users to intuitively understand the abnormal conditions of the photographed object, enabling them to obtain abnormal location information promptly and accurately, thus significantly improving the user experience.

[0098] For example, the method 100 further includes steps S192 and S193. Step S192 provides a human-computer interaction interface to the user. Step S193, in response to the user's operation on the human-computer interaction interface for a desired position point on the 3D model, determines a first position point on the actual 3D model corresponding to the position point based on a spatial mapping relationship, determines an image from multiple images for extracting color information of the first position point, and displays the determined image.

[0099] Human-computer interaction interfaces (HCIs) can be, for example, the examination interface of a photographed object displayed on a monitor. For instance, after a user, such as a doctor, completes an examination of a patient using an endoscope, they can click on a lesion area marked on the desired 3D model of the photographed object displayed on the examination interface. Upon receiving this click from the user, the corresponding area on the actual 3D model can be determined based on spatial mapping relationships. In the process of constructing the actual 3D model based on multiple images acquired by the endoscope, a correspondence is established between multiple images and points in the actual 3D model. It can be understood that if the position of a point in the actual 3D model is determined based on a certain image, then that point corresponds to that image. Furthermore, based on the correspondence between multiple images and points in the actual 3D model, one or more images corresponding to the lesion area can be determined from among the multiple images, and these images can be displayed on the examination interface. For example, after reviewing images related to the lesion area, a user, such as a doctor, can also select the desired examination result images based on the patient's examination results and perform corresponding HCI operations to output an examination report.

[0100] The above solution provides a human-computer interaction interface and can display a local image of the subject captured by the endoscope at any point on the desired 3D model, based on the user's operation on that point. By establishing a correspondence between the original image and the 3D model, this solution allows users to easily and accurately locate and output detailed images of the subject. Furthermore, this solution eliminates the need for users to save images during the examination, thereby improving examination efficiency and providing a better user experience.

[0101] For example, the method 100 further includes: displaying a desired three-dimensional model while displaying the determined image, and displaying the mapping relationship between the determined image and the corresponding points in the desired three-dimensional model.

[0102] For example, in response to a user's first operation on the image determined in step S193 within the human-computer interaction interface, an inspection report of the photographed object can be output. This inspection report includes an image of the desired 3D model (or, a standard 3D model) and the determined image, as well as the mapping relationship between position points in the image of the desired 3D model (or, a standard 3D model) and the determined image.

[0103] By way of example, but not limitation, the first operation can be a user clicking the selected image with the mouse, or it can be a series of operations in which the user sequentially clicks the selected image with the mouse, right-clicks, and clicks the "Output Report" control in the drop-down menu that pops up in the human-computer interaction interface. In a specific example, after the user selects the partial image of the stomach examination to be output, the output can include an image of a color 3D model of the stomach and the selected video frames of the stomach examination, as well as the mapping relationship between the position points in the color 3D model of the stomach and the selected video frames of the stomach examination.

[0104] The output inspection report can be in any suitable format. In one example, the report template for the output inspection report can be determined based on the number of images selected by the user. For example, when the user selects fewer than four images for the output report, an inspection report with a first preset template can be output. Figure 2a This diagram illustrates an inspection report using a first preset template according to an embodiment of this application. As shown, in this first preset template inspection report, the desired 3D model is located on the left side of the report area, and the partial inspection result images are located on the right side of the report area. Furthermore, each partial inspection result image can be mapped to the desired 3D model. For example, the location of a part of the partial inspection result image within the desired 3D model can be marked in the report using connecting lines. As another example, when the user selects more than four images, an inspection report using a second preset template can be output. Figure 2b This diagram illustrates an examination report using a second preset template according to an embodiment of this application. As shown, in this second preset template examination report, the desired 3D model can be located in the center of the report area, and the local examination result images can be located above and below the desired 3D model, respectively. Furthermore, each local examination result image can be mapped to the desired 3D model using connecting lines. This report presentation method not only allows patients to intuitively understand the examination results of each area but also improves the space utilization of the examination report. It solves the problem of unclear mapping between examination images and examination areas and addresses the difficulty patients face in reading reports.

[0105] Using the above method, a report can be generated that includes the desired 3D model of the subject and local examination result images according to the user's actual needs. Furthermore, the examination report also presents the mapping relationship between the location points in the desired 3D model of the subject and the local examination result images, which can help patients intuitively and clearly understand the examination situation of a specific part of themselves, resulting in a better user experience.

[0106] Figure 3 A schematic flowchart illustrating a method for reconstructing a three-dimensional model according to another embodiment of this application is shown. Figure 3As shown, during a gastroscopy, the video stream of the procedure can be acquired in real time as the doctor slowly moves the endoscope inside the patient's body. Then, a three-dimensional model of the stomach can be constructed based on the acquired video stream. Any suitable three-dimensional reconstruction algorithm can be used to construct the patient's actual three-dimensional stomach model. Furthermore, lesions can be marked and located in the image based on the video stream and methods such as intelligent lesion recognition or manual labeling, and the high-definition image information of these lesions can be synchronously mapped and displayed on the actual three-dimensional stomach model. Next, the currently acquired actual three-dimensional stomach model can be registered with a preset standard three-dimensional model to obtain the spatial mapping relationship between the two models. This spatial mapping relationship can be a rotation and translation matrix between corresponding matching points in the matching point set between the actual three-dimensional stomach model and the standard three-dimensional model. Then, based on the correspondence of matching points in the matching point set, the standard three-dimensional model can be downsampled to obtain a sparse standard three-dimensional model. Each feature point in the sparse standard three-dimensional model corresponds to a matching point in the actual three-dimensional stomach model. Based on the spatial mapping relationship between feature points on the actual 3D model of the stomach and feature points on the standard 3D model, actual color information can be mapped onto a sparse standard 3D model. For example, a mapping relationship can be constructed between each feature point on the sparse standard 3D model and a pixel on the video stream image. Point-by-point color mapping can be performed on each feature point on the sparse standard 3D model using a corresponding point coordinate index. For example, when multiple pixels correspond to the same feature point on the sparse standard 3D model, the spatial neighborhood of the pixel in the image can be filtered based on spatial neighborhood information, and the final color information corresponding to the feature point can be determined based on the pixel value of the filtered pixel in the image. For example, a weighted average or evaluation ranking can be performed on the pixel values ​​of multiple pixels, and the final color information corresponding to the feature point can be determined based on the result of the weighted average or evaluation ranking, resulting in a colored standard 3D model of the stomach. Furthermore, the clear color information of lesion areas in the actual 3D model of the stomach can be mapped and filled onto the colored standard 3D model of the stomach to present a clear texture effect of the lesion areas. Finally, a colorized standard 3D model can be presented on the inspection interface for user viewing. For example, areas in the desired 3D model that have not yet mapped the color information of the actual 3D model can be displayed in a preset style, such as light gray, to facilitate user confirmation of the current inspection progress. After the inspection is completed, in response to user actions on locations on the desired 3D model, the image corresponding to that location can be displayed. For example, based on a user's first action, such as selecting an image, an inspection report of the stomach can also be output.The report of the examination results may include an image of a color three-dimensional model of the subject's stomach and the determined stomach examination image, as well as the mapping relationship between the position points in the color three-dimensional stomach model image and the determined stomach examination image.

[0107] According to another aspect of this application, an endoscope system is provided. Figure 4 A schematic block diagram of an endoscope system 400 according to one embodiment of this application is shown. Figure 4 As shown, the endoscope system 400 according to an embodiment of this application includes a light source device 410, a camera device 420, and an image processing device 430.

[0108] A light source device 410 is used to emit illumination light to the subject. A camera device 420 is used to receive light signals reflected from the subject and generate image signals based on the light signals. An image processing device 430 is used to generate multiple images of the subject based on the image signals; construct an actual three-dimensional model of the subject based on the multiple images; register the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and map actual color information onto the standard three-dimensional model based on the spatial mapping relationship to obtain a desired three-dimensional model of the subject, wherein the actual color information is color information extracted from the multiple images.

[0109] According to another aspect of this application, an electronic device is also provided. Figure 5 A schematic block diagram of an electronic device according to an embodiment of this application is shown. The electronic device includes a memory 510 and a processor 520.

[0110] The memory 510 stores computer program instructions for implementing corresponding steps in the method for reconstructing a three-dimensional model according to embodiments of the present application.

[0111] The processor 520 is used to run computer program instructions stored in the memory 510 to perform corresponding steps of the three-dimensional model reconstruction method 100 according to an embodiment of the present application.

[0112] In one embodiment, computer program instructions executed by processor 520 are used to perform the following steps: acquiring multiple images of a subject captured using an endoscope; constructing an actual three-dimensional model of the subject based on the multiple images; registering the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and mapping actual color information onto the standard three-dimensional model based on the spatial mapping relationship to obtain a desired three-dimensional model of the subject, wherein the actual color information is color information extracted from the multiple images.

[0113] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When the program instructions are run by a computer or processor, they are used to execute corresponding steps of the three-dimensional model reconstruction method of the embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0114] In one embodiment, the program instructions are used to perform the following steps at runtime: acquiring multiple images of the subject captured by an endoscope; constructing an actual three-dimensional model of the subject based on the multiple images; registering the actual three-dimensional model to a standard three-dimensional model of the subject to obtain a spatial mapping relationship between the actual three-dimensional model and the standard three-dimensional model; and mapping actual color information onto the standard three-dimensional model based on the spatial mapping relationship to obtain a desired three-dimensional model of the subject, wherein the actual color information is color information extracted from the multiple images.

[0115] Those skilled in the art can understand the implementation methods, principles, and beneficial effects of the aforementioned endoscopic system, electronic device, and storage medium by reading the reconstruction method of the three-dimensional model. For the sake of brevity, further details are omitted here.

[0116] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0119] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0120] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more aspects of the various applications, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0121] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or elements of any method or apparatus so disclosed may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0122] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0123] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0124] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0125] The above are merely specific embodiments or descriptions of specific embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for reconstructing a three-dimensional model, comprising: Acquire multiple images of the subject using an endoscope; Based on the multiple images, construct an actual three-dimensional model of the photographed object; Feature point matching is performed between the feature points on the actual 3D model and the feature points on the standard 3D model of the subject to determine the set of matching points between the actual 3D model and the standard 3D model; Based on the positional correspondence of the matching points in the matching point set, the standard 3D model is downsampled to obtain a sparse standard 3D model, wherein each feature point on the sparse standard 3D model has a corresponding matching point on the actual 3D model. Based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model, the actual color information is mapped onto the sparse standard 3D model to obtain the desired 3D model of the photographed object. The actual color information is color information extracted from the multiple images.

2. The reconstruction method according to claim 1, wherein, The step of mapping the actual color information onto the sparse standard 3D model includes: For each feature point on the sparse standard 3D model Based on the correspondence between the coordinates of the feature point on the sparse standard 3D model and the position of the matching point, a first matching point corresponding to the position of the feature point on the actual 3D model is determined. Determine the pixel in the plurality of images corresponding to the first matching point; and Based on the determined pixels, the color information of the feature point on the sparse standard 3D model is determined.

3. The reconstruction method according to claim 2, wherein, Determining the color information of the feature point on the sparse standard 3D model based on the determined pixels includes: When the first matching point corresponds to multiple pixels, the spatial neighborhood of the determined pixel is filtered to obtain pixel value information of the pixels in the spatial neighborhood; and The color information of the feature point on the sparse standard 3D model is determined based on the pixel value information of the pixels in the spatial neighborhood.

4. The reconstruction method according to claim 2, wherein, The step of mapping the actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model further includes: Based on the pixel value information of the pixels in the determined spatial neighborhood of the pixel, the spatial neighborhood of the feature point on the sparse standard 3D model is filled.

5. The reconstruction method according to claim 2, wherein, The method further includes: Identify abnormal parts of the subject in the plurality of images respectively, and mark the abnormal location regions of the abnormal parts in the plurality of images; and The abnormal location region is mapped onto the actual 3D model to determine the abnormal location region on the actual 3D model; The step of mapping the actual color information onto the sparse standard 3D model based on the positional correspondence between feature points on the sparse standard 3D model and matching points on the actual 3D model further includes: For a first feature point on the sparse standard 3D model, the spatial neighborhood of the first feature point on the sparse standard 3D model is filled, wherein the matching point of the first feature point on the actual 3D model is located in an abnormal location region on the actual 3D model.

6. The reconstruction method according to any one of claims 1 to 5, wherein, The acquisition of multiple images of the subject captured by an endoscope includes: The image is acquired simultaneously with the image of the subject being captured by the endoscope; The steps from constructing the actual three-dimensional model of the photographed object to mapping the actual color information onto the standard three-dimensional model are all performed based on the currently acquired image; The method further includes: The desired 3D model is displayed in real time, wherein areas in the desired 3D model that have not yet mapped the actual color information are displayed or not displayed in a first preset style.

7. The reconstruction method according to claim 6 of claim 5, wherein, Prior to displaying the desired 3D model in real time, the method further includes: Based on the positional correspondence of the matching points, the abnormal location region is mapped onto the desired 3D model to determine the abnormal location region on the desired 3D model; Specifically, when the desired 3D model is displayed in real time, abnormal location regions in the desired 3D model are displayed in a second preset style.

8. The reconstruction method according to any one of claims 1 to 5, wherein, The method further includes: Provide users with a human-computer interaction interface; In response to a user's operation on the human-computer interaction interface for a position point on the desired 3D model, based on the spatial mapping relationship between the actual 3D model and the standard 3D model, a first position point corresponding to the position point on the actual 3D model is determined, an image for extracting color information of the first position point is determined from the plurality of images, and the determined image is displayed.

9. The reconstruction method according to claim 8, wherein, The method further includes: While displaying the determined image, the desired 3D model is also displayed, along with the mapping relationship between the determined image and the corresponding points in the desired 3D model.

10. An endoscope system, comprising: A light source device used to emit illumination light onto the subject; A camera device for receiving light signals returned from the subject and generating image signals based on the light signals; as well as An image processing device is used to generate multiple images of the photographed object based on the image signal; Based on the multiple images, construct an actual three-dimensional model of the photographed object; Feature point matching is performed between the feature points on the actual 3D model and the feature points on the standard 3D model of the subject to determine the set of matching points between the actual 3D model and the standard 3D model; Based on the positional correspondence of the matching points in the matching point set, the standard 3D model is downsampled to obtain a sparse standard 3D model, wherein each feature point on the sparse standard 3D model has a corresponding matching point on the actual 3D model; based on the positional correspondence between the feature points on the sparse standard 3D model and the matching points on the actual 3D model, the actual color information is mapped onto the sparse standard 3D model to obtain the desired 3D model of the photographed object, wherein the actual color information is color information extracted from the multiple images.

11. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the method for reconstructing a three-dimensional model as described in any one of claims 1 to 9.

12. A storage medium on which program instructions are stored, characterized in that, The program instructions, when executed, are used to perform the method for reconstructing the three-dimensional model as described in any one of claims 1 to 9.

Images