Graphics processing method and graphics processing apparatus

Abstract

A method and device for processing graphics can form a rendering image with excellent perspective performance while preserving physical details. The method includes: obtaining a real scene image of a first real object; obtaining a non-real scene image, such as a rendering or three-dimensional reconstruction image, of a second real object located at the back side of the first real object; obtaining perspective viewing area information and determining a perspective viewing area in the real scene image according to the information; extracting surface texture features of a part of the real scene image corresponding to the perspective viewing area to highlight the surface texture features in the perspective viewing area, thereby forming a real scene image after surface texture feature extraction; setting a transparency distribution of the perspective viewing area; and calculating and setting a color distribution of the perspective viewing area according to an included angle between a shooting direction of a camera and a normal direction of each pixel of the real scene image corresponding to the perspective viewing area, thereby obtaining a real scene image after surface texture feature extraction and transparency processing.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and virtual reality. Specifically, this invention relates to a graphics processing method, graphics processing apparatus and system, readable storage medium, and computer program product. Background Technology

[0002] In the fields of graphics processing and computer vision, the 3D representation and rendering of real-world objects is an important research direction. Existing graphics processing methods can render graphics of real objects based on depth information and the camera's field of view to generate images with perspective effects. These methods are commonly used in various applications such as medical imaging, augmented reality (AR), virtual reality (VR), 3D modeling, and game development, allowing users to observe different objects in 3D space through the generated images, thereby achieving a more realistic visual experience and effects. In particular, in the medical field, accurate medical imaging representation is crucial for disease diagnosis, surgical planning, and treatment evaluation.

[0003] In graphics processing or computer vision, one challenge in generating images with perspective effects is properly handling occlusion. Generally, for real-world objects with a front-to-back hierarchical relationship, one approach is to acquire, construct, or input a 3D model of these objects, or to capture a 2D image with depth information using a binocular vision camera (stereo camera), and then process occlusion based on this depth information to create a target image with front-to-back occlusion. In the medical field, creating target images with front-to-back occlusion is extremely valuable. For example, in minimally invasive surgery, correctly overlaying anatomical structures behind (posterior to) exposed tissue surfaces and visualizing the overlay provides more precise intraoperative navigation references, thus facilitating safe surgical procedures and avoiding damage to important structures beneath the tissue surface (such as blood vessels and nerves) or ensuring precise resection margins to prevent residual tumor tissue.

[0004] On the other hand, in the medical field, existing technologies also face challenges in handling human anatomical structures with hierarchical relationships. For example, during preoperative surgical planning, doctors need to clearly observe tissues or organs located behind anterior structures within the patient's body. In this case, specific areas of the anterior tissue structure need to be made transparent, allowing doctors to observe other anatomical structures behind that area. Regarding the transparency of these areas, some solutions propose rendering both anterior and posterior tissues transparent, while others propose setting a perspective window in a specific area of ​​the image corresponding to the anterior tissue and rendering that window transparent. However, the visual effect of the perspective areas created by these solutions is not ideal and is not conducive to user observation. Specifically, regarding the former solution, although rendering both anterior and posterior tissues transparent can improve depth perception of certain surfaces to some extent, in general, rendering two overlapping transparent areas can cause conflicting visual cues at the occlusion points, resulting in poor depth perception for the user. In particular, it can lead to poor accuracy in depth perception, causing users to mistakenly perceive that tissues that should be located at the back appear to be floating in front of tissues located at the front. Regarding the latter approach, if a transparent perspective window is set in a specific area of ​​the image corresponding to the tissue located at the front, the anatomical information of that tissue that was originally located at the perspective window will be lost.

[0005] In fact, the above-mentioned problems exist not only in the medical field, but also in other fields (e.g., equipment maintenance or repair, urban pipeline inspection, etc.). Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] This invention was developed in consideration of the above-mentioned circumstances, and its object is to provide a graphics processing method, as well as a graphics processing apparatus and system, storage medium, and computer program product using this method, capable of generating rendered images with good visual effects. More specifically, it can enhance perspective effects while preserving the physical details of realistic objects.

[0008] Technical solutions adopted to solve technical problems

[0009] This application provides a graphics processing method, characterized in that it includes:

[0010] The first real object is photographed with a camera to obtain a real-world image of the first real object;

[0011] Obtain a non-real-scene image of a second real object, which is usually located behind the first real object;

[0012] Obtain perspective observation area information and determine the perspective observation area in the real scene image based on the perspective observation area information, so as to be able to observe the part of the non-real scene image that corresponds to the perspective observation area;

[0013] The surface texture features of the portion of the real-scene image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a real-scene image after surface texture feature extraction.

[0014] Set the transparency distribution of the perspective observation area; and

[0015] Based on the shooting direction of the camera and the angle between the normal direction of each pixel in the real scene image and the perspective observation area, the color distribution of the perspective observation area is calculated and set to obtain the real scene image after surface texture feature extraction and transparency processing.

[0016] According to the graphic processing method described in this technical solution, firstly, the surface texture features of the first real object in the real-world image corresponding to the perspective observation area are extracted, preserving the details of the surface anatomical tissue of the first real object in the perspective observation area. Next, the transparency distribution of the aforementioned perspective observation area is set. Then, by calculating the angle between the camera's shooting direction and the normal direction of each pixel in the two-dimensional image of the perspective observation area after transparency processing, and further calculating the grayscale and contrast distribution of the perspective observation area based on this angle, color interpolation is performed so that the image located within the perspective observation area can present different color distributions under different shooting angles. As a result, the realism of the image is enhanced, making the rendered image more consistent with human visual experience and simulating a more realistic visual effect. In other words, through the above transparency processing, the perspective performance of the perspective observation area is enhanced. Thus, compared with existing technologies, while preserving the details of the surface anatomical tissue of the first real object located on the front side, the perspective performance of the perspective observation area is enhanced, achieving better virtual-real overlay and penetration effects. Therefore, this technical solution can be applied to fields such as medicine to display the perspective effect of human tissue structure, including the superposition of virtual and real and the penetration effect, making the display of human tissue more three-dimensional, and allowing other tissues and organs behind it to be seen through the penetration display method.

[0017] Optionally, in the graphic processing method described in this application, a fused image is formed by fusing and rendering the real scene image and the non-real scene image based on the spatial information of the real scene image and the non-real scene image, the perspective observation area information and the color distribution information. The fused image is formed by superimposing the real scene image after surface texture feature extraction and transparency processing with the image of the portion of the non-real scene image located behind the perspective observation area.

[0018] According to the graphics processing method described in this technical solution, graphic elements within the perspective observation area, graphic elements of non-realistic graphics located behind the perspective observation area, and graphic elements in the non-perspective observation area of ​​the real-world image can be seamlessly integrated in a spatial sense, thereby achieving a three-dimensional spatial depth fusion rendering effect. Thus, while observing the anatomical details of the first real object, the anatomical details of the second real object located behind the perspective observation area can be clearly observed. More specifically, during the fusion rendering process, texture mapping and normal mapping are added, real-time ray tracing is performed, surface light reflection is simulated, and the spatial depth information of the visible part located in front, i.e., the first real object (such as the actual inner wall of a pipe), and the occluded invisible part at the far end, i.e., the second real object (such as tracheal bundles, arteries and veins, nodules, and other three-dimensional anatomical tissues), is calculated. The far-end model of the perspective observation area (i.e., the three-dimensional model of the second real object), the near-end texture of the perspective observation area, and the surface of the non-perspective observation area are seamlessly integrated in space to achieve a three-dimensional spatial depth fusion rendering effect.

[0019] Optionally, in the graphics processing method described in this application, the spatial information includes depth information.

[0020] The image processing method described in this technical solution can overlay images based on depth information.

[0021] Optionally, setting the transparency distribution of the perspective observation area includes performing a smooth transition process on the perspective observation area.

[0022] According to the graphic processing method described in this technical solution, the transparency distribution of the perspective observation area can have a gradual effect, avoiding large visual abrupt changes at the boundary between the perspective observation area and the external area.

[0023] Optionally, in the graphics processing method described in this application, setting the transparency distribution of the perspective observation area includes: assigning basic transparency to each pixel in the perspective observation area; and assigning additional transparency to each pixel according to the position of each pixel in the perspective observation area.

[0024] According to the graphic processing method described in this technical solution, the transparency of the perspective observation area can have a gradual effect and the range can be controlled, which enhances the user's visual experience and can avoid large visual abrupt changes at the boundary between the perspective observation area and the external area.

[0025] Optionally, in the graphic processing method described in this application, the grayscale and contrast of the perspective observation area are adjusted before the surface texture features are extracted.

[0026] According to the graphic processing method described in this technical solution, by adjusting the grayscale and contrast of the perspective observation area, the threshold of image brightness is expanded, making it easier to extract textures.

[0027] Optionally, in the graphics processing method described in this application, edge detection is performed on the portion of the real-world image corresponding to the perspective observation area before or simultaneously with the extraction of the surface texture features.

[0028] The graphic processing method described in this technical solution can make the textures in the real scene image more clearly displayed, thereby improving the accuracy of texture extraction.

[0029] Optionally, in the graphic processing method described in this application, after forming a real-world image with extracted surface texture features, the black and white pixels in the perspective observation area are color-flipped to make the black pixels white pixels and the white pixels black pixels.

[0030] According to the image processing method described in this technical solution, transparency processing can be achieved for textureless or weakly textured regions. Specifically, in the formed texture image, the texture is represented by black pixels, while textureless or weakly textured regions are mostly represented by white pixels, and white is difficult to make transparent. Therefore, by performing a color inversion process on the black and white pixels in the original texture image, textureless and weakly textured regions are represented by black pixels, which are easier to make transparent. In this way, transparency of textureless and weakly textured regions can be easily achieved. For details on the color inversion process, please refer to step 3C described in the specific embodiment.

[0031] Optionally, in the graphic processing method described in this application, the perspective observation area after the color inversion processing is binarized so that the part of the real scene image corresponding to the perspective observation area is formed as a grayscale image.

[0032] According to the graphic processing method described in this technical solution, a color image is processed to grayscale to form an image with only black and white colors, thereby removing noise.

[0033] Optionally, in the graphic processing method described in this application, after the portion of the real scene image corresponding to the perspective observation area is formed into a grayscale image, the grayscale image is subjected to color balance processing to form a pseudo-color image.

[0034] The graphics processing method described in this technical solution can improve the visualization effect of rendered images. For details regarding color balance processing, please refer to step 6 of the specific implementation method.

[0035] Optionally, in the graphics processing method described in this application, the first real object may be a human anatomical structure.

[0036] On the one hand, according to the graphic processing method described in this technical solution, even if the first real object has a complex and varied human anatomical structure, the specific details of the second real object located behind the first real object can be clearly observed, thereby avoiding "getting lost" during navigation. That is, it can avoid the inability to determine the current specific location due to the shape of the first real object during navigation. Human anatomical structures include various organs and cavities in the human body. Various organs include the heart, lungs, liver, stomach, kidneys, spleen, pancreas, small intestine, large intestine, bladder, uterus, testes, etc., and cavities include digestive cavities (e.g., esophagus), respiratory cavities (e.g., trachea, bronchi, bronchioles), urinary cavities, reproductive cavities, blood vessels, etc. It can be found that human anatomical structures have various shapes, some with complex structures and shapes. For these human anatomical structures, during intraoperative navigation, when the first real object is, for example, the inner wall of the trachea or bronchus, the camera will move within the winding and tortuous cavity. In this situation, if other objects outside the cavity cannot be observed, the observer can easily become lost within the cavity. In other words, as the camera moves through the winding cavity and makes multiple turns, the observer may lose sight of the camera's exact location within the cavity, thus hindering their ability to perform the necessary operations. More specifically, in intraoperative navigation, the surgeon is unable to perform the required surgical procedures because they cannot determine the exact location of the endoscopic equipment, including the camera, within the body cavity. However, according to the image processing method described in this technical solution, a fluoroscopic observation area with good perspective performance is established while preserving the surface texture details of the body cavity. Therefore, during the camera's movement, the observer can clearly observe other anatomical tissues outside the cavity at any time, thus determining the camera's current location within the cavity. This avoids the situation of "getting lost" when navigating within the human anatomical structure.

[0037] On the other hand, according to the graphic processing method described in this technical solution, a certain human anatomical structure and other human structures and tissues located outside or inside it can be viewed in real time through the perspective observation area. For example, when the first real object is the inner wall of a human cavity, the inner wall of the human cavity and other human cavities (e.g., blood vessels), lymph nodes and diseased tissue models located outside the human cavity can be viewed through the perspective observation area, as well as the location coordinates of the area in three-dimensional space, the trend of the surgical planning path and other intraoperative guidance information.

[0038] Optionally, in the graphic processing method described in this application, before acquiring the perspective observation area information, it is determined whether a perspective execution command has been received. If the perspective execution command is not received, the perspective observation area information is not acquired. The perspective execution command can be used to instruct the acquisition of the perspective observation area information.

[0039] According to the graphic processing method described in this technical solution, the perspective observation area can be dynamically turned on or off according to actual needs.

[0040] Optionally, in the graphics processing method described in this application, a virtual navigation object marker is generated. The virtual navigation object marker is formed by graphically processing the camera based on the camera's coordinate information, and the virtual navigation object marker is drawn on the front side of the fused image.

[0041] According to the graphics processing method described in this technical solution, by providing virtual navigation object markers, it is possible to improve the visualization effect while using the virtual navigation object markers to link the current position of the camera in the real scene image with the non-real scene image part located behind the perspective observation area, thereby enhancing the fusion effect.

[0042] Optionally, in the graphics processing method described in this application, the position and range of the perspective observation area are determined according to a pre-set algorithm, or the perspective observation area is determined according to the user's line of sight, or the perspective observation area is determined according to the user's selection of the area of ​​interest, wherein the area of ​​interest is selected manually or by voice, or selected based on eye-tracking data of the gaze point provided on the display device.

[0043] According to the graphics processing method described in this technical solution, the perspective observation area can be dynamically set according to the user's needs. Specifically, in a manual mode, the user can select the area of ​​interest using a mouse or keyboard, or other types of interactive tools. Furthermore, eye-tracking data can be obtained using an eye tracker.

[0044] Furthermore, this application also provides a graphics processing method, characterized in that it includes:

[0045] Obtain the non-real-scene image of the first real object, i.e., the first non-real-scene image;

[0046] Obtain a non-real-scene image of the second real object, i.e., the second non-real-scene image, where the second real object is located behind the first real object;

[0047] A virtual image is formed based on the spatial information of the first non-real-scene graphic and the field of view of the virtual camera;

[0048] Obtain perspective observation area information and determine the perspective observation area in the virtual image based on the perspective observation area information, so as to be able to observe the part of the second non-real scene graphic corresponding to the perspective observation area;

[0049] The surface texture features of the portion of the virtual image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a virtual image after surface texture feature extraction.

[0050] Set the transparency distribution of the perspective observation area; and

[0051] Based on the shooting direction of the camera and the angle between the normal direction of each pixel of the virtual image and the perspective observation area, the color distribution of the perspective observation area is calculated and set to obtain a virtual image after surface texture feature extraction and transparency processing.

[0052] According to the graphics processing method described in this technical solution, a virtual image that retains anatomical details while possessing excellent perspective performance can be obtained.

[0053] Optionally, in the graphic processing method described in the application, a fused image is formed by fusing and rendering the virtual image and the second non-real scene image according to the spatial information, the perspective observation area information, and the color distribution information. The fused image is formed by superimposing the virtual image after surface texture feature extraction and transparency processing with the image of the portion of the second non-real scene image located behind the perspective observation area.

[0054] According to the image processing method described in this technical solution, by superimposing a virtual image (after surface texture feature extraction and transparency processing) with a second non-real-scene image of a second real object, a fused image can be formed. This image retains the anatomical details of the virtual image in front while allowing observation of the second non-real-scene image behind through a perspective observation area with good perspective performance. This fused image is well-suited for preoperative planning or intraoperative navigation. Specifically, during preoperative planning, by setting the virtual camera and its pose and moving the virtual camera, fused images can be continuously formed in the manner described above. During this process, the physician can observe the anatomical structures of the human body behind the perspective observation area in the fused image. Therefore, by continuously adjusting the pose and movement path of the virtual camera, the optimal movement path can be found, thus enabling better preoperative planning. On the other hand, during intraoperative navigation, by simultaneously observing the fused image and the real-scene image, the physician can clearly determine the accurate location of the current surgical equipment, thus providing good assistance for intraoperative navigation.

[0055] Optionally, in the graphics processing method described in this application, before acquiring the perspective observation area, a real-scene image of the first real object is acquired by shooting with a real camera, the poses of the real camera and the virtual camera are acquired, and the poses of the real camera and the virtual camera are compared so that the pose of the virtual camera becomes consistent with the pose of the real camera.

[0056] According to the graphics processing method described in this technical solution, intraoperative three-dimensional spatial registration combining real-time medical images (e.g., real-time images of human tissues and lesions) and virtual images can be achieved. During intraoperative navigation, on the one hand, a virtual image is formed by capturing and rendering non-real-scene graphics (e.g., a three-dimensional model of the inner wall of a human cavity) using a virtual camera; on the other hand, a real-scene image is formed by capturing a first real object (e.g., the inner wall of a real human cavity) using a real camera. Then, by aligning the pose of the virtual camera with the current pose of the real camera, the specific position of the endoscopic device including the real camera within the human cavity can be aligned with the specific position of the virtual camera. Based on this, a virtual image with surface texture feature extraction and transparency processing is formed, and a fused image is formed by fusing and rendering the virtual image and the second non-real-scene graphics, the information of the perspective observation area, and the color distribution information. The fused image at this time has a transparent perspective observation area that retains the surface texture features of the virtual image, and some details of the second non-real-scene graphics that can be observed through the perspective observation area. In this way, by comparing the fused images with the real-world images, the surgeon can understand the specific situation of other human anatomical structures outside the body cavity where the endoscope is currently located, and determine whether the endoscope has reached the target site, thus enabling precise navigation and positioning of the endoscope during the operation.

[0057] Optionally, in the graphics processing method described in this application, the virtual image is replaced with the real scene image to obtain the real scene image after surface texture feature extraction and transparency processing.

[0058] According to the image processing method described in this technical solution, real-time medical images and virtual navigation paths, along with images of human tissues and lesions, can be combined intraoperatively to achieve three-dimensional spatial registration of virtual and real images. Furthermore, based on the three-dimensional spatial registration of actual and virtual images, virtual-real fusion can be achieved, exhibiting strong stereoscopic spatial perception capabilities and assisting doctors in performing precise surgical treatments conveniently and intuitively.

[0059] Invention Effects

[0060] According to the graphics processing method described in this application, it is possible to generate rendered images that retain physical details while having excellent perspective effects, resulting in a strong augmented reality effect. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating the graphic processing method of the first embodiment of this application.

[0062] Figure 2It is a real-world image showing the inner wall space of the trachea, which is an example of the first real object.

[0063] Figure 3 This is a schematic diagram representing an example of a second real object captured by a camera in a certain pose, namely a three-dimensional model located outside the space of the inner wall of the trachea.

[0064] Figure 4 It is a schematic diagram showing the spatial relationship between a real-world image of the first real object and a 3D model of the second real object.

[0065] Figure 5(a) is a virtual image generated from a non-realistic graphic based on a local area of ​​the inner wall space of the trachea, an example of a first real object.

[0066] Figure 5(b) is a schematic diagram showing the results of adjusting the grayscale and contrast of a local area of ​​the tracheal wall space shown in Figure 5(a).

[0067] Figure 5(c) is a schematic diagram showing the results of texture extraction of a local area of ​​the tracheal wall space shown in Figure 5(b).

[0068] Figure 5(d) is a schematic diagram showing the result of color inversion processing on a local area of ​​the tracheal wall space shown in Figure 5(c).

[0069] Figure 5(e) is a schematic diagram showing the result of binarization processing of a local area of ​​the tracheal wall space shown in Figure 5(d).

[0070] Figure 5(f) is a schematic diagram showing the result of color balancing processing on a local area of ​​the tracheal wall space shown in Figure 5(e).

[0071] Figure 6 This is a flowchart illustrating another variation of the graphic processing method according to the first embodiment of this application.

[0072] Figure 7 It means through Figure 6 A schematic diagram of the fused image formed by the graphic processing method described in the modified example.

[0073] Figure 8 This is a flowchart illustrating yet another variation of the graphic processing method according to the first embodiment of this application.

[0074] Figure 9 This is a schematic diagram representing a fused image containing graphical controls.

[0075] Figure 10 This is a flowchart illustrating the graphic processing method of the second embodiment of this application.

[0076] Figure 11 This is a flowchart illustrating an example of an image registration method. Detailed Implementation

[0077] The following is for reference Figures 1 to 4 The main steps of the graphic processing method according to one embodiment of this application are described below. It should be noted that the steps of the graphic processing method described in this embodiment are all examples. As long as the desired technical effect can be achieved, the order of each step can be appropriately changed, and other steps can be added to these steps.

[0078] Figure 1 A flowchart of the image processing method according to the first embodiment of this application is shown. First, in step S1, a real-world image of the first real-world object is obtained by taking a picture of the first real-world object with a camera. The "first real-world object" referred to here refers to an object that actually exists in reality, such as human anatomical structures, machinery and equipment, buildings, urban pipelines, etc. Figure 2 A real-view image of the inner wall space of the trachea, serving as an example of a first real object, is shown. Specifically, the real-view image acquired by photographing the first real object using a stereo camera or other stereoscopic vision device includes not only two-dimensional coordinate information but also depth information for each pixel. This depth information can be used to determine the front-back positional relationship between the first real object and other real objects in three-dimensional space. Furthermore, the term "real-view image" here includes not only static images but also dynamic video images. Optionally, after acquiring the real-view image of the first real object, the image is displayed on a display device for determining and acquiring the perspective observation area in subsequent steps. After completing step S1, proceed to step S2.

[0079] In step S2, a non-real-scene graphic of the second real object is obtained. The second real object is located behind the first real object. It should be noted that "graphic" is a broad concept, encompassing various graphic and image forms used to display, simulate, or convey information. In this application, "graphic" refers to a two-dimensional image or a three-dimensional model. Based on this, in the implementation, "non-real-scene graphic" refers to a three-dimensional model formed by three-dimensional reconstruction of the second real object based on its three-dimensional coordinate information; it is a non-realistic virtual graphic. The non-real-scene graphic can be a 3D reconstructed CG image. In clinical practice, it can be a corresponding MR / CT / US structure or other structures not directly visible to the naked eye obtained from imaging. In non-clinical fields, it can be a 3D model structure or other functional structures not directly visible to the naked eye. For ease of understanding, the following description uses the tracheal endothelial space as an example of the first real object and human anatomical structures located outside the tracheal endothelial space (e.g., tracheal bundles, arteries and veins, nodules, etc.) as an example of the second real object. As mentioned above, Figure 2 The image shows a real-world view of the inner wall space of the trachea, taken intraoperatively by a real camera in a specific pose, serving as the first real object. Figure 3 The image shows a schematic diagram of a 3D model of a second real object captured by a virtual camera in a certain pose. Specifically, it depicts a human anatomical structure (hereinafter sometimes referred to as the posterior human anatomy) located on the outer (more precisely, posterior) side of the tracheal tube's inner wall space. More specifically... Figure 3 The 3D model of the posterior human anatomy shown is formed by combining medical imaging data of the acquired posterior human anatomy. Specifically, medical imaging data is typically acquired through a series of two-dimensional images (two-dimensional image slices), which show cross-sections of the internal structures of the human body (in this embodiment, the space within tubes) at different depths. For example, CT (computed tomography) and MRI (magnetic resonance imaging) scans generate multiple parallel two-dimensional slice images, each slice representing a specific layer of the patient's body. The process of combining these two-dimensional slice images to form a 3D model is called 3D reconstruction, a technique that uses information from multiple two-dimensional slice images to simulate the three-dimensional shape of body structures. Generally, the 3D reconstruction process includes image acquisition, image preprocessing, image segmentation, 3D model construction, and visualization. In the image acquisition step, a series of two-dimensional slice images are obtained by applying medical imaging techniques such as CT and MRI to the patient's body. In the image preprocessing step, the acquired two-dimensional slice images undergo denoising, contrast enhancement, and correction to improve image quality. Then, in the image segmentation step, the anatomical structures of interest, such as organs, tumors, and blood vessels, are identified and separated. Figure 4 This diagram illustrates the spatial relationship between a real-world image of the first real object and a 3D model of the second real object. Figure 4 In this study, the primary real-world object is the inner wall of the trachea, and the image of this inner wall is captured by an endoscopic device containing a real camera moving within the trachea. Additionally, in Figure 4 In this context, the second real-world object is the human anatomical structure located in the space outside the inner wall of the trachea in the photographed image; it is a three-dimensional model formed by combining acquired medical imaging data. Figure 4 It can be observed that, from the camera's shooting perspective, the inner wall of the trachea, which is the first real object, is located in front of the human anatomical structures in the space outside the inner wall of the trachea, which is the second real object. After completing step S2, proceed to step S3.

[0080] In step S3, perspective observation area information is acquired, and the perspective observation area is determined in the real-world image based on this information. The "perspective observation area information" referred to here refers to information related to the perspective observation area of ​​the display image drawn on the display device from the real-world image to be transparent, including the location (e.g., coordinates of the center point), extent, and shape of the perspective observation area. After acquiring the aforementioned perspective observation area information, the graphics processing program or device determines the perspective observation area in the real-world image displayed on the display device based on this information. Figure 4As shown, through this perspective observation area, graphic elements of non-real-scene graphics located behind the real-scene image can be observed. The perspective observation area can be a fixed area on the pre-defined display device, or it can be set by the user, or it can be automatically set according to pre-defined rules or algorithms. In the case of user-defined settings, the perspective observation area can be determined, for example, based on the user's line of sight relative to the display device. This can be achieved through eye-tracking technology based on an eye tracker (eye tracking sees points on the graphic corresponding to a ray originating from the eye). Regarding eye-tracking technology, it can be implemented through methods such as non-invasive video analysis, infrared light reflection, and electrophysiological methods. Non-invasive video analysis uses a camera to capture video images of the eyes and analyzes the pupil center and line of sight direction. Infrared light reflection uses an infrared light source and a camera to capture reflected light from the eyes to determine the line of sight direction. Electrophysiological methods measure bioelectrical signals related to eye movements and track eye movements through the potential difference in the skin around the eyes. Furthermore, in the case of user-defined settings, the perspective observation area can also be selected by operating a mouse or keyboard; for example, the perspective observation area can be set by selecting a box on the display device with a mouse. In addition, operation can also be performed via voice recognition when the user makes the settings independently. Furthermore, regarding automatic setting based on whether preset conditions are met, for example, when the graphic processing method of this application is applied to preoperative and intraoperative registration (described in detail below), if the distance between the surgical instrument and the inner wall of the trachea is small, it is determined that the preset conditions are met, and a specific area centered on the surgical instrument is set as the fluoroscopic observation area, thereby facilitating lesion localization and corresponding treatment. Furthermore, the number of fluoroscopic observation areas is not limited to one; it can be multiple. For example, multiple fluoroscopic observation areas can be set by repeatedly selecting boxes on the real-world image with a mouse, or the areas in the vertical, horizontal, and vertical directions of the surgical instrument during surgery can be set as multiple fluoroscopic observation areas. It should be noted that although step S3 is located after step S1 in this embodiment, it is not limited to this; step S3 can be performed before step S1 or simultaneously with step S1.

[0081] In step S4, the surface texture features of the first real object in the portion of the real-scene image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a real-scene image after surface texture feature extraction. In this embodiment, various well-known texture extraction algorithms can be used to extract surface texture features. For example, as texture extraction algorithms, well-known algorithms such as gray-level co-occurrence matrix, local binary mode, Gabori filter, wavelet transform, directional gradient histogram, and Fourier transform can be used.

[0082] In step S5, the transparency distribution of the determined perspective observation area is set. One method for setting the transparency distribution is to automatically assign a separate alpha value (i.e., transparency value) to each pixel of the image as a basic (initial) transparency value according to specific algorithm rules. Alternatively, a uniform alpha value can be automatically assigned to the entire image portion as a basic transparency value according to specific algorithm rules. Users can also manually assign a uniform alpha value to the entire image portion as a basic transparency value based on their experience, or manually and individually assign alpha values ​​to each pixel of the entire image as basic transparency values. However, considering the relatively obvious boundary between the transparent perspective observation area and the opaque area outside it, to avoid a decrease in visual effect caused by a sudden change from transparent to opaque, it is preferable to perform a smooth transition processing on the image portion of the perspective observation area to give the transparency distribution of the perspective observation area a gradual effect. The following uses the SmoothStep function in Matlab as an example to explain the smooth transition processing of transparency. After smooth transition processing using the SmoothStep function, the final transparency value of each pixel can be expressed by the following formula:

[0083] Opacity = Result1 + BaseOpacity

[0084] Where Opacity represents the final opacity value of each pixel, BaseOpacity represents the basic opacity value of each pixel (i.e., the base opacity), and Result1 represents the opacity correction value assigned to each pixel by the smooth transition processing (i.e., the additional opacity), which can be expressed by the following formula:

[0085] Result1=1-SmoothStep(0.0, 1.0, clamp(distance(GradientCenter, UV) / Radius, 0.0, 1.0)*Density)

[0086] in:

[0087] GradientCenter represents the gradient center, which in this embodiment is the center of the perspective observation area;

[0088] UV represents the two-dimensional coordinate values ​​of each pixel in the perspective viewing area of ​​a real-world image;

[0089] distance(GradientCenter,UV) represents the distance between each pixel in the perspective viewing area and the center of the perspective viewing area, i.e., the gradient center;

[0090] Radius represents the gradient radius, which is the distance from the center of the gradient to the point where the gradient reaches its maximum effect; in other words, the distance from the center of the perspective viewing area to its edge.

[0091] clamp(value, min, max) is a function that restricts value to between min and max. When value is less than min, value is set to min, and when value is greater than max, value is set to max.

[0092] SmoothStep(a, b, t) is a smooth step function that smoothly interpolates the parameter t between given values ​​a and b, aiming to create a transition from a to b. t is a parameter that varies from 0 to 1, representing the progress of the interpolation.

[0093] Density represents the density of the gradient, used to control the rate at which the gradient changes from the center outwards.

[0094] The above example uses the SmoothStep function in Matlab to illustrate how to smoothly transition the perspective viewing area, but the specific implementation method is not limited to this. For example, the following function can be used to achieve the same purpose:

[0095] f(x,y,Radius,Density)=1-Gradient(x,y,Radius,Density)^Power+BaseOpacity.

[0096] in:

[0097] f(x,y,Radius,Density) returns the final transparency value of the pixel at coordinates (x,y);

[0098] x, y represent the coordinates of the center position of the perspective observation area;

[0099] Radius represents the equivalent radius of the perspective viewing area (i.e., the radius of the circle when a non-circular perspective viewing area is equivalent to a circle of the same area).

[0100] Density represents the density of the gradient, which is used to control the rate at which the gradient changes from the center outwards;

[0101] Power represents the exponential parameter;

[0102] 1-Gradient(x,y,Radius,Density)^Power represents the added transparency;

[0103] BaseOpacity represents basic transparency.

[0104] Through the above processing, the transparency of the perspective viewing area can be made to radiate outward from the center of the area with a gradual effect, softening the boundary between the perspective viewing area and the non-transparent areas outside it, thus improving the visual effect. Therefore, the perspective viewing area, the gradual effect, and its range can be dynamically adjusted.

[0105] After completing step S5, proceed to step S6.

[0106] In step S6, the color distribution of the perspective observation area is calculated based on the angle between the camera's shooting direction and the normal directions of each pixel in the perspective observation area of ​​the two-dimensional image (i.e., the real-world image) displayed on the display device, to obtain the real-world image after surface texture feature extraction and transparency processing. For example, firstly, based on the two-dimensional coordinate values ​​of each pixel in the perspective observation area on the two-dimensional image displayed on the display device, the depth information of the part of the three-dimensional model corresponding to each pixel in the perspective observation area is obtained, and the normal direction of each pixel in the perspective observation area is determined based on the changes in the depth information of the aforementioned part. However, the determination of the normal direction of each pixel is not limited to the above method; other methods can also be used to calculate the normal vector of each pixel to determine the normal direction of each pixel. Next, the angle between the vector corresponding to the camera's shooting direction and the normal vector of each pixel is calculated. After determining the angle between the normal vector of each pixel and the vector corresponding to the shooting direction, the color of each pixel in the perspective observation area is calculated based on these angles, thereby forming the color distribution of the perspective observation area. Specifically, for each pixel in the perspective observation area, the region type corresponding to each pixel is first determined using the following mathematical formula:

[0107] g(x,y)=(1-CameraVector*PixelNormalWs(x,y)) Q *20.

[0108] in:

[0109] g(x,y) returns the region type corresponding to the pixel with coordinates (x,y);

[0110] CameraVector represents the direction vector of the camera's shooting direction;

[0111] PixelNormalWs(x,y) represents the normal vector of the pixel with coordinates (x,y);

[0112] Q represents the exponential parameter.

[0113] In the above mathematical formula, by performing a dot product operation on the camera's direction vector and the normal vector of each pixel, the concave and convex regions and flat regions in the first real object can be distinguished based on the result of the dot product operation. The value of the dot product operation is between -1 and 1. Specifically, when the value of the dot product operation is between -1 and 0, the region corresponding to the pixel can be considered a convex region; when the value of the dot product operation is between 0 and 1, the region corresponding to the pixel can be considered a concave region; and when the value of the dot product operation is very close to 0, the region corresponding to the pixel can be considered a flat region. Furthermore, based on the above dot product operation, by subtracting the dot product operation value from 1, the range of the value can be changed from -1 to 1 to 0 to 2, where 0 to 1 corresponds to concave regions and 1 to 2 corresponds to convex regions. On this basis, by performing a power operation (Q-th power operation) on 1-CameraVector*PixelNormalWs(x,y) and further multiplying it by a magnification factor (e.g., 20 in the above mathematical formula), the concave and convex regions in the first real object can be further highlighted.

[0114] Next, based on determining the region type corresponding to each pixel in the perspective observation area, the color of each pixel is calculated and set according to its corresponding region type. For example, the following preset mathematical formula in Matlab can be used to determine the color of each pixel:

[0115] Color(x,y)=lerp((0,0,0),(0,0,1),g(x,y))*baseTexture

[0116] in:

[0117] Color(x,y) returns the color of the pixel at coordinates (x,y);

[0118] (0,0,0) represents black in the RGB system;

[0119] (0,0,1) represents blue in the RGB system;

[0120] baseTexture represents the base texture value of each pixel;

[0121] The lerp function is a linear interpolation function used to smoothly transition between two values. In the above mathematical expression, it interpolates between (0,0,0) representing black and (0,0,1) representing blue.

[0122] In the above mathematical formula, if the value of g(x,y) is closer to 0, it indicates that the region type of the pixel corresponding to that coordinate is a flat region. The function Color(x,y) returns and assigns a lower grayscale black to this pixel, with an RGB value of (0,0,0). On the other hand, if the value of g(x,y) is closer to 0, it indicates that the camera's shooting direction vector is closer to orthogonal to the pixel's normal vector, indicating that the region type of the pixel corresponding to that coordinate is a concave or convex region. The function Color(x,y) returns and assigns a higher grayscale blue to this pixel, with an RGB value of (0,0,1).

[0123] Therefore, using the above mathematical formula, for pixels whose normal direction is nearly orthogonal to the camera's shooting direction, assigning a higher grayscale blue to their original texture pixel values ​​allows these pixels to be displayed as highlights. Conversely, for pixels whose normal direction is nearly parallel to the camera's shooting direction, assigning a lower grayscale black to their original texture pixel values ​​allows these pixels to be displayed as non-highlighted pixels. This enhances the perspective effect in the perspective viewing area. Generally, pixels whose normal direction is nearly orthogonal to the camera's shooting direction correspond to the parts of the perspective viewing area (of the real-world image) with salient features (i.e., more detail), while pixels whose normal direction is nearly parallel to the camera's shooting direction correspond to the parts of the perspective viewing area (of the real-world image) without salient features (i.e., less detail). Therefore, by displaying some pixels as highlights and others as non-highlighted pixels, a strong contrast between light and dark can be created. As a result, in the perspective viewing area, information about a second real-world object located behind it can be displayed while preserving the main details of the first real-world object. On the other hand, by extracting the camera orientation vector and the model normal vector (or the pixel orientation vector) data, the spatial undulation of the near visible area of ​​the 3D model can be calculated in real time, thereby enabling the acquisition of the model's lateral edge effect.

[0124] It should be noted that in the above mathematical formula, blue was used as the color with a high gray level, but it is not limited to this. Considering that black has the lowest gray level in the RGB system, other gray levels can also be used, as long as they can create a strong contrast between light and dark.

[0125] In step S7, a fused image is formed by rendering the real scene image and the non-real scene image based on spatial information including depth information, material information, texture information, light source information, perspective observation area information, color distribution information of the perspective observation area, and the field of view of the camera. The fused image is formed by superimposing the real scene image after surface texture feature extraction and transparency processing with the image of the non-real scene image located behind the perspective observation area.

[0126] According to the graphics processing method described in this embodiment, an image can be provided that includes a perspective viewing area with smooth transparency distribution and brightness distribution conforming to human visual perception, which can clearly display information of the real object located behind while retaining the main details of the real object located in the foreground. Furthermore, according to the graphics processing method described in this embodiment, by adding texture mapping and normal mapping, performing real-time ray tracing, simulating surface light reflection, and calculating the spatial depth information of the near-visible part (i.e., the real scene image, such as the inner wall of a pipe) and the far-occluded invisible 3D model (i.e., non-real scene graphics, such as tracheal bundles, arteries and veins, nodules, and other 3D anatomical tissues), the surfaces of the far-end model (i.e., non-real scene graphics), the near-end texture (i.e., the texture located in the perspective viewing area in the real scene image), and the non-perspective area (i.e., the image portion outside the perspective viewing area in the real scene image) of the perspective viewing area are spatially seamlessly blended, achieving a three-dimensional spatial depth and virtual-real fusion rendering effect.

[0127] Based on the above implementation methods, the following references Figures 5(a) to 5(f) A variation of the graphic processing method described in the above embodiments will be explained. To avoid repetition, this variation will focus on the differences between it and the above embodiments. Furthermore, while the above embodiments used a real-world image of the first real object as the processing object, this method is not limited to this; virtual images generated from non-real-world graphics of the first real object can also be used as processing objects. In other words, this variation applies not only to real-world images of the first real object but also to virtual images of the first real object. Figure 5(a) shows a virtual image generated from a non-real-world graphic of a local area of ​​the tracheal wall space, an example of the first real object. That is, although the example described here uses a virtual image of the first real object as the processing object, it is equally applicable to real-world images of the first real object.

[0128] As a difference from the above-described embodiments, in this modified example, step S3A can be executed before step S4, step S3B can be executed before or simultaneously with step S4, and steps S3C, S3D and S3E can be executed sequentially after step S4 is completed.

[0129] Specifically, in step S3A, as shown in Figure 5(b), before extracting the surface texture features, the grayscale and contrast of the perspective observation area are adjusted to make the perspective observation area brighter. When the first real object is the internal space of the trachea shown in Figure 5(a), such an internal space is often dark, resulting in indistinct or difficult-to-identify texture features. Therefore, to facilitate subsequent texture extraction and analysis, for example, the amount of light illuminating the inner wall of the tube can be increased by enhancing the luminous intensity of the light source, thereby making the originally indistinct or difficult-to-identify texture features stand out. However, it should be noted that step S3A is not a mandatory step, but rather an optional one. That is, when the environment in which the first real object is located is sufficiently bright, it is not necessary to adjust the grayscale and contrast.

[0130] In step S3B, edge detection is performed on the image portion of the perspective observation area before or simultaneously with the extraction of surface texture features. Sometimes, the quality of 2D images rendered from 3D models is low and contains a lot of noise. If surface texture extraction is directly performed on such 2D images, the extraction accuracy will not meet the specified requirements. Therefore, edge detection is performed on the image portion of the perspective observation area before surface texture extraction, resulting in the state shown in Figure 5(c). It should be noted that the purpose of (texture) edge detection is to identify the edges or boundaries in the texture pattern, thereby helping to highlight the boundaries and features of the texture and improving the accuracy and effect of texture extraction. Therefore, as shown in Figure 5(c), by performing edge detection on the image portion of the perspective observation area, the texture in that part of the image can be displayed more clearly. It should be noted that step 3B is also not a necessary step, but an optional one. When the quality of the 2D image formed from the 3D model is high and the noise is low, or when the accuracy of the texture extraction algorithm does not depend on edge detection, it is not necessary to perform edge detection on the image portion of the perspective observation area.

[0131] In step 3C, after forming the real-world image with extracted surface texture features, the black and white pixels (i.e., the grayscale of the image pixels) in the perspective observation area are color-flipped to turn black pixels (i.e., low-brightness pixels) into white pixels (i.e., high-brightness pixels) and white pixels into black pixels. In computer graphics and graphics processing, white pixels are generally considered to be completely opaque because white is typically composed of the maximum intensity (value of 255) of the red, green, and blue channels in the RGB color model. Therefore, white pixels need to be color-processed before rendering transparent areas. As shown in Figure 5(c), the texture image of the perspective observation area formed after texture extraction contains a large number of white pixels. Without color processing of these white pixels, it is difficult to render transparent areas. Therefore, in this embodiment, by performing color inversion processing on the white and black pixels in the texture image shown in FIG5(c), the texture features originally displayed by black pixels are transformed into those displayed by white pixels, and vice versa, thereby forming the color-inverted texture image shown in FIG5(d). In this way, good contrast can be maintained while preparing for the rendering of transparent areas.

[0132] In step 3D, after the color inversion process is completed, the perspective observation area after color inversion is processed into black and white (also known as binarization), so that the part of the real scene image corresponding to the perspective observation area is formed into a black and white image, i.e., a grayscale image, as shown in Figure 5(e). By processing the perspective observation area after surface texture feature extraction into black and white, compared with the perspective observation area before processing, the black and white image requires less storage space than the color image, thus reducing data storage and contributing to data compression. In addition, it can also remove noise.

[0133] In step 3E, after the portion of the real-world image corresponding to the perspective observation area is formed into a grayscale image, color balancing processing is performed on the grayscale image to form a pseudo-color image. Figure 5(f) shows the pseudo-color image formed after color balancing processing of the grayscale image. In this example, combining the color texture of the inner wall of the tube and the color of the three-dimensional model of the human anatomy structure located on the rear side, all white pixels in the grayscale image are converted to blue. This enhances the visual effect. It should also be noted that step 2E is not a necessary step, but an optional one. When the black and white texture image is relatively coordinated with the color texture of the surrounding inner wall of the tube and the color of the three-dimensional model of the human anatomy structure on the rear side, since a good visual effect is already achieved, the action of step 3E is not required. In addition, it should be noted that when step 3E is included, the same color (e.g., blue) used in the color balancing processing in step S6 can be used when determining the color distribution.

[0134] Figure 6 A flowchart illustrating another variation of the graphics processing method described in this embodiment is shown. The difference from the above embodiment lies in the inclusion of step S8. In step S8, a virtual navigation object marker is drawn in front of the fused image, which is displayed relative to the image formed by steps S1 to S7, containing a perspective viewing area with a certain degree of transparency and a portion of a three-dimensional model located behind that perspective viewing area. The virtual navigation object marker is a marker formed by graphically processing the camera based on the camera's coordinate information. Figure 7 A schematic diagram of a fused image formed by the graphics processing method described in this variation is shown, in which an arrow marker, an example of a virtual navigation object, is formed. By forming a virtual navigation object marker on the fused image, the user's gaze can be guided when a video image is formed by continuously moving the camera, thereby achieving a better visual navigation effect. It should be noted that in this variation, a virtual image of the first real object is used for illustration, but it is not limited to this and can also be applied to the case of a real-world image of the first real object.

[0135] Figure 8A flowchart of another variation of the graphics processing method described in this embodiment is shown. The difference from the above embodiment is the inclusion of step SA. Specifically, in step SA, before acquiring perspective viewing area information, or before setting the transparency distribution of the perspective viewing area, or before setting the color distribution of the perspective viewing area, or before rendering and forming a blended image, a determination is made as to whether a perspective execution command has been received, and whether to execute perspective display. If the command is not to execute perspective display, the perspective graphics processing work ends. In this way, image rendering can be flexibly performed based on whether the user needs to observe information about graphics located in the background through the perspective viewing area, avoiding unnecessary waste of computational resources. Figure 9 A schematic diagram of a fused image of a graphical control is shown, illustrating one implementation of the functionality described in step SA. For example... Figure 9 As shown, in Figure 9 A dynamic switch control is drawn in the upper left corner, which can be turned on or off manually by touching it. This dynamic switch control is associated with a callback procedure in the main program containing steps S1 to S6. When the user clicks the dynamic switch control, the GUI thread starts the callback procedure. For example, if the user clicks the dynamic switch control to turn it on, the GUI thread starts the callback procedure, which sends a perspective execution command to the main program. Upon receiving the perspective execution command, the main program executes the steps related to the transparency operation of the perspective viewing area. On the other hand, if the user clicks the dynamic switch control again to turn it off, the GUI thread does not start the callback procedure, and therefore does not send a perspective execution command to the main program. Without receiving the perspective execution command, the main program does not execute the steps related to the transparency operation of the perspective viewing area.

[0136] By setting a switch control, the size and position of the fluoroscopic observation area can be dynamically adjusted, and the model of blood vessels, lymph nodes and lesions (a second real object) outside the tube (an example of the first real object) can be viewed in real time, as well as the spatial coordinates of the area, the trend of the surgical planning path and other intraoperative guidance information.

[0137] In the first embodiment and its variations described above, an example was given where one of the first and second real objects is the inner wall space of the trachea located on the anterior side, and the other is another human anatomical structure located on the posterior side. However, this is not a limitation; the first real object can also be any other human cavity other than the trachea, or any other human anatomical structure other than a human cavity. Furthermore, in the above embodiment and its variations, an example was given where the camera is located inside a human anatomical structure such as the inner wall space of the trachea, which is the first real object. However, this is not a limitation; the camera can also be located on the lateral side of the trachea. Thus, using the image processing method described in the above embodiment and its variations, the internal tissue structure of the trachea can be observed through the perspective observation area. For example, when the first real object is a non-cavity type human anatomical structure such as the uterus, lesions such as uterine fibroids may sometimes grow inside or outside the uterus. By implementing the image processing method described in the above embodiment and its variations, regardless of whether the camera is located on the lateral or lateral side of the uterus, the situation on the other side can be observed through the perspective observation area. Furthermore, the application of this application is not limited to the medical field, but can also be used in areas such as urban pipeline health monitoring. Additionally, the shapes of the first and second real objects are not limited to tubular or internal space.

[0138] Figure 10 A flowchart of the graphic processing method according to the second embodiment of this application is shown. The difference from the above embodiments is that, in this embodiment, instead of using a real-world image of the first real object as the processing object, a non-real-world image of the first real object, i.e., a first non-real-world image, is used as the processing object. Furthermore, considering that some of the technical descriptions overlap with the first embodiment described above, only the differences between this embodiment and the first embodiment will be explained here.

[0139] First, in step ST100, a non-real-scene image of the first real object is obtained, namely the first non-real-scene image.

[0140] Next, in step ST200, in the same manner as in the above embodiment, a non-real-scene image of the second real object, namely the second non-real-scene image, is obtained.

[0141] Then, in step ST300, a virtual image is formed based on the spatial information of the first real object and the field of view of the virtual camera. Figure 5(a) shows a virtual image generated from a non-real-scene graphic of the inner wall space of the trachea, an example of the first real object.

[0142] Subsequently, in step ST400, perspective observation area information is obtained and a perspective observation area is determined in the virtual image based on the perspective observation area information, so as to be able to observe the part of the second non-real scene graphic that corresponds to the perspective observation area.

[0143] Next, in step ST500, the surface texture features of the portion of the virtual image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a virtual image after surface texture feature extraction.

[0144] Then, in step ST600, the transparency distribution of the perspective observation area is set.

[0145] In step ST700, the color distribution of the perspective observation area is calculated and set according to the shooting direction of the camera and the angle between the normal direction of each pixel of the virtual image and the perspective observation area, so as to obtain the virtual image after surface texture feature extraction and transparency processing.

[0146] In step ST800, a fused image is formed by fusing and rendering based on the spatial information of the first non-real scene graphic and the second non-real scene graphic, the perspective observation area information, and the color distribution information. The fused image is formed by superimposing the virtual image after surface texture feature extraction and transparency processing with the image of the part of the second non-real scene graphic located behind the perspective observation area.

[0147] It should be noted that the variations of the first embodiment described above are all applicable to this second embodiment.

[0148] The graphics processing method described in this embodiment and its variations can be well applied to preoperative surgical planning, intraoperative surgical navigation, and preoperative and intraoperative registration, and can also be well applied to medical imaging research. Specifically, regarding preoperative and intraoperative registration, for example, firstly, a real-world image of the first real object in its actual pose and the actual pose information of the real camera are acquired, and an actual depth image containing the real-world image's real-world information is generated based on the actual pose of the real camera. Next, based on the preoperative tracheal tree 3D reconstruction result (i.e., the 3D model of the first real object), a virtual endoscopic depth image, i.e., a virtual depth image, is generated based on the virtual camera's pose. Then, a normalized cross-joint similarity metric is calculated between the virtual image and the depth information of the intraoperative real endoscopic image, i.e., the actual depth image. In other words, the similarity between the real-time image and the virtual image is evaluated based on the depth information of the actual depth image and the virtual depth image. Based on this, according to the evaluation results, the similarity between the real and virtual effects is improved by continuously improving the virtual environment camera pose, thereby achieving camera pose estimation. The registration algorithm utilizes the RefineNet model to extract and match features from two consecutive frames of bronchoscopic views to estimate the motion and position of the endoscope. It also incorporates a cross-modal template image matching algorithm to eliminate drift caused by registration between consecutive frames. Specifically, RefineNet uses ResNet as the backbone to extract feature representations, employs ReconDecoder to predict the depth map, and uses Pose Decoder to decode the relative displacement and rotation angles at two time points from the features obtained from the encoder. The design of the Detail Encoder and Detail Decoder further improves depth estimation, refining image features through a detailed encoder and decoder network. The template image matching algorithm measures the similarity between the real-time image and the template image through edge detection and normalized cross-correlation index, and uses a boundary-constrained global optimization method, BOBYQA, to find the optimal alignment parameters. It should be noted that the above registration algorithm is only one example; other registration algorithms can be used as long as preoperative and intraoperative registration can be achieved. Figure 11 The flowchart of the above example of the image registration method is shown.

[0149] Based on the image registration method described above, it is possible to combine real-time medical images (e.g., real-time images of human tissues and lesions) and virtual images for three-dimensional spatial registration during surgery.

[0150] Furthermore, based on the three-dimensional spatial registration of the aforementioned real-time and virtual images, by replacing the real-scene image of the first real object (a virtual image formed based on a three-dimensional model) with the real-time image, i.e., the real image, and by fusing and rendering it with the non-real-scene image of the second real object according to the graphics processing method described above, virtual-real fusion can be further achieved. It has a strong ability to perceive three-dimensional space and can achieve precise intraoperative navigation and positioning, thereby assisting doctors to perform precise surgical treatment conveniently and intuitively.

[0151] The present invention also provides a graphics processing apparatus capable of implementing the graphics processing method of the first embodiment and its modifications described above. The graphics processing apparatus includes an image acquisition module, a graphics acquisition module, a region determination module, a feature extraction module, a transparency setting module, a color setting module, and a rendering module.

[0152] The image processing module acquires a real-scene image of the first real object as described in step S1 above. The real-scene image of the first real object is acquired, for example, by taking a picture of the first real object with a camera.

[0153] The image acquisition module acquires a non-real-scene image of the second real object as described in step S2 above, wherein the second real object is located behind the first real object.

[0154] The region determination module acquires perspective observation region information as described in step S3 and determines the perspective observation region in the real scene image based on the perspective observation region information.

[0155] As described in step S4, the feature extraction module extracts the surface texture features of the first real object in the real scene image corresponding to the perspective observation area, so as to highlight the surface texture features in the perspective observation area, thereby forming a real scene image after surface texture feature extraction.

[0156] The transparency setting module sets the transparency distribution of the determined perspective observation area as described in step S5.

[0157] As described in step S6 of the color setting module, the color distribution of the perspective observation area is calculated based on the angle between the shooting direction of the camera and the normal direction of each pixel in the perspective observation area of ​​the two-dimensional image displayed on the display device, i.e., the real scene image, so as to obtain the real scene image after surface texture feature extraction and transparency processing.

[0158] As described in step ST7, the rendering module renders the image by combining spatial information including depth information, material information, texture information, light source information, perspective observation area information, color distribution information of the perspective observation area, and the camera's field of view, as well as the spatial information of the real scene image and the non-real scene image, to form a fused image.

[0159] Furthermore, it should be noted that the graphic processing methods of the above-described embodiments and their variations of the present invention can also be implemented as computer software programs. For example, one embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the figures.

[0160] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A graphics processing method, characterized in that, include: The first real object is photographed with a camera to obtain a real-world image of the first real object; Obtain a non-real-scene image of a second real object, which is located behind the first real object; Obtain perspective observation area information and determine the perspective observation area in the real scene image based on the perspective observation area information, so as to be able to observe the part of the non-real scene image corresponding to the perspective observation area; The surface texture features of the portion of the real-scene image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a real-scene image after surface texture feature extraction. Set the transparency distribution of the perspective observation area; as well as Based on the shooting direction of the camera and the angle between the normal direction of each pixel in the real scene image and the perspective observation area, the color distribution of the perspective observation area is calculated and set to obtain the real scene image after surface texture feature extraction and transparency processing.

2. The graphics processing method as described in claim 1, characterized in that, The spatial information of the real scene image and the non-real scene image, the perspective observation area information, and the color distribution information are fused and rendered to form a fused image. The fused image is formed by superimposing the real scene image after surface texture feature extraction and transparency processing with the image of the non-real scene image located behind the perspective observation area.

3. The graphics processing method as described in claim 2, characterized in that, The spatial information includes depth information.

4. The graphics processing method as described in claim 1, characterized in that, Setting the transparency distribution of the perspective observation area includes performing a smooth transition process on the perspective observation area.

5. The graphics processing method as described in claim 4, characterized in that, Smoothing the transition of the perspective observation area includes: Assign basic transparency to each pixel in the perspective viewing area; and Additional transparency is assigned to each pixel based on its position within the perspective viewing area.

6. The graphics processing method as described in claim 1, characterized in that, Before extracting the surface texture features, the grayscale and contrast of the perspective observation area are adjusted.

7. The graphics processing method as described in claim 1, characterized in that, Before or simultaneously with the extraction of the surface texture features, edge detection is performed on the portion of the real-world image corresponding to the perspective observation area.

8. The graphics processing method as described in claim 1, characterized in that, After forming the real-world image with surface texture features extracted, the grayscale of the image pixels in the perspective observation area is flipped to make high-brightness pixels become low-brightness pixels and vice versa.

9. The graphics processing method as described in claim 8, characterized in that, The perspective observation area after the color inversion process is binarized so that the part of the real scene image corresponding to the perspective observation area is formed as a grayscale image.

10. The graphics processing method as described in claim 9, characterized in that, After the portion of the real-world image corresponding to the perspective observation area is formed into a grayscale image, the grayscale image is subjected to color balance processing to form a pseudo-color image.

11. The graphics processing method as described in claim 2, characterized in that, The first real object is the human anatomical structure.

12. The graphics processing method as described in claim 1, characterized in that, Before acquiring the perspective observation area information, a determination is made as to whether a perspective execution command has been received. If the perspective execution command is received, the perspective observation area information is acquired. The perspective execution command is used to instruct the acquisition of information in the perspective observation area.

13. The graphics processing method as described in claim 2, characterized in that, Virtual navigation object markers are generated, which are formed by graphically processing the camera based on its coordinate information. The virtual navigation object marker is drawn on the front side of the fused image.

14. The graphics processing method as described in claim 1, characterized in that, The location and extent of the perspective observation area are determined according to a pre-set algorithm, or The location of the perspective observation area is determined according to the user's line of sight, or The location of the perspective observation area is determined based on the user's selection of the area of ​​interest, wherein, The region of interest can be selected manually or by voice, or based on eye-tracking data of the gaze point on the display device.

15. A graphics processing method, characterized in that, include: Obtain the non-real-scene image of the first real object, i.e., the first non-real-scene image; Obtain a non-real-scene image of the second real object, i.e., the second non-real-scene image, where the second real object is located behind the first real object; A virtual image is formed based on the spatial information of the first non-real-scene graphic and the field of view of the virtual camera; Obtain perspective observation area information and determine the perspective observation area in the virtual image based on the perspective observation area information, so as to be able to observe the part of the second non-real scene graphic corresponding to the perspective observation area; The surface texture features of the portion of the virtual image corresponding to the perspective observation area are extracted to highlight the surface texture features in the perspective observation area, thereby forming a virtual image after surface texture feature extraction. Set the transparency distribution of the perspective observation area; as well as Based on the shooting direction of the camera and the angle between the normal direction of each pixel of the virtual image and the perspective observation area, the color distribution of the perspective observation area is calculated and set to obtain a virtual image after surface texture feature extraction and transparency processing.

16. The graphics processing method as described in claim 15, characterized in that, The virtual image and the second non-real scene graphic's spatial information, perspective observation area information, and color distribution information are fused and rendered to form a fused image. The fused image is formed by superimposing the virtual image, which has undergone surface texture feature extraction and transparency processing, with the image of the portion of the second non-real scene graphic located behind the perspective observation area.

17. The graphics processing method as described in claim 16, characterized in that, Before acquiring the perspective observation area A real-world image of the first real object is obtained by taking pictures with a real camera. Obtain the poses of the real camera and the virtual camera. The pose of the real camera is compared with the pose of the virtual camera to make the pose of the virtual camera consistent with the pose of the real camera.

18. The graphics processing method as described in claim 17, characterized in that, The virtual image is replaced with the real-world image to obtain a real-world image after surface texture feature extraction and transparency processing.

19. A graphics processing system, the graphics processing system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is executed by the processor, it implements the method of any one of claims 1 to 18.

20. A graphics processing device, characterized in that, include: Image acquisition module, wherein the image acquisition module acquires a real-world image of the first real object; The image acquisition module acquires a non-real-scene image of a second real object, which is located behind the first real object. The region determination module acquires perspective observation region information and determines the perspective observation region in the real scene image based on the perspective observation region information, so as to be able to observe the part of the non-real scene image corresponding to the perspective observation region; The feature extraction module extracts the surface texture features of the part of the real scene image corresponding to the perspective observation area, so as to highlight the surface texture features in the perspective observation area, thereby forming a real scene image after surface texture feature extraction. A transparency setting module, wherein the transparency setting module sets the transparency distribution of the perspective observation area; as well as The color setting module calculates and sets the color distribution of the perspective observation area based on the shooting direction of the camera and the angle between the normal direction of each pixel in the real scene image and the perspective observation area, so as to obtain the real scene image after surface texture feature extraction and transparency processing.

21. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 18.

22. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 18.

Images