Maintaining proximity awareness with amplification

Abstract

The present disclosure relates to preserving proximate environmental awareness with zooming. A region of interest in a visual output of a surgical site is identified. A region of interest in a visual output of a surgical site is identified. A zoom operation is performed on the region of interest to generate a zoomed-in region of interest. Based on an amount of zoom associated with the zoom operation, a blocked-out region surrounding the region of interest in the visual output is determined. The blocked-out region is a region of the visual output that is blocked out by placing the zoomed-in region of interest over the region of interest in the visual output. A non-linear compression is applied to the blocked-out region of the visual output to generate a compressed blocked-out region. The zoomed-in region of interest is updated to include the compressed blocked-out region.

Description

Technical Field

[0001] At least one embodiment involves performing a magnification operation on a portion of image data according to the various new techniques described herein, while maintaining environmental awareness of the magnified portion. For example, embodiments provide environmental awareness of a region surrounding the magnified region of interest according to the various new techniques described herein. Background Technology

[0002] Endoscopes are widely used in medical procedures inside the human body. In particular, optical zoom endoscopes are used to provide high-resolution imaging with close-up (e.g., magnified) views of areas of interest within a wide field of view inside the body. The magnified or enlarged area of ​​interest allows the user to view the enlarged image for further examination while maintaining high image quality. Generally, to maintain the magnified area of ​​interest within the field of view of other parts of the body, it is covered over the wide field of view. However, the magnified area of ​​interest occupies additional space within the wide field of view inside the body, causing a portion of the wide field of view inside the body to be obscured by the covered magnified area of ​​interest. Therefore, the user is unaware of the details of the portion of the wide field of view inside the body obscured by the covered magnified area of ​​interest. These details may be important for the surgeon to know during the procedure. Attached Figure Description

[0003] Figure 1 A diagram of a system for maintaining proximity awareness during an enlarged surgical procedure, according to at least one embodiment;

[0004] Figure 2A and 2B This is an example of nonlinear compression performed on a portion of a frame during an enlarged surgical procedure, according to at least one embodiment, to maintain proximity awareness.

[0005] Figure 3 A flowchart illustrating a process for maintaining proximity awareness during an enlarged surgical procedure, according to at least one embodiment;

[0006] Figure 4 A flowchart illustrating a process for maintaining proximity awareness during an amplified surgical procedure based on event triggering, according to at least one embodiment;

[0007] Figures 5A-5C Examples of maintaining proximity awareness during magnified surgery based on event triggering according to at least one embodiment; and

[0008] Figure 6 A block diagram illustrating a computer system according to at least one embodiment. Detailed Implementation

[0009] The embodiments described herein relate to systems and methods for enlarging portions of an image while maintaining proximity awareness of the non-enlarged portions of the image. In embodiments, during a medical procedure, an endoscope may capture images (e.g., video frames) of a surgical site within the body. During surgery or another medical procedure, the images / videos may be displayed on a monitor that can be seen by a physician (e.g., via visual output). In some embodiments, the visual output may be a single image or video of the surgical site (e.g., a sequence of image frames). Depending on the embodiment, the clinician may identify a region of interest (AOI) in the visual output. Alternatively, processing logic may automatically identify the AOI. Once the AOI is identified in the visual output, a magnification operation may be performed on the identified AOI in the visual output.

[0010] Depending on the embodiment, the identified region of interest can be cropped from the visual output before performing a magnification operation on it. In some embodiments, a cutoff percentage is set based on the clinician's intention to retain a portion of the magnified identified region of interest (e.g., the magnified region). Based on the set cutoff percentage, a compressed region (e.g., the remaining portion of the identified region of interest) is determined around the region of interest. Once the compressed region is determined, nonlinear compression is applied to it to generate a compressed region. This compressed region can be a ring around the magnified region. In some embodiments, performing nonlinear compression of the compressed region may include obtaining a scaling factor based on the magnification operation and obtaining the radius of the magnified region based on the set cutoff percentage. Once the scaling factor and the radius of the magnified region are obtained, a power factor can be determined based on these factors. Furthermore, for each pixel located within the compressed region, the power factor can be applied to that pixel to obtain a new pixel value to update the pixel. In some embodiments, the pixel coordinates are converted from Cartesian coordinates to polar coordinates so that the power factor is applied to the radial coordinates of the pixel. The pixel coordinates are then converted back to Cartesian coordinates to obtain a new pixel value. Once the power factor is applied to each pixel within the compressed region, the compressed region is generated.

[0011] After the compressed region is generated, the visual output is updated to include a modified region of interest, which includes both the magnified and compressed regions. Depending on the embodiment, the visual output can be updated by stitching the modified region of interest, including both the magnified and compressed regions, to a visual output from which the region of interest has been cropped. The updated visual output, including the identified region of interest with both the magnified and compressed regions, can be displayed on a display device.

[0012] This disclosure addresses the aforementioned and other drawbacks by compressing the portion of the surgical site that is obscured by the magnified area of ​​the surgical site for the clinician. Additionally and / or alternatively, the portion of the surgical site surrounding the magnified area that is not obscured may be compressed to accommodate the increased size of the magnified area.

[0013] The advantages of this disclosure include, but are not limited to, providing environmental awareness of portions of the surgical site obscured by the magnified area and portions of the surgical site surrounding the magnified area, thus allowing clinicians to view aspects of the surgical site outside the area of ​​interest within the magnified area without obstruction.

[0014] The embodiments discussed herein relate to performing operations on medical images generated by an endoscope. However, it should be understood that the embodiments described herein with respect to images from an endoscope are also applicable to other types of medical images, which may or may not be optical images. Examples of other types of medical images to which the embodiments can be applied include fluoroscopic images, radiographic images (e.g., X-ray images), magnetic resonance imaging (MRI) images, ultrasound images, elastography images, photoacoustic images, etc. Furthermore, the embodiments described herein with respect to images from an endoscope are applicable to non-medical images, such as images generated for quality control purposes, artistic purposes, etc.

[0015] Figure 1 A system, generally shown as system 100, is illustrated for performing a magnification operation on an area of ​​interest (AOI) in an image or video while maintaining proximity awareness of the area surrounding the AOI, according to some embodiments of the present disclosure. System 100 includes a computer system 110, an image input device 130, a surgical device 140, a display 150, and an input device 160. In some embodiments, more than one component, such as surgical device 140, may be omitted.

[0016] Computer system 110 may be a server, system-on-a-chip (SoC), desktop computer, laptop computer, mobile computing device, video game console, cloud computing environment, and / or any other computer system. In embodiments, computer system 110 may be a component of a device such as a video game console, mobile phone, autonomous vehicle, non-autonomous vehicle, video surveillance system, laptop computer, desktop computer, quality analysis (QA) inspection system, or other system. In at least one embodiment, computer system 110 may include, but is not limited to, one or more processors 120 representing one or more graphics processing units (GPUs), central processing units (CPUs), and / or any other processors. Computer system 110 may further include caches, data storage, and / or other components and features not shown.

[0017] In at least one embodiment, the computer system 110 may include data storage (e.g., memory). In at least one embodiment, the data storage may be or include on-chip memory of the computer system 110, which may store instructions for a compression component 115 that can be executed on the processor 120 of the computer system 110. The data storage may additionally or alternatively store one or more components of an image or moving image (e.g., video) captured by the image input device 130. In at least one embodiment, the data storage may include a Level 3 (“L3”) cache and / or a Level 2 (“L2”) cache, which are available to the processor 120 of the computer system 110. The data storage may additionally or alternatively include a hard disk drive and / or a solid-state drive.

[0018] Image input device 130 may be a device including one or more image sensors capable of generating image data such as images and / or videos. In one embodiment, image input device 130 is an endoscope that captures images and / or videos (e.g., motion images) of a surgical site. During surgery, the endoscope may be inserted into a human or animal and can perform surgical procedures and generate image data of the surgical procedures. Other types of medical image input devices include ultrasound machines, X-ray imagers, MRI machines, etc. Image input device 130 may also be a camera (e.g., a mobile phone camera, a camera in a quality analysis (QA) system, or other types of image input device 130). The generated image data may include two-dimensional (2D) or three-dimensional (3D) images. The generated image data may include color images, monochrome images, images generated using light of a specific wavelength (e.g., infrared or near-infrared (NIRI) images), and / or other types of image data.

[0019] In at least one embodiment, the processor 120 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. In at least one embodiment, the always-on processor engine may include, but is not limited to, a processor core, tightly coupled RAM, support for peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0020] In at least one embodiment, processor 120 may further include a real-time camera engine, which may include, but is not limited to, a dedicated processor subsystem for handling real-time camera management. In at least one embodiment, processor 120 may further include a signal processor, such as a high dynamic range signal processor, which may include, but is not limited to, an image signal processor. Processor 120 may further interact with a camera (e.g., image capture device 130) or an image sensor to receive and process received images.

[0021] A processor 120, such as a GPU, can feed frames (e.g., output images) to a display device (e.g., a display 150) operatively coupled to the computer system 110. In some embodiments, the GPU may include multiple cores, and each core may be capable of executing multiple threads. Each core may run multiple threads concurrently (e.g., in parallel).

[0022] In at least one embodiment, the surgical device 140 performs a surgical procedure within the surgical site under the control of a clinician. In some embodiments, the surgical device 140 may be coupled to a computer system 110. In at least one embodiment, the surgical device 140 may include, but is not limited to, one or more cutting instruments (e.g., scissors, scalpel blades, blades, and scalpels), grasping instruments (e.g., forceps), retractors, and / or any other surgical instruments. In at least one embodiment, the computer system 110 may include an input device 160 that provides the clinician with the ability to manipulate the display of still and / or moving images of the surgical site captured by the image input device 130.

[0023] In some embodiments, the computer system 110 includes a compression component 115 executed by one or more processors 120. The compression component 115 may be implemented using one or more processors 120 and optionally one or more other components. The compression component 115 may identify and enlarge an area of ​​interest (AOI) within received image data (e.g., the area of ​​a surgical site), while maintaining awareness of the surrounding environment of the AOI by performing non-linear compression on a portion of the enlarged or enlarged AOI (e.g., the area of ​​the surgical site).

[0024] In at least one embodiment, the compression component 115 identifies a region of interest in the visual output of the surgical site. In some embodiments, the region of interest in the visual output of the surgical site can be selected in real time by a clinician. Depending on the embodiment, the region of interest can be identified by image processing based on one of the following: the presence of surgical instruments, bleeding detected at the surgical site, a specific anatomical structure located within the surgical site, or any other suitable identifier available through image processing. The image processing is trained by a machine learning model to aid in further detection of regions of interest within the surgical site. Once a region of interest is detected, the machine learning model outputs the coordinates of the region of interest (e.g., the coordinates of the center of the region of interest and / or the radius of the region of interest). Additionally and / or alternatively, during playback (e.g., a non-real-time surgical procedure), the clinician can identify a region of interest by moving a mouse pointer over the region of interest that the clinician wishes to designate as a region of interest. Thus, the mouse pointer will be designated as the center of the region of interest. The compression component 115 enlarges the region of interest. According to an embodiment, the clinician can identify an enlargement value applied to the region of interest. In other embodiments, the region of interest can be enlarged by a predetermined enlargement value. The compression component 115 can determine a portion of the enlarged region of interest to be compressed (e.g., a compressed region) based on a cutoff factor. In some embodiments, the cutoff factor is determined based on the radius of a portion of the enlarged region of interest that the clinician intends to retain without compression, chosen between the original radius of the region of interest before enlargement and the radius of the enlarged region of interest. Therefore, the cutoff factor is used to identify compressed regions within the enlarged region of interest in order to determine the pixel values ​​within the compressed region before enlargement to be compressed. Additionally or alternatively, nonlinear compression can be used to compress occluded regions of the image and include them in the compressed region. According to nonlinear compression, the amount of compression can be inversely proportional to the distance from the center of the region of interest. In at least one embodiment, compression component 115 can apply nonlinear compression to occluded regions of the image (e.g., the difference between the size of the enlarged and unenlarged AOI), and the result can be included in the compressed region of the enlarged region of interest. After compression of the compressed region, compression component 115 can stitch the enlarged region of interest, including the compressed region, to the visual output at the location of the original region of interest.

[0025] Figure 2A and 2BNonlinear compression performed on a portion of a frame or image according to at least one embodiment is illustrated to maintain proximity awareness during magnification. For example, nonlinear compression can be performed in real-time or near real-time during surgery, allowing the physician to view the current frame of a video with magnified AOI (e.g., indicating what is currently being viewed through the endoscope). Nonlinear compression can also be performed on previously generated videos and / or images to generate magnified AOIs on those videos and / or images. According to an embodiment, image 200 can be a visual output (e.g., an image or frame of video) showing the surgical site 220.

[0026] refer to Figure 2A The compression component 115 of the computer system 110 can identify a region of interest 222 in the image 200 (e.g., in the surgical site 220 shown in the image 200), as described above. After determining the region of interest 222 in the image 200, the compression component 115 can perform a magnification operation by using a configurable magnification factor (e.g., Z). factor (e.g., 2X, 4X, 6X, 8X, etc.) enlarges the region of interest 222 to have an enlarged radius (e.g., R). magnified The increased area of ​​interest is 224.

[0027] In one embodiment, the radius R is increased. magnified Equal to the amplification factor Z factor Half of the region of interest multiplied by the radius of the region of interest (e.g., R). Interest Amplification factor Z factor Half of (e.g., 1X, 2X, 3X, 4X, etc.) represents a scaling factor (e.g., s) in a single direction. The variable radius R magnified The calculation can be performed as follows:

[0028] (1)R magnified =s*R Interest

[0029] Therefore, in order to determine the radius R of the region of interest in the image 200 of the surgical site 220 Interest The radius R can be increased magnified Divide by the scaling factor s. The radius R of the region of interest. Interest The calculation is as follows:

[0030] (2)R Interest =R magnified / s

[0031] In an embodiment, the occluded portion 223 of the image 200 may be the difference between the initial region of interest 222 and the enlarged region of interest 224.

[0032] In some embodiments, reference Figure 2BThe compression component 115 sets a cutoff factor. The cutoff factor represents the percentage (e.g., 85% or 0.85) of the enlarged and uncompressed region of interest 222m that the user wants to retain (e.g., enlarged region 224). Therefore, the remaining portion of the enlarged region of interest (e.g., compressed region 228) will be compressed.

[0033] Once the cutoff factor is set, the compression component 115 determines the radius of the amplification region (e.g., R). zoomed The cutoff factor is equal to the radius R of the magnified region. zoomed Divide by the increasing radius R magnified The cutoff coefficient is calculated as follows:

[0034] (3) Cutoff coefficient = R Zoomed / R magnified

[0035] Therefore, in order to determine the radius R of the magnified region zoomed The compression component 115 multiplies the cutoff coefficient by the increased radius R. magnified Once the radius R of the magnified region is determined... zoomed , by the magnification region R zoomed The generated circle 226 and the circle with radius R magnified The region between the generated circles (e.g., the enlarged region of interest 222m) is the compression zone 228. Subsequently, the center of the enlarged region of interest 222m intersects with the area enlarged by the magnification zone R. zoomed The area between the generated circles (e.g., circle 226) is the magnified area 224. Magnified area R zoomed The radius is calculated as follows:

[0036] (4)R Zoomed = Cutoff coefficient x R magnified

[0037] In this embodiment, the occluded region 223 and / or a portion of the AOI 222 are compressed into the compression region 228 according to nonlinear compression. The innermost diameter 229 of the compression region 228 may have the same magnification as the magnification region 224. The outermost diameter 230 of the compression region 228 may have the same magnification as the rest of the image 200. Therefore, the innermost diameter 230 of the compression region 228 can be successfully stitched to the magnification region 224, and the outermost diameter 230 of the compression region 228 can be successfully stitched to the rest of the image 200.

[0038] Once the radius R of the magnification region is determined... zoomed The compression component 115 can determine the power coefficient of the power law applied to the compression of pixels within the compression region 228. The power law equation can be a scaling factor s multiplied by the radius R of the magnified region increased to the power coefficient P. zoomed The power coefficient P is calculated as follows:

[0039] (5)R Zoomed =s*R Zoomed P

[0040] The calculation of the power coefficient P can be simplified as follows:

[0041] (6) Power coefficient P = (log R) Zoomed –log s) / log R Zoomed

[0042] or

[0043] (7) Power coefficient P = 1 – (log s / log R) Zoomed ).

[0044] Compression component 115 can determine whether to apply compression to a pixel based on a compression threshold. In one embodiment, if the radius (R) of the pixel... pixel The square of the distance from the pixel to the center of the enlarged region of interest (222m) is greater than the radius R of the magnified region. zoomed The square of this value satisfies the compression threshold. For the center of the enlarged region of interest (222m), the pixel radius R... pixel The square of the number is equal to the x-coordinate of the pixel (e.g., x). pixel The square of the pixel plus the y-coordinate of the pixel (e.g., y = 0). pixel The square of ) . The center of the enlarged region of interest 222m can be determined by the size of the enlarged region of interest 222m (e.g., D ). magnified Divide by 2 to determine. Therefore, the pixel radius R pixel The square of x equals the x-coordinate of the pixel. pixel Subtract the size D of the enlarged region magnified The square of half of the value, plus the y-coordinate of the pixel. pixel Subtract the size D of the enlarged region magnified The square of half of the pixel radius R. pixel The square of is calculated as follows:

[0045] (8)R pixel 2 =(x pixel –(D magnified / 2)) 2 +(y pixel -(D magnified / 2)) 2

[0046] or

[0047] (9)R pixel 2 =1 / 4[(2*x pixel –D magnified )2 -(2*y pixel –D magnified )] 2

[0048] The radius R of the magnified region zoomed The square of this is equal to the square of the center of the enlarged region of interest 222m multiplied by the square of the cutoff coefficient. As mentioned earlier, the center of the enlarged region of interest 222m is determined by the size of the enlarged region of interest 222m (e.g., D). magnified The radius R of the magnified region is determined by dividing by 2. zoomed The square of the result is equal to the size D of the enlarging region. magnified The square of half of the value multiplied by the square of the cutoff factor. The radius R of the magnified region. zoomed The square of is calculated as follows:

[0049] (10)R zoomed 2 =((D) magnified / 2)*cutoff coefficient) 2

[0050] or

[0051] (11)R zoomed 2 =(D magnified / 2) 2 * Cutoff coefficient 2

[0052] or

[0053] (12)R zoomed 2 =(D magnified 2 / 4)* Cutoff coefficient 2

[0054] Therefore, the compression threshold can be calculated as follows:

[0055] (13)1 / 4[(2*x pixel –D magnified ) 2 -(2*y pixel –D magnified )] 2 >(D magnified 2 / 4)* Cutoff coefficient 2

[0056] Based on the determination that a pixel (e.g., pixel 224P) does not meet the compression threshold, compression component 115 does not compress or modify the pixel (e.g., pixel 224P). Based on the determination that a pixel meets the compression threshold, compression component 115 applies nonlinear compression to the pixel (e.g., pixel 228P). In one embodiment, nonlinear compression is applied to pixels in occlusion region 223. In one embodiment, to apply nonlinear compression to pixels in occlusion region 223, compression component 115 determines the polar coordinates of the pixel, applies a power factor P to the radial coordinates to determine the new polar coordinates of the pixel, and transforms the new polar coordinates back to Cartesian coordinates.

[0057] In one embodiment, in order to apply non-linear compression to pixel 228P (e.g., which is in compression region 228), compression component 115 subtracts x from the coordinates of the center of the enlarged region of interest 222m. pixel and y pixel Coordinates, determining the coordinates of pixel 228P relative to the center of the enlarged region of interest 222m (e.g., coordinates x). cor y cor Coordinates). As previously stated, the center of the enlarged region of interest 222m is determined by the size of the enlarged region of interest 222m (e.g., D). magnified Divide x by 2 to determine. Therefore, x cor y cor The coordinates are calculated as follows:

[0058] (14)x cor =x pixel –(D magnified / 2);

[0059] as well as

[0060] (15)y cor =y pixel -(D magnified / 2)

[0061] Once the coordinates of pixel 228P are determined, the compression component 115 determines the coordinates of pixel 228P in the visual output before enlargement (e.g., x...). prior y prior Coordinates). To determine x prior y prior Coordinates, compression component 115 will x cor y cor Each coordinate is divided by the scaling factor s. Therefore, x prior y prior The coordinates are calculated as follows:

[0062] (16)x prior =x cor / s

[0063] as well as

[0064] (17)y prior =y cor / s.

[0065] Once the visual output is enlarged to 228 pixels x prior y prior Once the coordinates are determined, compression component 115 obtains the x coordinates. prior y prior Polar coordinates (e.g., radial coordinates) Angular coordinates θ). According to the embodiment, polar coordinates can be obtained by using a polar coordinate function or any other suitable function. Therefore, for pixel 228P in the visual output before enlargement, Cartesian coordinates (x, y, θ) are returned. prior y prior The associated polar coordinates (e.g., the old radial coordinates) The angular coordinates (e.g., θ) of pixel 228P in the visual output before the enlargement. According to an embodiment, since the visual output is enlarged, the visual output is not sufficiently manipulated to change the angular coordinates between the visual output before and after the enlargement.

[0066] Once the polar coordinates are returned, the compression component 115 can apply the power coefficient P to the old radial coordinates. To obtain new polar coordinates (e.g.) To apply the power coefficient P to the old radial coordinates. Compression component 115 will replace the old radial coordinates Increased to the power coefficient P. Therefore, the new polar coordinates... The calculation is as follows:

[0067]

[0068] Once the new polar coordinates are determined Compression component 115 obtains The Cartesian coordinates of θ determine the x-axis of the replacement pixel 228P. pixel and y pixel The coordinates of the coordinates. According to an embodiment, Cartesian coordinates can be obtained by using a Cartesian coordinate function or any other suitable function. The result is compared with the radial coordinates ( Cartesian coordinates (x) associated with θ) new ,y new The coordinates (x, y) are returned. Therefore, the compression component 115 can use the coordinates (x, y) from the visual output before enlargement. new ,y new Replace the pixel value at coordinate (x) pixel ,y pixel ).

[0069] In some embodiments, the shape used to enlarge the region of interest or designate portions of the region of interest to be enlarged can be any shape suitable for designating the enlarged region, other than a circle (e.g., rectangle, ellipse, square, etc.). Therefore, calculations can be adjusted to accommodate different shapes applied to the enlarged region of interest or portions of the enlarged region of interest. Furthermore, and / or alternatively, combinations of shapes can be applied (e.g., a square shape for the enlarged region of interest 222m, a circular shape for the enlarged region 224, and a square shape for the compressed region 228).

[0070] Figure 3 A flowchart depicts an example method 300 for utilizing amplified proximity awareness according to one or more aspects of this disclosure. This method can be executed by processing logic that may include hardware (circuit, special-purpose logic, etc.), computer-readable instructions (running on a general-purpose computer system or a special-purpose machine), or a combination of both. In an illustrative example, method 300 may be executed by a processor, such as... Figure 1 The processor 120 (such as a GPU) in the process. Alternatively, part or all of method 300 can be executed by another module or machine. It should be noted that Figure 3 The boxes shown can be executed simultaneously, or in a different order than that shown.

[0071] In block 310, processing logic identifies a region of interest in the visual output (e.g., an image or video) of an imaging device. In one embodiment, the visual output depicts a surgical site. The region of interest may be determined manually based on user input (e.g., based on mouse pointer coordinates) and / or automatically based on image processing and / or machine learning applications. According to an embodiment, after identifying the region of interest in the visual output, the processing logic crops the region of interest from the visual output. In one embodiment, the processing logic determines the size the region of interest would have if enlarged, and crops the region to the same size as the enlarged size of the region of interest.

[0072] In box 320, the processing logic performs a zoom operation on the region of interest to generate an enlarged region of interest, as described above.

[0073] In box 330, the processing logic sets a cutoff percentage based on a portion of the enlarged area of ​​interest to be preserved. This portion of the enlarged area of ​​interest to be preserved can be an enlarged region. As previously described, the cutoff percentage (e.g., a cutoff factor) represents the percentage of the enlarged, uncompressed area of ​​interest that the user wishes to retain (e.g., 85%, 80%, 75%, 70%, etc.). The remaining portion of the enlarged area of ​​interest can be compressed by the processing logic. In one embodiment, the processing logic determines the occluded area that will be obscured by the enlarged area of ​​interest.

[0074] In box 340, the processing logic determines a compressed region around the region of interest in the visual output based on a cutoff percentage. As previously described, the compressed region around the region of interest in the visual output is the area between a circle generated by a radius determined by the cutoff percentage and a circle generated by a radius determined by the enlarged region of interest. In one embodiment, the processing logic compresses the occluded region.

[0075] In one embodiment, the processing logic applies nonlinear compression to compress data (e.g., data from an occluded region) into a compressed region of the visual output. In one embodiment, to apply nonlinear compression to this region, the processing logic obtains a scaling factor based on a magnification operation and a radius of the magnified region based on a cutoff percentage. As previously described, the processing logic determines a power factor based on the scaling factor and the radius of the magnified region. Once the power factor is determined, the processing logic applies the power factor to each pixel in the compressed region to obtain new pixel values ​​for each pixel in the compressed region. As previously described, to obtain new pixel values ​​for each pixel in the compressed region, the processing logic obtains the polar coordinates of each pixel in the compressed region (or occluded region) and applies the power factor to the radial coordinates of each pixel in the compressed region (or occluded region) by boosting the radial coordinates to the power factor. Once the radial coordinates are boosted to the power factor, the processing logic obtains the Cartesian coordinates of the boosted radial coordinates, as well as the angular coordinates of each pixel in the compressed region.

[0076] Once a new pixel value is obtained, the processing logic replaces each pixel in the compressed area (e.g., from the occluded area) with the new pixel value, and in box 350, the processing logic updates the enlarged area of ​​interest to include the compressed area.

[0077] In some embodiments, the processing logic output includes a real-time feed of video containing both the visual output and the enlarged region of interest (AOI). To output the real-time feed of video containing both the visual output and the enlarged AOI, the processing logic may overlay the enlarged AOI (including compressed regions) onto the visual output. This may include stitching the enlarged AOI containing the compressed regions onto the visual output.

[0078] In some embodiments, the visual output may be a video frame of the surgical site. Therefore, the processing logic may identify a region of interest within each frame of the video of the surgical site, perform a scaling operation on the region of interest to generate an enlarged region of interest, set a cutoff percentage based on a portion of the enlarged region of interest to be retained, determine a compressed region around the region of interest within each frame of the video based on the cutoff percentage, apply non-linear compression to the occluded region to generate the content of the compressed region, and update the enlarged region of interest to include the compressed region.

[0079] Figure 4A flowchart depicts an example method 400 for maintaining proximity awareness based on event triggering using amplification, according to one or more aspects of this disclosure. This method can be executed by processing logic that may include hardware (circuit, special-purpose logic, etc.), computer-readable instructions (running on a general-purpose computer system or a special-purpose machine), or a combination of both. In an illustrative example, method 400 may be executed by a processor, such as... Figure 1 The processor 120 in the process (such as a GPU). Alternatively, part or all of method 400 may be executed by another module or machine. It should be noted that Figure 4 The boxes shown can be executed simultaneously, or in a different order than the boxes shown.

[0080] At box 410, the processing logic obtains a frame of visual output of the surgical site. (Reference) Figure 5A Frame 500 is the visual output of surgical site 510. Frame 500 of surgical site 510 may include tissue or any other object found in the surgical site, as well as surgical instruments 520. According to an embodiment, frame 500 of the visual output of surgical site 510 may be received from an endoscope (not shown).

[0081] At frame 420, the processing logic determines whether a triggering event has occurred. According to an embodiment, the triggering event can be input from a mouse click, as shown by cursor 530 in frame 500 (see, for example, see...). Figure 5B According to embodiments, the triggering event may be based on the position of the surgical instrument 520. In some embodiments, the surgical instrument is determined by a learning model that identifies devices, instruments, and other tools in the surgical site 510. In some embodiments, based on the learning model, the processing logic may determine that a triggering event has occurred based on the operation of the surgical instrument. Furthermore, and / or alternatively, in some embodiments, the processing logic may determine that a triggering event has occurred based on the presence of the surgical instrument 520 in frame 500. In some embodiments, no triggering event has occurred, therefore, the processing logic proceeds to frame 480 to display the frame on a display device.

[0082] In block 430, the processing logic obtains the region associated with the trigger event in a frame of visual output of the surgical site. According to an embodiment, the region associated with the trigger event in the frame is derived based on the position of cursor 530 (e.g., the pointer tip of cursor 530 would indicate the center of the region). In some embodiments, the region associated with the trigger event in the frame may be generated based on the position of surgical instrument 520 (e.g., the distal end of surgical instrument 520 would indicate the center of the region). The distal end of surgical instrument 520 may be the portion of surgical instrument 520 furthest from the clinician. According to an embodiment, the region associated with the trigger event in the frame may be a region defined by a predetermined distance from the position of cursor 530 or surgical instrument 520.

[0083] In block 440, the processing logic clips the region of interest in the frame associated with the triggering event. For example, the clipped AOI could be a circle with a predetermined radius centered on the position of the surgical instrument or cursor. The processing logic proceeds to block 450, where it performs a magnification operation on the clipped region in the frame associated with the triggering event (e.g., magnified clipped region 540). According to embodiments, the magnification operation can be controlled by manipulating an endoscope (not shown), an input device of the surgical system (e.g., a foot pedal), a mouse wheel, left-clicking and / or right-clicking the mouse, and any suitable means for adjusting the magnification operation to control the magnification level (e.g., 2X, 3X, 4X, etc.).

[0084] In box 460, the processing logic performs nonlinear compression on the boundaries of the enlarged cropped region 540 and / or on the occluded regions that will be obscured by the enlarged cropped region 540. As previously described, to apply nonlinear compression to a portion of the enlarged cropped region 540, the processing logic may obtain a scaling factor based on the enlargement operation and obtain the radius of the enlarged cropped region based on a cutoff percentage. The cutoff percentage may be set by the processing logic based on the clinician's or the processing logic's desire to retain an uncompressed portion of the enlarged cropped region 540 (e.g., 85% of the enlarged cropped region 540). The processing logic may compress the remaining portion of the enlarged cropped region 540 (e.g., the boundaries of the enlarged cropped region 540).

[0085] As previously described, the processing logic determines a power factor based on a scaling factor and a radius of the magnified region based on a cutoff percentage. Once the power factor is determined, the processing logic applies it to each pixel within the boundary of the magnified cropped region 540 to obtain new pixel values ​​for each pixel within the boundary of the magnified cropped region 540. In one embodiment, the processing logic applies the power factor to each pixel in an occluded region that will be occluded by the magnified cropped region 540. As previously described, to obtain new pixel values ​​for each pixel within the boundary of the magnified cropped region 540, the processing logic obtains the polar coordinates of each pixel within the boundary of the magnified cropped region 540 and / or the occluded region, and applies the power factor to the radial coordinates of each pixel within a portion of the magnified cropped region 540 and / or the occluded region by raising the radial coordinates to the power factor. Once the radial coordinates are raised to the power factor, the processing logic obtains the angular coordinates of each pixel within the boundary of the magnified cropped region 540 and the Cartesian coordinates of the radial coordinates raised to the power factor. As previously described, the processing logic replaces the individual pixels in the radial and angular Cartesian coordinates of the individual pixels in the boundary of the enlarged cropping region 540 with the pixel values ​​of the individual pixels in the boundary of the enlarged cropping region 540 at the radial and angular coordinates of the individual pixels located ...

[0086] In frame 470, once each pixel in the bounds of the enlarged cropped region 540 and / or in the occluded region is compressed, the processing logic stitches the enlarged cropped region 540 to the location of the region in the frame that triggered the event. The processing logic proceeds to frame 480, and the frame is displayed on the display device (see [link]). Figure 5C Furthermore, according to the embodiment, additional image operations (e.g., thresholding, contrast adjustment, adding artificial exposure, edge detection, etc.) can be performed on the enlarged cropped region 540.

[0087] Figure 6 This is a block diagram of a processing system 600 according to at least one embodiment. The processing system 600 may correspond to the embodiments described. Figure 1 The computer system 110 is described above. In at least one embodiment, system 600 includes one or more processors 602 and one or more graphics processors 608, and may be a single-processor desktop system, a multi-processor workstation system, or a server system having a large number of processors 602 or processor cores 607. In at least one embodiment, system 600 is a processing platform incorporated within a system-on-a-chip (SoC) integrated circuit for use in mobile, handheld, or embedded devices.

[0088] In at least one embodiment, system 600 may include or be incorporated into a server-based gaming platform, a game console including a game and media console, a mobile game console, a handheld game console, or an online game console. In at least one embodiment, system 600 is a mobile phone, smartphone, tablet computing device, or mobile internet device. In at least one embodiment, processing system 600 may also include, be incorporated into, or be integrated into a wearable device, such as a smartwatch, smart glasses, augmented reality, or virtual reality device. In at least one embodiment, processing system 600 is a television or set-top box device having one or more processors 602 and a graphical interface generated by one or more graphics processors 608. In at least one embodiment, system 600 is a desktop computer or server computer.

[0089] In at least one embodiment, one or more processors 602 each include one or more processor cores 607 for processing instructions that, when executed, perform operations against the system and user software. In at least one embodiment, each of the one or more processor cores 607 is configured to process a specific instruction sequence 609. In at least one embodiment, the instruction sequence 609 may facilitate Complex Instruction Set Computing (CISC), Reduced Instruction Set Computing (RISC), or computation via Very Long Instruction Word (VLIW). In at least one embodiment, each processor core 607 may process a different instruction sequence 609, which may include instructions that facilitate the emulation of other instruction sequences. In at least one embodiment, the processor core 607 may also include other processing devices, such as a digital signal processor (DSP).

[0090] In at least one embodiment, processor 602 includes cache memory 604. In at least one embodiment, processor 602 may have a single internal cache or multiple levels of internal caches. In at least one embodiment, the cache memory is shared among various components of processor 602. In at least one embodiment, processor 602 also uses an external cache (e.g., a Level 3 (L3) cache or a last-level cache (LLC)) (not shown), which can be shared among processor cores 607 using known cache coherence techniques. In at least one embodiment, processor 602 further includes a register file 606, which may include different types of registers (e.g., integer registers, floating-point registers, status registers, and instruction pointer registers) for storing different types of data. In at least one embodiment, register file 606 may include general-purpose registers or other registers.

[0091] In at least one embodiment, one or more processors 602 are coupled to one or more interface buses 610 to transmit communication signals, such as address, data, or control signals, between the processor 602 and other components in the system 600. In at least one embodiment, the interface bus 610 may be a processor bus, such as a version of the Direct Media Interface (DMI) bus. In at least one embodiment, the interface bus 610 is not limited to the DMI bus and may include one or more peripheral component interconnect buses (e.g., PCI, PCI Express), memory buses, or other types of interface buses. In at least one embodiment, the processor 602 includes an integrated memory controller 616 and a platform controller hub 630. In at least one embodiment, the memory controller 616 facilitates communication between memory devices and other components of the processing system 600, while the platform controller hub (PCH) 630 provides connectivity to input / output (I / O) devices via a local I / O bus.

[0092] In at least one embodiment, memory device 620 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a phase-change memory device, or a device with suitable performance for use as processor memory. In at least one embodiment, memory device 620 may be used as system memory of processing system 600 to store data 622 and instructions 621 for use when one or more processors 602 execute an application or process. In at least one embodiment, memory controller 616 is also coupled to an optional external graphics processor 612, which may communicate with one or more graphics processors 608 of processor 602 to perform graphics and media operations. In at least one embodiment, display device 611 may be connected to processor 602. In at least one embodiment, display device 611 may include one or more internal display devices, such as in mobile electronic devices or laptop devices, or external display devices connected via a display interface (e.g., DisplayPort). In at least one embodiment, display device 611 may include a head-mounted display (HMD), such as a stereoscopic display device for virtual reality (VR) or augmented reality (AR) applications.

[0093] In at least one embodiment, the platform controller hub 630 enables peripheral devices to connect to the storage device 620 and the processor 602 via a high-speed I / O bus. In at least one embodiment, the I / O peripheral devices include, but are not limited to, an audio controller 646, a network controller 634, a firmware interface 628, a wireless transceiver 626, a touch sensor 625, and a data storage device 624 (e.g., a hard disk drive, flash memory, etc.). In at least one embodiment, the data storage device 624 may be connected via a storage interface (e.g., SATA) or via a peripheral bus, such as a peripheral component interconnect bus (e.g., PCI, PCIe). In at least one embodiment, the touch sensor 625 may include a touchscreen sensor, a pressure sensor, or a fingerprint sensor. In at least one embodiment, the wireless transceiver 626 may be a Wi-Fi transceiver, a Bluetooth transceiver, or a mobile network transceiver, such as a 3G, 4G, or LTE transceiver. In at least one embodiment, the firmware interface 628 enables communication with the system firmware and may be, for example, a Unified Extensible Firmware Interface (UEFI). In at least one embodiment, the network controller 634 may enable network connectivity to a wired network. In at least one embodiment, a high-performance network controller (not shown) is coupled to interface bus 610. In at least one embodiment, audio controller 646 is a multi-channel high-definition audio controller. In at least one embodiment, processing system 600 includes an optional legacy I / O controller 640 for coupling legacy (e.g., Personal System 2 (PS / 2)) devices to system 600. In at least one embodiment, platform controller hub 630 may also be connected to one or more Universal Serial Bus (USB) controllers 642 that connect input devices, such as a keyboard and mouse combination 643, a camera 644, or other USB input devices.

[0094] In at least one embodiment, instances of the memory controller 616 and platform controller hub 630 may be integrated into a discrete external graphics processor, such as external graphics processor 612. In at least one embodiment, the platform controller hub 630 and / or the memory controller 616 may be external to one or more processors 602. For example, in at least one embodiment, system 600 may include an external memory controller 616 and platform controller hub 630, which may be configured as a memory controller hub and peripheral controller hub in a system chipset communicating with processor 602.

[0095] In at least one embodiment, the processing system 600 includes a compression component 115 that can execute on an external graphics processor 612. The compression component 115 can be used to compress a selected area of ​​interest to be magnified to accommodate the increased width and height of the magnified area of ​​interest. Details regarding the compression component 115 are provided herein in conjunction with the figures described above. In at least one embodiment, the compression component 115 can be used in the system 600 for using magnification to maintain proximity awareness as described herein.

[0096] Other variations are within the spirit of this disclosure. Therefore, while the disclosed technology is readily adaptable to various modifications and alternative constructions, certain embodiments thereof are illustrated in the accompanying drawings and have been described in detail above. However, it should be understood that the disclosure is not intended to be limited to one or more specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of this disclosure as defined in the appended claims.

[0097] Unless otherwise stated or obviously contradicted by the context, the terms “a,” “an,” and “the,” and similar references, used in the context of describing the disclosed embodiments (particularly in the context of the appended claims), should be interpreted as encompassing both singular and plural forms, rather than as definitions of the terms. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” should be interpreted as open-ended terms (meaning “including, but not limited to”). The term “connection” (referring to a physical connection where not modified) should be interpreted as partially or wholly contained, attached to, or joined together, even with some intervention. Unless otherwise indicated herein, references to numerical ranges herein are intended only as a way of abbreviating each individual value falling within that range, and each individual value is incorporated into the specification as if it were separately described herein. In at least one embodiment, unless otherwise indicated or contradicted by the context, the use of the terms “set” (e.g., “item set”) or “subset” should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by the context, the term “subset” of the corresponding set does not necessarily mean an appropriate subset of the corresponding set, but rather that the subset and the corresponding set can be equal.

[0098] Unless otherwise explicitly stated or clearly contradicted by the context, connective phrases such as “at least one of A, B, and C” or “at least one of A, B, and C” are understood in the context to generally refer to items, terms, etc., which can be A or B or C, or any non-empty subset of the set A, B, and C. For example, in an illustrative example of a set with three members, the connective phrases “at least one of A, B, and C” and “at least one of A, B, and C” refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Therefore, such connective language is generally not intended to imply that some embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless otherwise stated or contradicted by the context, the term “multiple” indicates a plural state (e.g., “multiple items” means multiple items). In at least one embodiment, the number of items in the multiple items is at least two, but may be more if explicitly indicated or indicated by the context. Furthermore, unless otherwise stated or clearly understood from the context, the phrase “based on” means “at least partially based on” rather than “based on only”.

[0099] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations thereof and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that is executed jointly on one or more processors via hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagating transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuitry (e.g., buffers, caches, and queues). In at least one embodiment, code (e.g., executable code or source code) is stored on one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) on which executable instructions are stored, which, when executed by one or more processors of a computer system (i.e., as a result of execution), cause the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media comprises multiple non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media lack all the code, but the multiple non-transitory computer-readable storage media collectively store all the code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors; for example, the non-transitory computer-readable storage media store the instructions, and the main central processing unit (“CPU”) executes some instructions while the graphics processing unit (“GPU”) executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and the different processors execute different subsets of the instructions.

[0100] Therefore, in at least one embodiment, the computer system is configured to implement one or more services that perform the operations of the processes described herein, either individually or collectively, and such a computer system is configured with suitable hardware and / or software to enable the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of this disclosure is a single device, and in another embodiment it is a distributed computer system comprising multiple devices operating in different ways, such that the distributed computer system performs the operations described herein, and that a single device does not perform all the operations.

[0101] The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the disclosure.

[0102] All references cited in this article, including publications, patent applications and patents, are incorporated herein by reference as if each reference were individually and specifically indicated to be incorporated herein by reference and the entire contents of which are described herein.

[0103] The terms “coupled” and “connected”, and their derivatives, may be used in the specification and claims. It should be understood that these terms may not be intended to be synonyms with each other. Rather, in certain examples, “connected” or “coupled” may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. “Coupled” may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0104] Unless otherwise expressly stated, it will be understood that throughout this specification, terms such as “processing,” “computing,” “determining,” etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that process and / or convert data represented as physical quantities (e.g., electrons) in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other such information storage, transmission, or display devices of the computing system.

[0105] Similarly, the term "processor" can refer to any device or part of memory that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, a "software" process can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process can refer to multiple processes that execute instructions sequentially or intermittently, or in parallel. In at least one embodiment, the terms "system" and "method" are used interchangeably herein, provided that a system can embody one or more methods, and a method can be considered a system.

[0106] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in various ways, such as by receiving data as a parameter to a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be implemented by transmitting data as an input or output parameter to a function call, an application programming interface, or an inter-process communication mechanism.

[0107] While this document describes example implementations of the described technologies, other architectures can be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities have been defined above for descriptive purposes, various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0108] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

Claims

1. A method for maintaining proximity perception using magnification, comprising: Identify the area of ​​interest in the visual output of the surgical site; A magnification operation is performed on the region of interest to generate a larger region of interest; Based on the magnification amount associated with the magnification operation, an occluded region is determined around the region of interest in the visual output, wherein the occluded region is a region of the visual output that becomes occluded by placing the enlarged region of interest on top of the region of interest in the visual output. Nonlinear compression is applied to the occluded region of the visual output to generate a compressed occluded region. as well as The enlarged region of interest is updated to include the compressed occluded region, wherein the visual output of the surgical site outside the enlarged region of interest remains unchanged, and wherein the outermost diameter of the compressed occluded region has the same magnification as the surgical site outside the enlarged region of interest.

2. The method according to claim 1, wherein the visual output of the surgical site is an image.

3. The method of claim 1, wherein identifying the region of interest in the visual output at the surgical site comprises: The area of ​​interest is cropped from the visual output of the surgical site.

4. The method of claim 3, further comprising: The enlarged area of ​​interest, including the compressed and occluded region, is stitched into the visual output of the surgical site.

5. The method of claim 1, wherein the visual output of the surgical site is a frame of a video of the surgical site, and wherein recognition, execution, determination, application, and update are performed for multiple frames of the video.

6. The method according to claim 5, wherein the video is a real-time video of the surgical site, the method further comprising: The output includes the visual output and a real-time feed of the video of the enlarged area of ​​interest.

7. The method of claim 1, wherein applying nonlinear compression to the occluded region comprises: Based on the aforementioned amplification operation, the scaling factor is obtained; The percentage of the magnified region is determined based on the radius between the center of the enlarged region of interest and the radius of the enlarged region of interest. The power coefficient is determined based on the percentage of the magnified region and the scaling factor. For each pixel in the occluded area, a new pixel value is determined based on the power coefficient; as well as Update the corresponding pixel using the new pixel value.

8. The method of claim 7, wherein determining the new pixel value comprises: Obtain the polar coordinates of the corresponding pixel; determine a new radius value by applying the power factor to the radius value of the polar coordinates of the corresponding pixel; And obtain the Cartesian coordinates of the new pixel based on the new radius value.

9. A system that utilizes magnification to maintain proximity awareness, comprising: An image capturing device configured to acquire at least one frame of the surgical site; as well as One or more processors operatively coupled to the image capture device, the one or more processors being configured to perform operations including: Based on at least one frame of the surgical site, identify the region of interest in the visual output of the surgical site; A magnification operation is performed on the region of interest to generate a larger region of interest; Based on the magnification associated with the magnification operation, an occluded region is determined around the region of interest in the visual output, wherein the occluded region is a region of the visual output that becomes occluded by placing the enlarged region of interest on top of the region of interest in the visual output. Nonlinear compression is applied to the occluded region of the visual output to generate a compressed occluded region; and The enlarged region of interest is updated to include the compressed occluded region, wherein the visual output of the surgical site outside the enlarged region of interest remains unchanged, and wherein the outermost diameter of the compressed occluded region has the same magnification as the surgical site outside the enlarged region of interest.

10. The system of claim 9, wherein the visual output of the surgical site is an image.

11. The system of claim 9, wherein identifying the region of interest in the visual output at the surgical site comprises: The area of ​​interest is cropped from the visual output of the surgical site.

12. The system of claim 11, wherein the one or more processors are configured to perform further operations, the further operations including: The enlarged area of ​​interest, including the compressed and occluded region, is stitched into the visual output of the surgical site.

13. The system of claim 9, wherein the at least one frame associated with the visual output of the surgical site is a frame of a video of the surgical site, and wherein recognition, execution, determination, application, and update are performed for multiple frames of the video.

14. The system of claim 13, wherein the video is a real-time video of the surgical site, the system further being used to: The output includes the visual output and a real-time feed of the video of the enlarged area of ​​interest.

15. The system of claim 9, wherein applying nonlinear compression to the occluded region comprises: Based on the aforementioned amplification operation, the scaling factor is obtained; The percentage of the magnified region is determined based on the radius between the center of the enlarged region of interest and the radius of the enlarged region of interest. The power coefficient is determined based on the percentage of the magnified region and the scaling factor. For each pixel in the occluded area, a new pixel value is determined based on the power coefficient; as well as Update the corresponding pixel using the new pixel value.

16. The system of claim 15, wherein determining the new pixel value comprises: Obtain the polar coordinates of the corresponding pixel; determine a new radius value by applying the power factor to the radius value of the polar coordinates of the corresponding pixel; And based on the new radius value, obtain the Cartesian coordinates of the new pixel.

17. A non-transitory computer-readable storage medium comprising instructions, which, when executed by a processing device, cause the processing device to perform an operation, the operation comprising: Identify the area of ​​interest in the visual output of the surgical site; A magnification operation is performed on the region of interest to generate a larger region of interest; Based on the magnification associated with the magnification operation, an occluded region is determined around the region of interest in the visual output, wherein the occluded region is a region of the visual output that becomes occluded by placing the enlarged region of interest on top of the region of interest in the visual output. Nonlinear compression is applied to the occluded region of the visual output to generate a compressed occluded region. as well as The enlarged region of interest is updated to include the compressed occluded region, wherein the visual output of the surgical site outside the enlarged region of interest remains unchanged, and wherein the outermost diameter of the compressed occluded region has the same magnification as the surgical site outside the enlarged region of interest.

18. The non-transitory computer-readable storage medium of claim 17, wherein applying nonlinear compression to the occluded region comprises: Based on the aforementioned amplification operation, the scaling factor is obtained; The percentage of the magnified region is determined based on the radius between the center of the enlarged region of interest and the radius of the enlarged region of interest. The power coefficient is determined based on the percentage of the magnified region and the scaling factor. For each pixel in the occluded area, a new pixel value is determined based on the power coefficient; as well as Update the corresponding pixel using the new pixel value.

19. The non-transitory computer-readable storage medium of claim 17, wherein identifying the region of interest in the visual output at the surgical site comprises: The area of ​​interest is cropped from the visual output of the surgical site.

20. The non-transitory computer-readable storage medium of claim 17, wherein the visual output of the surgical site is a frame of a video of the surgical site, and wherein recognition, execution, determination, application, and update are performed for a plurality of frames of the video.

Images