Digital pathological section intelligent screenshot method and device, equipment and storage medium

Abstract

The invention discloses an intelligent screenshot method, device and equipment for a digital pathological section and a storage medium, and relates to the technical field of medical data processing, the method is applied to front-end equipment, and the method comprises the steps that in response to a section screenshot instruction, viewport parameters of a target section image at a current viewport and size parameters of a screenshot visual area are captured; according to the viewport parameters, drawing the slice image layer locked at the current viewport, the first annotation rendering layer and the second annotation rendering layer to an off-screen synthetic canvas corresponding to the size parameters in sequence, and generating initial screenshot data; and according to the viewport parameter and the size parameter, superposing the scale data on the initial screenshot data to generate target screenshot data. According to the method, the section image, the multi-source labeling information and the dynamic scale can be automatically synthesized during digital pathological section screenshot, manual splicing is not needed, errors are reduced, pure front-end real-time screenshot is achieved, rear-end participation is not needed, the response speed is high, and the real-time interaction requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and particularly relates to a digital pathological section intelligent screenshot method and device, equipment and a storage medium. BACKGROUND

[0002] The digital pathological section is a technology of analyzing and diagnosing after digitizing the pathological section, and the section screenshot can assist users in diagnosis, sharing and information retention.

[0003] However, the existing digital pathological section screenshot only saves the original image data, cannot save the dynamic information of the digital pathological section, needs manual post-processing, is low in efficiency and prone to errors, and cannot meet the real-time interaction requirement.

[0004] Therefore, how to meet the real-time screenshot interaction requirement of the digital pathological section becomes a problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide a digital pathological section intelligent screenshot method, device, equipment and a storage medium, and aims to solve the technical problem of how to meet the real-time screenshot interaction requirement of the digital pathological section.

[0006] To achieve the above-mentioned purpose, the present application provides a digital pathological section intelligent screenshot method, which comprises the following steps: In response to a section screenshot instruction, the viewport parameters of a target section image in a current viewport and the size parameters of a screenshot visible area are captured; According to the viewport parameters, the section image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport are sequentially drawn to the off-screen composition canvas corresponding to the size parameters to generate initial screenshot data; According to the viewport parameters and the size parameters, scale data is superimposed on the initial screenshot data to generate target screenshot data.

[0007] In an embodiment, before the step of sequentially drawing the section image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport to the off-screen composition canvas corresponding to the viewport parameters according to the viewport parameters to generate initial screenshot data, the method further comprises the following steps: A unified annotation protocol is constructed to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data having the same coordinate system; The first protocol data is mapped to the first annotation rendering layer and the second protocol data is mapped to the second annotation rendering layer based on the spatial coordinates of the view engine corresponding to the section image layer, and the display states of the first annotation rendering layer and the second annotation rendering layer are synchronously updated in the case of listening to a viewport change event.

[0008] In an embodiment, the step of constructing the unified annotation protocol comprises: receiving a first JSON object of a first annotation source output updated based on a user interaction event, the first JSON object carrying key fields including at least coordinates, graphic types and attribute information; receiving a second JSON object of a second annotation source output updated based on a slice tissue density of the target slice image, the second JSON object carrying the same key fields as the first JSON object; converting the first JSON object and the second JSON object into first protocol data and second protocol data respectively, both of which include source identification and rendering priority.

[0009] In an embodiment, the second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

[0010] In an embodiment, the view engine registers a viewport change event listener to broadcast view parameters including a zoom coefficient, a rotation angle and a view coordinate to the first annotation rendering layer and the second annotation rendering layer in the case that the target slice image is zoomed, translated or rotated; The step of synchronously updating the display states of the first annotation rendering layer and the second annotation rendering layer in the case that the viewport change event is listened to comprises: adjusting a geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameters in the case that the viewport change event is listened to; calling a drawing interface of a rendering engine corresponding to the first annotation rendering layer and the second annotation rendering layer according to the adjusted geometric transformation matrix to redraw the display states of the first annotation rendering layer and the second annotation rendering layer.

[0011] In an embodiment, the step of drawing the slice image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport in sequence to an off-screen composition canvas corresponding to the size parameter according to the viewport parameter to generate initial screenshot data comprises: obtaining slice image data from a view engine corresponding to the slice image layer and drawing the slice image data on a bottom layer of the off-screen composition canvas corresponding to the size parameter; clipping off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameter and the size parameter to obtain first rendering content; clipping off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameter and the size parameter to obtain second rendering content; convert the first rendering content and the second rendering content into a first optimized mark and a second optimized mark according to a preset canvas optimization rule; draw the first optimized mark to a middle layer of the off-screen composition canvas and draw the second optimized mark to an upper layer of the off-screen composition canvas to generate initial screenshot data.

[0012] In an embodiment, the step of superimposing scale data on the initial screenshot data according to the viewport parameter and the size parameter to generate target screenshot data comprises: generating a target scale length according to a preset scale factor and the size parameter; generating an optimized scale length based on the target scale length and the viewport parameter; obtaining an intelligent unit corresponding to the optimized scale length; drawing scale data on the initial screenshot data based on the optimized scale length and the intelligent unit according to a superimposable area analysis result of the slice image to generate target screenshot data.

[0013] In addition, to achieve the above object, the present application further provides a digital pathology slice intelligent screenshot device, which comprises: a parameter acquisition module configured to capture a viewport parameter of a target slice image in a current viewport and a size parameter of a screenshot visible area in response to a slice screenshot instruction; a preliminary rendering module configured to draw a slice image layer locked in the current viewport, a first mark rendering layer and a second mark rendering layer in sequence to an off-screen composition canvas corresponding to the size parameter according to the viewport parameter to generate initial screenshot data; a target rendering module configured to superimpose scale data on the initial screenshot data according to the viewport parameter and the size parameter to generate target screenshot data.

[0014] In addition, to achieve the above object, the present application further provides a digital pathology slice intelligent screenshot device, which comprises: a memory, a processor and a digital pathology slice intelligent screenshot program stored in the memory and executable on the processor, the digital pathology slice intelligent screenshot program being configured to implement the steps of the digital pathology slice intelligent screenshot method as described above.

[0015] In addition, to achieve the above object, the present application further provides a storage medium, which is a storage medium, and the storage medium stores a digital pathology slice intelligent screenshot program, the digital pathology slice intelligent screenshot program being executable by a processor to implement the steps of the digital pathology slice intelligent screenshot method as described above.

[0016] The application provides a digital pathological section intelligent screenshot method, device, equipment and storage medium. The method is applied to a front-end device and includes the following steps: in response to a section screenshot instruction, capturing viewport parameters of a target section image in a current viewport and size parameters of a screenshot visible area; according to the viewport parameters, sequentially drawing a section image layer locked in the current viewport, a first annotation rendering layer and a second annotation rendering layer to an off-screen composition canvas corresponding to the size parameters to generate initial screenshot data; and superimposing scale data on the initial screenshot data according to the viewport parameters and the size parameters to generate target screenshot data. The application can automatically synthesize a section image, multi-source annotation information and a dynamic scale when performing a digital pathological section screenshot, does not require manual splicing, reduces errors, realizes pure front-end real-time screenshot, does not require the participation of a back-end, has fast response speed and meets real-time interaction requirements. BRIEF DESCRIPTION OF DRAWINGS

[0017] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0019] Figure 1 The first flowchart of the first embodiment of the digital pathological section intelligent screenshot method of the present application is shown in the figure. Figure 2 The second flowchart of the first embodiment of the digital pathological section intelligent screenshot method of the present application is shown in the figure. Figure 3 The first flowchart of the second embodiment of the digital pathological section intelligent screenshot method of the present application is shown in the figure. Figure 4 The second flowchart of the second embodiment of the digital pathological section intelligent screenshot method of the present application is shown in the figure. Figure 5 The simple flowchart of the digital pathological section intelligent screenshot method of the present application is shown in the figure. Figure 6 The module structure diagram of the digital pathological section intelligent screenshot device of the embodiment of the present application is shown in the figure. Figure 7 The device structure diagram of the hardware running environment involved in the digital pathological section intelligent screenshot method of the embodiment of the present application is shown in the figure.

[0020] The purpose implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0022] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0023] The main solution of the present application is: in response to a slice screenshot instruction, capturing the viewport parameters of a target slice image in a current viewport and the size parameters of a screenshot visible area; according to the viewport parameters, sequentially drawing a slice image layer, a first annotation rendering layer and a second annotation rendering layer locked in the current viewport to an off-screen composition canvas corresponding to the size parameters, to generate initial screenshot data; superimposing scale data on the initial screenshot data according to the viewport parameters and the size parameters, to generate target screenshot data.

[0024] The existing digital pathology slice screenshot only saves the original image data, cannot retain dynamic information (such as scales changing with zooming, superimposed annotation layers), needs manual post-processing, is low in efficiency and prone to errors, and cannot meet real-time interaction requirements.

[0025] To solve the above problems, the present application provides an intelligent screenshot method realized purely in the front end, which can automatically synthesize the current dynamic scale of a slice (calculated in real time with zooming levels), annotations of a first annotation rendering layer (user-annotated regions) and a second annotation rendering layer (suspicious regions analyzed by AI), and a digital pathology slice image according to configuration requirements, without the need for manual splicing, reducing errors, retaining complete diagnostic context information, and improving user communication, sharing and archiving efficiency. Therefore, the present application can realize real-time screenshot purely in the front end, without the need for backend participation, fast response speed, ensuring the integrity, layout aesthetics and smoothness of information, and meeting real-time interaction requirements.

[0026] It should be noted that the execution subject of the present embodiment can be a digital pathology slice intelligent screenshot system, or a front-end device with data processing, network communication and program running functions, such as a user workstation, tablet computer, personal computer, mobile phone, etc. installed with a browser, or a digital pathology slice intelligent screenshot device capable of realizing the above functions, etc. The present embodiment does not specifically limit this. The present embodiment and each of the following embodiments will be described below with the front-end device as the execution subject.

[0027] Based on this, the present embodiment provides a digital pathology slice intelligent screenshot method, which refers to Figure 1 , Figure 1 The first flowchart of the first embodiment of the digital pathology slice intelligent screenshot method of the present application.

[0028] In this embodiment, the digital pathology section intelligent screenshot method includes steps S10-S30: Step S10, in response to the section screenshot instruction, capturing the viewport parameters of the target section image in the current viewport and the size parameters of the screenshot visible area; It should be understood that the section screenshot instruction can be a screenshot operation signal triggered by the user through the front-end interface, which can be triggered by, for example, clicking the "screenshot" button, pressing a pre-set shortcut key, etc. The above-mentioned target section image can be the original digital pathology section (such as DICOM, SDPC format digital pathology section) currently requiring screenshot.

[0029] And the above-mentioned current viewport can be the section image area currently visible to the user in the front-end device display interface (i.e. the visible window area of the browser or application). Accordingly, the above-mentioned viewport parameters can be key data describing the current viewport state, including but not limited to viewport coordinates (i.e. the pixel position of the top-left corner of the current visible area in the section coordinate system of the entire target section image, such as {x:1200, y:800}, unit: pixel (px, pixel)), zoom factor (i.e. the magnification multiple of the original target section image corresponding to the current digital pathology section, such as 5x, 20x), pixel size information (physical size corresponding to each pixel in the section data, such as 0.25 μm / pixel (micron per pixel)). The size parameters of the above-mentioned screenshot visible area can be the width and height (unit: pixel) of the user's screenshot section area, which is usually consistent with the display size of the current viewport (such as 1024px×768px).

[0030] For example, when the user clicks the "screenshot" button of the front-end interface (triggering the section screenshot instruction), the front-end device can pause the interactive behavior of the section, and capture the above-mentioned viewport parameters and size parameters in real time through the API of the view engine (such as OpenSeaDragon) or Konva.

[0031] For example, the zoom factor can be obtained through "viewer.viewport.getZoom()" of OpenSeaDragon or Konva bufferCanvas, the rotation angle can be obtained through "viewer.viewport.getRotation()", and the size parameters can be obtained through "viewer.container.clientWidth" and "viewer.container.clientHeight".

[0032] Step S20, according to the viewport parameters, sequentially rendering the section image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport to the off-screen composition canvas corresponding to the size parameters, to generate initial screenshot data; It is easy to understand that the front-end device can pause the interactive operation (such as zooming, panning, and rotating) of the current viewport, freeze the current state of the slice image layer, the first annotation rendering layer, and the second annotation rendering layer, and ensure that the layer content does not change during the screenshot process when detecting the slice screenshot instruction.

[0033] In the embodiment, the slice image layer can be a layer for displaying an original digital pathology slice image, and the data is derived from a slice source file (such as an sdpc or dcm format file) of a target slice image. The first annotation rendering layer and the second annotation rendering layer can be layers for displaying annotation information. In the embodiment, the first annotation rendering layer can correspond to a user manual annotation layer, such as a rectangle, a circle, an arrow, and the like drawn by the user through an interactive tool. The second annotation rendering layer can correspond to a layer of image analysis results of the target slice image obtained by pre-setting a lightweight AI model (such as an ONNX Runtime front-end inference) or a pre-trained neural network structure, such as a suspicious area polygon, a cell dot, or a text annotation (such as “high-risk area, recommend biopsy”) identified by an AI or a large model.

[0034] It should be noted that the off-screen composition canvas can be a Canvas element in the front-end device that is not directly displayed on the interface, and is used for temporarily synthesizing the data of the layers (to avoid visual flicker caused by directly operating the display canvas). The size of the off-screen composition canvas is consistent with the size parameter of the screenshot visible area. The image data of the layers in the off-screen composition canvas without the scale bar is the initial screenshot data.

[0035] For example, in the embodiment, the front-end device can draw the locked slice image layer, the first annotation rendering layer, and the second annotation rendering layer in the order of “bottom layer-middle layer-top layer” to the off-screen composition canvas according to the captured viewport parameters to form the initial screenshot data. 1) Draw the slice image layer on the bottom layer: the slice image data of the current viewport can be obtained through the “viewer.drawer.canvas” of the view engine, and the “drawImage()” method of the Canvas API is called to draw the slice image data on the bottom layer of the off-screen composition canvas.

[0036] 2) Draw the first annotation rendering layer on the middle layer: the first annotation rendering layer is drawn on the middle layer of the off-screen composition canvas from the off-screen canvas (such as the bufferCanvas of Konva) corresponding to the first annotation rendering layer, and the area corresponding to the current viewport is cropped according to the viewport parameters (for example, the width and height of Konva.layer are set according to the client width and height of viewer.container).

[0037] 3) Draw the second annotation rendering layer on the upper layer: from the off-screen canvas corresponding to the second annotation rendering layer (such as the renderer.view of PixiJS), the area corresponding to the current viewport is cut according to the viewport parameters, and is drawn to the upper layer of the off-screen synthesis canvas. Finally, the initial screenshot data containing the slice image and annotation information is generated.

[0038] It is easy to understand that when multiple layers (slice image layer, first annotation rendering layer, and second annotation rendering layer) are directly synthesized, the screenshot may be blurred or the synthesis efficiency may be low due to data format differences and too much redundant information. Therefore, in a feasible implementation manner, with reference to Figure 2 , Figure 2 This is a second flowchart of the first embodiment of the digital pathology slice intelligent screenshot method. In this embodiment, step S20 can include steps A1-A5: Step A1, obtain slice image data from the view engine corresponding to the slice image layer, and draw it on the bottom layer of the off-screen synthesis canvas corresponding to the size parameters; Step A2, cut the off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameters and the size parameters to obtain the first rendering content; Step A3, cut the off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameters and the size parameters to obtain the second rendering content; Step A4, convert the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to the preset canvas optimization rules; Step A5, draw the first optimized annotations to the middle layer of the off-screen synthesis canvas, and draw the second optimized annotations to the upper layer of the off-screen synthesis canvas to generate initial screenshot data.

[0039] It is understood that the front-end device can obtain the slice image data (including pixel color, resolution, etc.) of the current viewport from the "viewer.drawer.canvas" of the view engine (such as OpenSeaDragon) corresponding to the slice image layer, and then call the "drawImage()" of the Canvas API to draw it to the bottom layer (coordinate origin (0, 0)) of the off-screen synthesis canvas as the basic layer of the screenshot.

[0040] Then, the area data matching the current viewport is cut from the image data (such as the "bufferCanvas" of Konva and the "renderer.view" of PixiJS) stored in the off-screen canvas corresponding to the first annotation rendering layer and the second annotation rendering layer, i.e., the off-screen Canvas of the first annotation rendering layer and the second annotation rendering layer, to obtain the first rendering content and the second rendering content.

[0041] Exemplarily, the embodiment can calculate the clipping area (e.g., the area of {x: 1200, y: 800, width: 1024, height: 768} clipped from the off-screen canvas of the first annotation rendering layer) based on the viewport parameter (e.g., the viewport coordinates {x: 1200, y: 800}) and the size parameter (e.g., 1024px x 768px), to obtain the first rendering content (user annotation) and the second rendering content (AI annotation).

[0042] It can be understood that in an implementable manner, the preset canvas optimization rule described above can be a rule for simplifying annotation data, which can be used to filter redundant annotation graphics beyond the current viewport, adjust the clarity of overlapping annotation graphics, convert the first rendering content into first optimized annotation and the second rendering content into second optimized annotation, so as to reduce the data amount and improve the drawing efficiency.

[0043] Exemplarily, the front-end device can simplify a polygon of 100 vertices into a polygon of 20 vertices, retaining the key contour; or when the user annotation and the AI annotation spatially overlap, if the first rendering content is of the "strikeout" type, the user overrules the AI analysis result, at which time the AI annotation can be overlaid and marked as invalid, and the corresponding overlapping part in the second rendering content can be deleted; if the first rendering content is supplementary annotation, the user annotation and the AI annotation can be displayed simultaneously through a semi-transparent blending mode (i.e., the first rendering content remains as it is, and the transparency of the overlapping part of the second rendering content is adjusted to 50%).

[0044] In another implementable manner, in order to meet the requirement of pathological diagnosis on the completeness of screenshot information, the preset canvas optimization rule described above can also be a rule for guaranteeing the completeness of annotation data, which can be used to guarantee the completeness of annotation graphics beyond the current viewport, adjust the clarity of overlapping annotation graphics, convert the first rendering content into first optimized annotation and the second rendering content into second optimized annotation, so as to guarantee the image completeness and improve the screenshot accuracy.

[0045] Exemplarily, the front-end device can check the completeness of the first rendering content and the second rendering content locked by the current viewport through OCR or shape detection, and in the case that the completeness of any annotation is less than 95%, the size of the current viewport and the off-screen composition canvas can be expanded to a range satisfying 95% annotation, and the step of capturing the viewport parameter of the target slice image in the current viewport and the size parameter of the screenshot visible area can be returned, and then the subsequent multi-layer data drawing in sequence can be performed again. When the user annotation and the AI annotation spatially overlap, the specific judgment rule is the same as that in the first embodiment, which will not be described herein again.

[0046] It is easy to understand that the above is only two implementations of the preset canvas optimization rule proposed in the embodiment, and does not mean a limitation, and the user can select as needed.

[0047] Then, the front-end device can continue to draw the initial screenshot data in sequence, that is, draw the first optimization mark (user mark) to the middle layer of the off-screen composite canvas, and draw the second optimization mark (AI mark) to the upper layer (because the user mark has a higher priority), and finally generate initial screenshot data containing the slice image and the optimized mark information.

[0048] Therefore, the embodiment can reduce redundant data and improve layer synthesis efficiency through cropping and optimization processing; and ensure the display priority of the mark layer by drawing in sequence according to the preset canvas optimization rule, so as to ensure that the user mark is not blocked by the AI mark.

[0049] In step S30, scale data is superimposed on the initial screenshot data according to the viewport parameter and the size parameter, and target screenshot data is generated.

[0050] It should be noted that the above scale data can be graphical and textual information (such as a 250μm line segment + “250μm” text) reflecting the correspondence between pixels in the screenshot and actual physical size (such as microns μm, millimeters mm). In the prior art, such data needs to be manually spliced to the screenshot slice by the user after the screenshot, which is low in efficiency and high in error rate, while the embodiment can automatically generate and merge it into the initial screenshot data according to the viewport parameter and the size parameter. After superimposing the scale data, the front-end device generates the final image data of the picture (such as PNG, JPEG format) containing the slice image, the mark information and the dynamic scale, that is, the above target screenshot data, and the front-end device can take the key information such as the slice name and the patient as the name of the target screenshot data, and write the key information into the picture metadata through EXIF. After superimposition is completed, further, the scale of the existing screenshot is fixed (such as only showing “100μm”), which cannot be dynamically adjusted with the viewport scaling, so that the user cannot accurately judge the actual size at different scaling levels, affecting the diagnosis accuracy.

[0051] In a feasible implementation, in the embodiment, step S30 can include steps B1-B4: Step B1, generating a target scale length according to a preset scale factor and the size parameter; Step B2, generating an optimized scale length based on the target scale length and the viewport parameter; Step B3, obtaining an intelligent unit corresponding to the optimized scale length; Step B4, drawing scale data on the initial screenshot data based on the optimized scale length and the intelligent unit according to the analysis result of the superimposable area of the slice image, and generating target screenshot data.

[0052] It is easy to understand that the above preset scale factor can be a coefficient (such as 0.15-0.20, which can be configured according to actual conditions, and the specific value is not limited in the embodiment) for calculating the scale length and the screenshot width ratio, so as to ensure that the scale length is moderate (neither occupies too much screenshot space, nor is not clear and visible). Then the front-end device can calculate the initial scale pixel length based on the screenshot width contained in the size parameter and the preset scale factor, that is, the above target scale length. Exemplarily, assuming that the preset scale factor is 0.17, if the width in the size parameter of the screenshot visible area is 1000px, then the target scale length = 1000px x 0.17 = 170px; meanwhile, the front-end device can further set that the length constraint range of the target scale length is between 30px (minimum) and 200px (maximum), and if the calculation result exceeds the range, the boundary value is taken; the width constraint range can be set to not more than 15% of the screenshot width.

[0053] It is easy to understand that the above optimized scale length can be the actual scale length (such as 42.5μm rounded to 40μm) obtained after the target scale length is converted into a physical length combined with the pixel physical size in the viewport parameter, and then processed by an intelligent rounding strategy. Exemplarily, assuming that the pixel physical size contained in the viewport parameter is 0.25μm / px, and the target scale length is 170px, then the target physical length = 170px x 0.25μm / px = 42.5μm. Then, the front-end device can adopt a user-friendly intelligent rounding strategy (preferentially rounding to 1, 2, 2.5, 5 and times, for example, a calculation value of 237μm can be rounded to 250μm; a calculation value of 0.48mm can be rounded to 0.5mm; a calculation value of 1.23mm can be rounded to 1.2mm or 1.25mm (the specific strategy can be selected)), and 42.5μm is optimized to 40μm, that is, the optimized scale length is 40μm.

[0054] It needs to be understood that the embodiment can automatically switch units (pm->nm) at an ultra-high zoom level, and set a switching threshold in combination with medical standards (such as ASCO / CAP guidelines) to facilitate quick reading and understanding of users. Accordingly, the above-mentioned intelligent unit can be a physical unit (pm or mm) automatically selected by the front-end device according to the optimized scale length, to ensure that the unit is adapted to the length size (such as pm for small length and mm for large length). In the embodiment, the threshold logic of unit switching can be: when switching to mm unit, keep 1 decimal place (avoid 0.25mm and other non-intuitive values). For example: when the scale calculation value is greater than or equal to 1000pm, convert to mm unit and round (such as 1500pm is optimized to 1.5mm); otherwise, use pm (such as 40pm). In addition, if the converted value is in (0.1, 1), keep 1 decimal place; if it is greater than or equal to 1, round to 0.5mm step.

[0055] It can be understood that the above-mentioned superimposable area analysis result can be the analysis result of the initial screenshot data, such as the distribution density (or cancerous area aggregation degree) of pathological tissues in the slice, which determines the blank (i.e. no user annotation and AI annotation) or non-key area (such as an area without important tissues or low tissue density) that is suitable for placing a scale. In addition, in another implementation, the embodiment can also directly superimpose the scale figure and unit text in the initial screenshot data according to the preset configuration item (such as top left, bottom left, top right, bottom right) by the user.

[0056] At the same time, in the embodiment, the line thickness, color, text font / size / color of the scale figure can be customized by the user to ensure clear readability under different backgrounds (dark / light). For example, after the front-end device analyzes the initial screenshot data and determines the superimposable area (such as the lower right corner), the scale can be drawn through the Canvas API: first, draw a horizontal line segment (length corresponding to 40pm, line width 2px, color black); then, draw a vertical short scale (length 5px) at both ends of the line segment; finally, draw the text "40pm" above the line segment (font 12px, color black, background semi-transparent white color to ensure readability).

[0057] Therefore, in the embodiment, the front-end device can dynamically calculate the length and unit (such as μm or mm) of the scale bar based on the viewport parameters and size parameters at the time of triggering the slice screenshot instruction, and draw the scale bar graphics and text at the specified position (such as the lower right corner) of the initial screenshot data through the Canvas API. For example, if the screenshot width is 1000px and the scale factor is 0.17, the scale bar length is 170px; combined with the pixel physical size of 0.25 μm / px, the physical length is calculated to be 42.5 μm, which is rounded to 40 μm, and finally the scale bar of “40 μm” is drawn. That is, the embodiment can break through the mechanical setting of the existing scale bar, dynamically generate a scale bar matching the current viewport, and ensure that the user can accurately determine the actual size of the slice; and by introducing a pathology prior knowledge base (such as tumor cell size distribution), combined with intelligent rounding and unit switching, the readability of the scale bar is improved, and the diagnostic error is reduced.

[0058] In summary, the embodiment can realize pure front-end real-time screenshot, without the participation of the back-end, fast response speed, meet the real-time interaction demand; and automatically synthesize the slice image, annotation information (user annotation, AI result) and dynamic scale bar, without manual splicing, reduce the error; while retaining the complete diagnostic context information, improve the user communication, sharing and archiving efficiency.

[0059] The embodiment provides a digital pathology slice intelligent screenshot method, which is applied to a front-end device and includes the following steps: in response to a slice screenshot instruction, capturing viewport parameters of a target slice image in a current viewport and size parameters of a screenshot visible area; obtaining slice image data from a view engine corresponding to a slice image layer and drawing the slice image data on a bottom layer of an off-screen composition canvas corresponding to the size parameters; cutting off-screen canvas content corresponding to a first annotation rendering layer according to the viewport parameters and the size parameters to obtain first rendering content; cutting off-screen canvas content corresponding to a second annotation rendering layer according to the viewport parameters and the size parameters to obtain second rendering content; converting the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule; drawing the first optimized annotations to a middle layer of the off-screen composition canvas and drawing the second optimized annotations to an upper layer of the off-screen composition canvas to generate initial screenshot data; generating a target scale bar length according to a preset scale factor and the size parameters; generating an optimized scale bar length based on the target scale bar length and the viewport parameters; obtaining an intelligent unit corresponding to the optimized scale bar length; and drawing scale bar data on the initial screenshot data based on the optimized scale bar length and the intelligent unit according to a slice image superimposable area analysis result to generate target screenshot data. The embodiment can realize pure front-end real-time screenshot, without the participation of the back-end, fast response speed, meet the real-time interaction demand; and automatically synthesize the slice image, annotation information (user annotation, AI result) and dynamic scale bar, without manual splicing, reduce the error; while retaining the complete diagnostic context information, improve the user communication, sharing and archiving efficiency.

[0060] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned embodiment one can be referred to the above introduction, and the subsequent will not be described in detail.

[0061] It is easy to understand that in the prior art, the coordinate systems of user labeling (such as manual labeling by doctors) and AI analysis results and other multi-source labeling data are not unified, resulting in misplacement of labeling information and slice images when taking screenshots, and unable to accurately superimpose.

[0062] Therefore, on the basis of the first embodiment, please refer to Figure 3 , Figure 3 The first flowchart of the second embodiment of the digital pathology slice intelligent screenshot method of the present application, in this embodiment, before step S20, the digital pathology slice intelligent screenshot method further includes steps C1-C2: Step C1, a unified labeling protocol is constructed to convert the graphic data of the first labeling source and the second labeling source into first protocol data and second protocol data with the same coordinate system; It is understood that the above-mentioned unified labeling protocol can be a pre-set unified protocol middleware, which is a unified format specification for defining multi-source labeling data, ensuring that labeling data from different sources can be processed under the same coordinate system. The first labeling source can be a user interaction generated labeling data source (such as user drawing labeling through Konva tool). The second labeling source can be a labeling data source generated by AI, large model or algorithm analysis (such as suspicious areas output by AI through slice analysis). The first protocol data and the second protocol data can be data obtained by converting the original data of the first labeling source and the second labeling source into the same labeling protocol respectively.

[0063] Therefore, this embodiment can pre-prepare a protocol specification containing key fields such as "coordinates, graphic types, attribute information, source identification, rendering priority", and convert the original data of the first labeling source (such as Konva graphic data) and the original data of the second labeling source (such as AI output analysis results) into first protocol data and second protocol data with the same format. For example, the user's rectangular labeling and the AI's polygon recognition are both converted into protocol data containing "coordinate array, type (rect / polygon), color, source (user / ai), priority (high / low)".

[0064] In a feasible implementation manner, referring to Figure 4 , Figure 4 The second flowchart of the second embodiment of the digital pathology slice intelligent screenshot method of the present application, in this embodiment, step C1 can include steps C11-C13: Step C11, receiving a first JSON object of the first annotation source output updated based on the user interaction event, the first JSON object carrying key fields including at least coordinates, graphic type and attribute information; Step C12, receiving a second JSON object of the second annotation source output updated based on the slice tissue density of the target slice image, the second JSON object carrying the same key fields as the first JSON object; Step C13, converting the first JSON object and the second JSON object into first protocol data and second protocol data respectively, both of which include source identification and rendering priority.

[0065] It can be understood that the user interaction event can refer to the operation of the user on the annotation tool, such as drawing, modifying and deleting annotations, driving Konva by the event, and performing dirty rectangle update. The JSON format data output by the first annotation source (user annotation) based on the user interaction event, i.e. the first JSON object described above, can include: coordinates (such as the coordinates of the upper left corner and the lower right corner of a rectangle; or the coordinates of the three vertices of a triangle), graphic type (such as rect (rectangle), circle (circle), arrow (arrow), polygon (polygon)), attribute information (such as color (Color), line width (LineWidth), text description (Description)) and the like.

[0066] Exemplarily, when the user draws a rectangular annotation through the annotation tool (such as the drawing component provided by Konva) of the front-end interface, the tool triggers an interaction event, and the first annotation source outputs a first JSON object in real time, for example: { "Coordinates": [{"x":100,"y":100},{"x":200,"y":200}]; "Type":"rect" / / rect with upper left corner coordinates (100, 100) and lower right corner coordinates (200, 200); "Color":"#ff0000"; "LineWidth":2; } The front-end device receives the first JSON object and temporarily stores it.

[0067] It is easy to understand that the above-mentioned slice tissue density can be the density of cells or tissues in the slice image (such as high cell density areas and low cell density areas). In this embodiment, the JSON format data of the second annotation source (output by a lightweight AI model (such as ONNXRuntime front-end inference)), namely the second JSON object, can be dynamically updated based on the slice tissue density. For example, the frequency is low (≤5fps) in areas with normal cell density and increased to 10fps in areas with high cell density, to improve the visual continuity of key diagnostic areas.

[0068] In addition, the second JSON object may contain the same key fields (coordinates, graphic type, attribute information) as the first JSON object, which facilitates subsequent data conversion in the same coordinate system.

[0069] For example, the AI ​​analysis engine detects the tissue density of the slice in real time, outputs more detailed analysis results in high cell density areas (such as tumor areas), and generates a second JSON object, for example: { "Coordinates": [{"x":150,"y":150},{"x":180,"y":160},{"x":170,"y":190}]; "Type":"polygon"; "Color":"#00ff00"; "Description":"Suspicious cell clusters"; } The front-end device receives the second JSON object and temporarily stores it.

[0070] It should be noted that in order to facilitate subsequent sequential drawing, the front-end device also needs to convert the temporarily stored first JSON object and second JSON object into first protocol data and second protocol data containing source identification and rendering priority. Among them, the source identification (Source) can be an identifier to distinguish the source of the annotation data, such as "user" can represent the first annotation source, and "ai" can represent the second annotation source. The rendering priority (layerPriority) can indicate the drawing order of the annotation data, such as "high" and "low" or "001" and "002". In this embodiment, the priority of the first annotation source can be set higher than that of the second annotation source to ensure that the user annotation corresponding to the first annotation source is not obscured by the AI ​​annotation corresponding to the second annotation source.

[0071] Exemplarily, the front-end device may convert the first JSON object into first protocol data (adding a source identifier and rendering priority), for example: { "Coordinates": [{"x": 100, "y": 100}, {"x": 200, "y": 200}]; "Type": "rect"; "Color": "#ff0000"; "LineWidth": 2; "Source": "user"; "layerPriority": "high"; } Similarly, the second JSON object is converted into a second protocol data (source identification "ai", priority "low"), ensuring that the formats are unified.

[0072] Therefore, in view of the problem that the original formats of multi-source annotation data are not unified (such as user annotation output XML format and AI analysis output custom JSON format), it is difficult to directly convert them into unified protocol data, which affects the efficiency of layer synthesis. The embodiment can unify the formats of multi-source annotation data, reduce the complexity of data processing; and through source identification and rendering priority, the rationality of annotation display is ensured.

[0073] It is easy to understand that the AI analysis data source may contain a large amount of data (such as all suspicious areas of the whole slice), and if all of them are directly rendered, it will cause the performance of the front end to decline (such as lag, delay), which affects the user experience. Therefore, in a feasible implementation manner, in the embodiment, the second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

[0074] It can be understood that the above-mentioned slice analysis data source can be a raw data set of the second annotation source (AI analysis), which can include a plurality of sub-data sources such as low-precision analysis data of the whole slice, high-precision analysis data of the local area, etc. For example, in the embodiment, the at least one slice analysis data source corresponding to the second annotation source includes but is not limited to: whole slice low-precision data, 20x zoom level high-precision data, high cell density area detail data, etc.

[0075] The actual graphics data source can be AI analysis data currently required to be rendered to the second annotation rendering layer. The front-end device can filter data matching the current viewport from at least one slice analysis data source according to viewport parameters (such as a zoom level, a viewport coordinate, etc.). For example, when the viewport parameters of the current viewport are that the current zoom level is 20x and the viewport coordinate corresponds to a high cell density area, the front-end device can select the matching "20x zoom level high-precision data" and "high cell density area detail data" from multiple data sources as the actual graphics data source, render only the AI annotation information required by the current viewport, and reduce the data processing amount.

[0076] Therefore, the embodiment can dynamically select required AI analysis data, avoid redundant data rendering, improve front-end performance, and ensure accurate and efficient display of AI annotation information of the current viewport.

[0077] Step C2, mapping the first protocol data to the first annotation rendering layer and the second protocol data to the second annotation rendering layer based on the spatial coordinates of the view engine corresponding to the slice image layer, and synchronously updating the display states of the first annotation rendering layer and the second annotation rendering layer when a viewport change event is listened to.

[0078] It can be understood that the view engine can be a basic engine (such as OpenSeaDragon) used by the front-end device to render a digital pathology slice, and the spatial coordinates can be a global coordinate system of the entire slice. For example, the front-end device can map the coordinates of the first protocol data to the local coordinate system of the first annotation rendering layer based on the spatial coordinates of the view engine, so as to drive the first annotation rendering layer Konva to perform dirty rectangle update (Dirty Rectangle) by a user interaction event, convert the first protocol data into an interactive graphic or text through a Konva API, and map the coordinates of the second protocol data to the local coordinate system of the second annotation rendering layer, and perform graphic rendering through a Pixi API of the second annotation rendering layer PixiJS, so as to ensure that the annotation data corresponds to the position of the slice image.

[0079] In a feasible implementation, the view engine registers a viewport change event listener to broadcast view parameters including a zoom coefficient, a rotation angle, and a view coordinate to the first annotation rendering layer and the second annotation rendering layer when the target slice image is zoomed, translated, or rotated. In the embodiment, step C2 can include steps C21-C22: Step C21, adjusting the geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameters when the viewport change event is listened to. Step C22, calling the drawing interface of the rendering engine corresponding to the first and second annotation rendering layers according to the adjusted geometric transformation matrix to redraw the display state of the first and second annotation rendering layers.

[0080] It can be understood that the above-mentioned viewport change event can be an event detected by the front-end device due to user operation (such as zooming, panning, and rotating) causing the current viewport state of the target slice image to change. The viewport change event listener can be a program module in the view engine for monitoring the change of the viewport state, such as the callback function registered by the “addHandler” interface of OpenSeaDragon. The front-end device can register a viewport change event listener (such as the “viewport-change” event of OpenSeaDragon) for the view engine to listen to the viewport change event in real time. The rendering frequency is real-time (such as 60fps), which can be achieved by OpenSeaDrag dynamic tile loading (LOD). When the viewport changes (such as zooming, panning, and rotating), if the annotation layer (the first and second annotation rendering layers) cannot be updated synchronously, it will cause the annotation information to be misaligned with the slice image, affecting the accuracy of the screenshot.

[0081] It can be understood that the above-mentioned view parameters can be a set of zooming factors, rotation angles, and view coordinates (i.e. the pixel position of the upper left corner of the changed image in the entire slice coordinate system), which are used to describe the state of the changed viewport. Therefore, in the case of viewport change of the target slice image, the front-end device can broadcast the view parameters to the first and second annotation rendering layers through a custom event bus (such as EventEmitter).

[0082] For example, the front-end device can pre-register a listener in the view engine (such as OpenSeaDragon) through “viewer.addHandler(“viewport-change”, callback)”. When the user zooms (such as scroll wheel operation), pans (such as mouse dragging), or rotates the slice, the listener triggers and calculates the view parameters (such as zooming factor 5x, rotation angle 30°, and view coordinates {x: 1000, y: 500}), and then broadcasts them to the first and second annotation rendering layers through a custom event bus. The parameter transmission can be as follows: { zoom: viewer.viewport.getZoom(), / / current zoom factor (such as 5x); rotation: viewer.viewport.getRotation(), / / rotation angle (degrees); position: viewer.viewport.pixelFromPoint(newPoint(0, 0)), / / view coordinate } It should be noted that the above geometric transformation matrix can be a mathematical matrix (such as a 3xn matrix, including scaling factor, rotation angle, and translation amount parameter (including n coordinates)) used to describe scaling, rotation, translation, etc. of the first annotation rendering layer and the second annotation rendering layer. The drawing interface can be a method provided by the rendering engine for redrawing the layer (such as Konva's "draw()" and PixiJS's "render()").

[0083] Therefore, in the embodiment, after receiving the parameters, the first annotation rendering layer and the second annotation rendering layer can adjust the scaling, rotation, and position of themselves synchronously to ensure that the annotation information is always aligned with the slice image. For example, the Konva layer can be adjusted by the "scale()" and "rotation()" methods, and the PixiJS layer can be adjusted by the "stage.scale" and "stage.rotation".

[0084] Subsequently, the first annotation rendering layer calls the "Konva.layer.draw()" interface, and the second annotation rendering layer calls the "Pixijs.app.render()" interface to redraw the layer based on the adjusted matrix, so as to ensure that the annotation information is completely matched with the position, angle, and scaling ratio of the slice image.

[0085] Exemplarily, after the first annotation rendering layer (Konva) and the second annotation rendering layer (PixiJS) receive the view parameters, the respective geometric transformation matrices are adjusted according to the parameters. The geometric transformation matrix can include a scaling factor (zoom), a rotation angle (rotation), and a view coordinate (position). Further, the dynamic redrawing of the first annotation rendering layer (Konva) can be as follows: synchronous scaling by Konva.layer.scale({x: zoom, y: zoom}); synchronous offset by Konva.layer.position({x: position.x, y: position.y}); synchronous rotation by Konva.layer.rotation = rotation; re-canvas rendering by Konva.layer.draw().

[0086] The dynamic redrawing of the second annotation rendering layer (PixiJS) can be as follows: Synchronize zoom by Pixijs.app.stage.scale(zoom); Synchronize offset by Pixijs.app.stage.position.set(position.x, position.y); Synchronize rotation by Pixijs.app.stage.rotation = rotation * (Math.PI / 180); Re-render the canvas by Pixijs.app.render().

[0087] Further, to solve the coordinate system floating-point number error of OpenSeaDragon and Konva, the front-end device can add a coordinate correction function during the redrawing process.

[0088] In this embodiment, real-time synchronization of the labeling layer when the viewport changes can be realized, ensuring that the labeling information is always aligned with the slice image; the transformation parameters are accurately calculated through the geometric transformation matrix, improving the accuracy of labeling display.

[0089] In summary, the present embodiment can solve the coordinate system difference problem of multi-source labeling data, ensure the accurate alignment of labeling information and slice image; and realize real-time synchronization of the labeling layer when the viewport changes, ensuring the coherence of labeling display during user operation.

[0090] The embodiment discloses receiving a first JSON object of a first labeling source output updated based on a user interaction event, the first JSON object carrying key fields including at least coordinates, a graph type and attribute information; receiving a second JSON object of a second labeling source output updated based on a slice image of a target slice, the second JSON object carrying the same key fields as the first JSON object; the second labeling source has at least one slice analysis data source, and an actual graph data source of the second labeling source is selected from the at least one slice analysis data source according to a viewport parameter. The first JSON object and the second JSON object are respectively converted into first protocol data and second protocol data each containing a source identifier and a rendering priority, so as to convert graph data of the first labeling source and the second labeling source into the first protocol data and the second protocol data with the same coordinate system; the first protocol data is mapped to a first labeling rendering layer and the second protocol data is mapped to a second labeling rendering layer based on a space coordinate of a view engine corresponding to a slice image layer, and in the case that a viewport change event is listened to, a geometric transformation matrix corresponding to the first labeling rendering layer and the second labeling rendering layer is adjusted according to the view parameter; a drawing interface of a rendering engine corresponding to the first labeling rendering layer and the second labeling rendering layer is called according to the adjusted geometric transformation matrix, so as to redraw a display state of the first labeling rendering layer and the second labeling rendering layer. The embodiment can solve the problem of coordinate system difference of multi-source labeling data, ensure accurate alignment of labeling information and slice images, and realize real-time synchronization of labeling layers when the viewport changes, and ensure the coherence of labeling display in the user operation process.

[0091] By way of example, in order to facilitate understanding of the technical concept or technical principle of the digital pathology slice intelligent screenshot method combined with the above embodiment one and embodiment two, please refer to Figure 5 , Figure 5 The following is a brief flowchart of the digital pathology slice intelligent screenshot method of the present application: The present application provides a specific method for realizing digital pathology slice intelligent screenshot in the Web browser environment of a front-end device. The method aims to solve the problem that the traditional screenshot method cannot preserve key information such as dynamic scale, user labeling layer and AI analysis result, ensure that the shared pathology screenshot contains complete diagnostic context information, and has real-time and smooth interaction. The method is purely front-end implemented, without the need for backend participation in rendering, which can significantly reduce response delay. As shown in Figure 5 The implementation process of the present application can be as follows: Step 1, initialize view and labeling layer: Load the target slice image (e.g. DICOM or SDPC format) in the front-end device (such as the browser of the user workstation computer). Use OpenSeaDragon (OSD) as the basic view engine to render the slice, and initialize two independent labeling canvas layers: First annotation rendering layer, namely user annotation layer: rendered using Konva library, supports user interactive annotation (rectangle, circle, arrow, text).

[0092] Second annotation rendering layer, namely AI / model analysis layer: rendered using PixiJS library, used to display suspicious areas identified by AI algorithms (such as polygons, dots).

[0093] Register viewport change event listener for OSD view's viewport-change event (zoom, pan, rotate). When the viewport changes, the event handler gets the view parameters: zoom (zoom factor, such as 20x), rotation (rotation angle), position (top-left corner of the visible area in the whole image coordinate system in pixel coordinates). And broadcast the view parameters through a custom event bus (such as EventEmitter).

[0094] Dynamic coordinate mapping: First annotation rendering layer (Konva): receive viewport parameters, adjust geometric transformation matrix, set the size of KonvaLayer to OSD container clientWidth and clientHeight. Call konvaLayer.scale({x:zoom,y:zoom}), konvaLayer.position({x:position.x,y:position.y}), konvaLayer.rotation(rotation), and finally call konvaLayer.draw() to redraw. Ensure that user annotation graphics are consistent with the slice view.

[0095] Second annotation rendering layer (PixiJS): receive viewport parameters, adjust geometric transformation matrix, and call pixiApp.stage.scale.set(zoom,zoom), pixiApp.stage.position.set(position.x,position.y), pixiApp.stage.rotation=rotation*(Math.PI / 180), and finally call pixiApp.render() to redraw. Ensure that AI annotation graphics move, zoom, and rotate with the slice.

[0096] Step 2, trigger screenshot and state capture: Users can click the "screenshot" button on the web interface of the front-end device. The system immediately suspends all interactive events (zoom, pan, rotate) of the OSD. Freeze the current AI layer (PixiJS) and user annotation layer (Konva) graphics state to prevent annotations from being modified during the screenshot process.

[0097] Capture key parameters, viewport parameters of current viewport and size parameters of screenshot visible area: 1)Viewport coordinates: Call OSD API "viewer.viewport.pixelFromPoint(newPoint(0,0))" to get the pixel coordinates of the top-left corner of the current visible area in the entire slice (e.g., {x:1500,y:800}).

[0098] 2)Zoom level: Call OSD API (viewer.viewport.getZoom()) to get the current magnification (e.g., 20x).

[0099] 3)Pixel physical size: Read from slice metadata (e.g., 0.25 μm / pixel).

[0100] 4)Size parameters: Get the clientWidth and clientHeight of the OSD container element as the width and height (px) of the screenshot canvas.

[0101] Step 3, Multi-canvas layer composition: Create an off-screen Canvas element (i.e., off-screen composition canvas) with size parameters corresponding to the size of the screenshot area captured in step 2. Then compose the layers in order from bottom to top: 1)Bottom layer - slice image layer: Directly get the slice image rendered by the current view from the OSD's DrawerCanvas (viewer.drawer.canvas). Draw it to the corresponding position on the composition canvas (usually coordinates (0,0)).

[0102] 2)Middle layer - AI annotation layer: Get the current frozen PixiJS Canvas (pixiApp.renderer.view). According to the viewport coordinates and screenshot size captured in step 2, use Canvas API (ctx.drawImage()) to draw the corresponding visible area in the PixiJS canvas to the composition canvas. Preserve the original style of AI annotations (color, line width).

[0103] 3)Top layer - user annotation layer: Get the current frozen KonvaStageCanvas (konvaStage.toCanvas()). Similarly, according to the parameters captured in step 2, use Canvas API to draw the visible area in the Konva canvas to the top layer of the composition canvas. Preserve the original style of user annotations (color, line width, font). The user annotation layer is by default overlaid on the AI annotation layer.

[0104] In this process, the AI annotation layer or user annotation layer can be filtered by preset canvas optimization rules to filter out redundant annotation graphics that exceed the current viewport, adjust the clarity of overlapping annotation graphics, reduce data volume, and improve drawing efficiency; or to ensure the completeness of annotation graphics that exceed the current viewport, adjust the clarity of overlapping annotation graphics, ensure image integrity, and improve screenshot accuracy. Specific requirements can be set according to actual conditions.

[0105] Step 4, dynamic scale intelligent calculation and drawing: Draw a dynamic scale on the composite canvas: 1) Target length calculation: scale length = screenshot width * scale factor (configuration item, default 0.17, range 0.15-0.20). The result is constrained between the minimum length (30px) and the maximum length (200px). Example: screenshot width 1000px, target length = 1000*0.17 = 170px.

[0106] 2) Actual physical length calculation: physical length = target length (px) * pixel physical size (μm / px). Example: 170px * 0.25μm / px = 42.5μm.

[0107] 3) Intelligent rounding: round the physical length to a user-friendly value. Preferably round to 1, 2, 2.5, 5, etc. or 10^n times thereof. Example: 42.5μm rounded to 40μm; 237μm rounded to 250μm; 1.23mm rounded to 1.2mm or 1.25mm (optional strategy).

[0108] 4) Intelligent unit switching: if the rounded physical length >= 1000μm (1mm), automatically switch to millimeters (mm) and display the text as 1.5mm; otherwise, use microns (μm) and display the text as 250μm. Avoid values that are not easy to estimate (such as 233μm).

[0109] 5) Scale drawing: use CanvasAPI to draw a horizontal line segment (thickness and color can be configured) at the specified position (configurable: top left, bottom left, top right, bottom right, default bottom right) on the composite canvas or in the analysis result of the overlayable area of the slice image, and draw vertical short scale lines at both ends of the line segment. Then draw a text label (content: rounded physical length + unit, such as 40μm) above or below the scale (font, size, color can be configured to ensure clear contrast with the background). In addition, according to user requirements, draw the zoom level text (such as 20x) corresponding to the current viewport near the scale data.

[0110] Step 5, output integrated picture: The toDataURL() or toBlob() method of the composite canvas is called to convert the final synthesized image data into a picture (such as PNG, JPEG format). Further, a picture file name that is easy for users to identify or retrieve can be generated: it can contain information such as slice name, patient ID, timestamp (for example, Patient123_LiverBiopsy_20230806_1600.png). Optionally, libraries such as EXIF can be used to write key information (such as zoom level 20x, pixel size 0.25 μm / px, screenshot time) into the metadata of the picture file.

[0111] Step 6, restore interaction: After completing the screenshot, the interaction event of the OSD is re-enabled, the frozen state of the AI layer and the user annotation layer is released, and the user can continue to operate.

[0112] Therefore, the present application can realize pure front-end intelligent screenshot of digital pathology slices, ensure that the output picture contains complete diagnostic information (original slice, dynamic scale, user annotation, AI result), and eliminate the tediousness and errors of manual splicing. Through the precise coordinate mapping and layer synthesis mechanism, the position accuracy of the annotation information in the screenshot is ensured. The intelligent calculation and drawing of the dynamic scale provide accurate and easy-to-read scale reference. The whole process responds quickly, the user experience is smooth, and it has good browser compatibility, significantly improving the efficiency and accuracy of users to share, communicate and archive diagnostic information.

[0113] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the method of intelligent screenshot of digital pathology slices of the present application. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0114] The present application also provides an intelligent screenshot device for digital pathology slices, which is described in detail in the following. Figure 6 , Figure 6 The present application also provides an intelligent screenshot device for digital pathology slices, which is described in detail in the following. The parameter acquisition module 601 is configured to capture the viewport parameter of the target slice image in the current viewport and the size parameter of the screenshot visible area in response to a slice screenshot instruction; The preliminary rendering module 602 is configured to draw the slice image layer, the first annotation rendering layer and the second annotation rendering layer in the current viewport in sequence to the off-screen composite canvas corresponding to the size parameter according to the viewport parameter, and generate initial screenshot data; The target rendering module 603 is configured to superimpose scale data on the initial screenshot data according to the viewport parameter and the size parameter, and generate target screenshot data.

[0115] As an implementable manner, in this embodiment, the preliminary rendering module 602 is further configured to construct a unified annotation protocol to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data having the same coordinate system; map the first protocol data to the first annotation rendering layer and the second protocol data to the second annotation rendering layer based on the spatial coordinates of the view engine corresponding to the slice image layer; and in the case of listening to a viewport change event, synchronously update the display states of the first annotation rendering layer and the second annotation rendering layer.

[0116] As an implementable manner, in this embodiment, the preliminary rendering module is further configured to receive a first JSON object output by the first annotation source based on a user interaction event update, the first JSON object carrying key fields including at least coordinates, graphic types and attribute information; receive a second JSON object output by the second annotation source based on slice tissue density update of the target slice image, the second JSON object carrying the same key fields as the first JSON object; and convert the first JSON object and the second JSON object into first protocol data and second protocol data respectively, both of which include source identification and rendering priority.

[0117] As an implementable manner, in this embodiment, the second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

[0118] As an implementable manner, the view engine registers a viewport change event listener to broadcast view parameters including a scaling coefficient, a rotation angle and a view coordinate to the first annotation rendering layer and the second annotation rendering layer in the case of zooming, panning or rotating the target slice image; in this embodiment, the preliminary rendering module is further configured to adjust the geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameters in the case of listening to the viewport change event; and call the drawing interface of the rendering engine corresponding to the first annotation rendering layer and the second annotation rendering layer according to the adjusted geometric transformation matrix to redraw the display states of the first annotation rendering layer and the second annotation rendering layer.

[0119] As an implementable manner, in this embodiment, the preliminary rendering module is further configured to acquire slice image data from a view engine corresponding to the slice image layer, and draw the slice image data on a bottom layer of an off-screen composite canvas corresponding to the size parameter; crop off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameter and the size parameter to obtain first rendering content; crop off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameter and the size parameter to obtain second rendering content; convert the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule; draw the first optimized annotations to a middle layer of the off-screen composite canvas, and draw the second optimized annotations to an upper layer of the off-screen composite canvas to generate initial screenshot data.

[0120] As an implementable manner, in this embodiment, the target rendering module is further configured to generate a target scale length according to a preset scale factor and the size parameter; generate an optimized scale length based on the target scale length and the viewport parameter; acquire an intelligent unit corresponding to the optimized scale length; draw scale data on the initial screenshot data based on the optimized scale length and the intelligent unit according to a superimposable region analysis result of the slice image to generate target screenshot data.

[0121] The digital pathology slice intelligent screenshot device provided in the present application adopts the intelligent screenshot method in the above embodiments, and can solve the technical problems in the prior art that digital pathology slice screenshots cannot save dynamic information, need to be manually spliced, and have poor real-time performance, and cannot meet the real-time screenshot interaction requirements of digital pathology slices. Compared with the prior art, the device provided in the present application realizes millisecond-level response of complete screenshot information by implementing multi-source annotation protocol construction, dynamic coordinate mapping and intelligent scale generation through a pure front end, eliminates human error and improves diagnosis efficiency, and meets the real-time screenshot interaction requirements of digital pathology slices. Other technical features in the device are the same as the features disclosed in the above method embodiments, and will not be repeated here.

[0122] The present application provides a digital pathology slice intelligent screenshot device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the digital pathology slice intelligent screenshot method in the above embodiment one.

[0123] Reference will be made to the following Figure 7, which shows a schematic diagram of the structure of a digital pathology slide intelligent screenshot device suitable for implementing the embodiments of the present application. The digital pathology slide intelligent screenshot device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones equipped with browsers, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The digital pathology slice intelligent screenshot device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 7 As shown, the intelligent digital pathology slide screenshot device may include a processing device 1001 (e.g., a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the intelligent digital pathology slide screenshot device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the digital pathology slide intelligent screenshot device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a digital pathology slide intelligent screenshot device with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0125] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a digital pathology section intelligent screenshot program product, which includes a digital pathology section intelligent screenshot program carried on a computer readable medium, and the digital pathology section intelligent screenshot program contains program codes for executing the method shown in the flowchart. In such embodiments, the digital pathology section intelligent screenshot program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the digital pathology section intelligent screenshot program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0126] The digital pathology section intelligent screenshot device provided by the present application adopts the digital pathology section intelligent screenshot method in the above-mentioned embodiments, and can solve the technical problem of how to meet the real-time screenshot interaction requirement of digital pathology sections. Compared with the prior art, the digital pathology section intelligent screenshot device provided by the present application has the same beneficial effects as the digital pathology section intelligent screenshot method provided by the above-mentioned embodiments, and other technical features in the digital pathology section intelligent screenshot device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0127] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0129] The present application provides a storage medium having computer readable program instructions (i.e. digital pathology section intelligent screenshot program) stored thereon, and the computer readable program instructions are used to execute the digital pathology section intelligent screenshot method in the above-mentioned embodiments.

[0130] The storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the storage medium may, for example, include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the storage medium can be transmitted in any suitable medium, including, but not limited to, an electrical wire, an optical cable, an RF (Radio Frequency) cable, or the like, or any suitable combination of the above.

[0131] The storage medium described above can be included in the digital pathology slice intelligent screenshot device; or can exist separately and not be assembled into the digital pathology slice intelligent screenshot device.

[0132] The storage medium described above carries one or more programs, which, when executed by the digital pathology slice intelligent screenshot device, cause the digital pathology slice intelligent screenshot device to: digitally take a pathology slice intelligent screenshot.

[0133] The digital pathology slice intelligent screenshot program code for performing the operations of the present application can be written in one or more programming languages or combinations of languages including an object-oriented programming language such as Java, Smalltalk, C++, or a conventional procedural programming language such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In situations in which the remote computer is involved, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through an Internet service provider to connect through the Internet).

[0134] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and digital pathology slide intelligent screenshot program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems which perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation of the unit itself in some cases.

[0136] The readable storage medium provided by the present application is a storage medium, and the storage medium stores computer readable program instructions (i.e. digital pathology slide intelligent screenshot program) for executing the above-mentioned digital pathology slide intelligent screenshot method, which can solve the technical problem of how to meet the real-time screenshot interaction demand of digital pathology slide. Compared with the prior art, the beneficial effects of the storage medium provided by the present application are the same as those of the digital pathology slide intelligent screenshot method provided by the above-mentioned embodiments, which will not be repeated here.

[0137] The above is only some embodiments of the present application, and does not limit the scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the protection scope of the present application.

Claims

1. A digital pathology section intelligent screenshot method, characterized in that: The method is applied to a front-end device, and includes: In response to the slice screenshot instruction, capturing the viewport parameters of the target slice image in the current viewport and the size parameters of the screenshot visible area; Based on the viewport parameters, the slice image layer, the first annotated rendering layer, and the second annotated rendering layer locked in the current viewport are sequentially drawn to the off-screen composition canvas corresponding to the size parameters to generate initial screenshot data; Scale data is superimposed on the initial screenshot data according to the viewport parameters and the size parameters to generate target screenshot data.

2. The method for intelligently capturing digital pathological sections according to claim 1, wherein: The method of sequentially rendering the slice image layer, the first annotated rendering layer, and the second annotated rendering layer locked in the current viewport to the off-screen synthesis canvas corresponding to the viewport parameters according to the viewport parameters, before generating the initial screenshot data, includes: Constructing a unified annotation protocol to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data having the same coordinate system; Based on the spatial coordinates of the view engine corresponding to the slice image layer, the first protocol data is mapped to the first annotation rendering layer, and the second protocol data is mapped to the second annotation rendering layer. When a viewport change event is monitored, the display status of the first annotation rendering layer and the second annotation rendering layer are synchronously updated.

3. The method for intelligently capturing digital pathological sections according to claim 2, wherein: The steps of constructing a unified annotation protocol include: Receive a first JSON object output by a first annotation source updated based on a user interaction event, wherein the first JSON object carries key fields including at least coordinates, graphic type, and attribute information; receiving a second JSON object output by a second annotation source based on the slice tissue density update of the target slice image, wherein the second JSON object carries the same key fields as the first JSON object; The first JSON object and the second JSON object are respectively converted into first protocol data and second protocol data, both of which include a source identifier and a rendering priority.

4. The method for intelligently capturing digital pathological sections according to claim 3, wherein: The second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

5. The method for intelligently capturing digital pathological sections according to claim 3, wherein: The view engine is registered with a viewport change event listener to broadcast view parameters including a zoom factor, a rotation angle, and view coordinates to the first annotation rendering layer and the second annotation rendering layer when the target slice image is scaled, translated, or rotated; The step of synchronously updating the display states of the first annotation rendering layer and the second annotation rendering layer when a viewport change event is monitored includes: When a viewport change event is monitored, adjusting the geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameter; The rendering interfaces of the rendering engines corresponding to the first annotation rendering layer and the second annotation rendering layer are called according to the adjusted geometric transformation matrix to redraw the display states of the first annotation rendering layer and the second annotation rendering layer.

6. The method for intelligently capturing digital pathological sections according to claim 1, wherein: The step of sequentially rendering the slice image layer, the first annotated rendering layer, and the second annotated rendering layer locked in the current viewport to the off-screen synthesis canvas corresponding to the size parameter to generate initial screenshot data according to the viewport parameter includes: Obtain the slice image data from the view engine corresponding to the slice image layer, and draw it on the bottom layer of the off-screen composite canvas corresponding to the size parameters; Clipping the off-screen canvas content corresponding to the first marked rendering layer according to the viewport parameters and the size parameters to obtain first rendering content; Clipping the off-screen canvas content corresponding to the second annotated rendering layer according to the viewport parameters and the size parameters to obtain second rendering content; Converting the first rendering content and the second rendering content into a first optimized annotation and a second optimized annotation according to a preset canvas optimization rule; The first optimized annotation is drawn to the middle layer of the off-screen synthesis canvas, and the second optimized annotation is drawn to the upper layer of the off-screen synthesis canvas to generate initial screenshot data.

7. The method for intelligently capturing digital pathological sections according to claim 1, wherein: The step of superimposing scale data on the initial screenshot data according to the viewport parameters and the size parameters to generate target screenshot data comprises: generating a target scale length according to a preset scale factor and the size parameter; generating an optimized scale length based on the target scale length and the viewport parameter; Obtain the intelligent unit corresponding to the optimized scale length; According to the superimposable area analysis result of the slice image, scale data is drawn on the initial screenshot data based on the optimized scale length and the smart unit to generate target screenshot data.

8. A digital pathology slice intelligent screenshot device, characterized in that: The digital pathology slice intelligent screenshot device comprises: A parameter acquisition module, configured to capture the viewport parameters of the target slice image in the current viewport and the size parameters of the screenshot visible area in response to the slice screenshot instruction; A preliminary rendering module, for the viewport parameters, sequentially rendering the slice image layer, the first annotated rendering layer, and the second annotated rendering layer locked in the current viewport to an off-screen composite canvas corresponding to the size parameters, to generate initial screenshot data; A target rendering module is used to superimpose scale data on the initial screenshot data according to the viewport parameters and the size parameters to generate target screenshot data.

9. A digital pathology slice intelligent screenshot device, characterized in that: The device includes: a memory, a processor, and a digital pathology slice intelligent screenshot program stored in the memory and executable on the processor, wherein the digital pathology slice intelligent screenshot program is configured to implement the steps of the digital pathology slice intelligent screenshot method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a digital pathology slice intelligent screenshot program, and when the digital pathology slice intelligent screenshot program is executed by the processor, the steps of the digital pathology slice intelligent screenshot method according to any one of claims 1 to 7 are implemented.

Images