Digital pathology slice intelligent screenshot method, device and equipment and storage medium

Abstract

The application discloses a kind of digital pathological section intelligent screenshot method, device, equipment and storage medium, and the application relates to medical data processing technical field, the method is applied to front-end equipment, comprising: in response to section screenshot instruction, the size parameter of the viewport parameter and screenshot visible area of target section image in current viewport is captured;According to the viewport parameter, the section image layer locked in current viewport, first annotation rendering layer and second annotation rendering layer are sequentially drawn to the off-screen composition canvas corresponding to size parameter, generate initial screenshot data;According to the viewport parameter and size parameter, scale data is superimposed on initial screenshot data, and target screenshot data is generated.The application can automatically synthesize section image, multi-source annotation information and dynamic scale when carrying out digital pathological section screenshot, without manual splicing, reduce error, and realize pure front-end real-time screenshot, without the participation of back-end, response speed is fast, satisfy real-time interaction demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and particularly relates to a digital pathological section intelligent screenshot method and device, equipment and a storage medium. BACKGROUND

[0002] The digital pathological section is a technology of analyzing and diagnosing after digitizing the pathological section, and the section screenshot can assist users in diagnosis, sharing and information retention.

[0003] However, the existing digital pathological section screenshot only saves the original image data, cannot save the dynamic information of the digital pathological section, needs manual post-processing, is low in efficiency and prone to errors, and cannot meet the real-time interaction requirement.

[0004] Therefore, how to meet the real-time screenshot interaction requirement of the digital pathological section becomes a problem to be solved. SUMMARY

[0005] The main purpose of the present application is to provide a digital pathological section intelligent screenshot method, device, equipment and storage medium, and aims to solve the technical problem of how to meet the real-time screenshot interaction requirement of the digital pathological section.

[0006] To achieve the above purpose, the present application provides a digital pathological section intelligent screenshot method, which comprises the following steps:

[0007] In response to a section screenshot instruction, the viewport parameters of a target section image in a current viewport and the size parameters of a screenshot visible area are captured;

[0008] According to the viewport parameters, a section image layer, a first annotation rendering layer and a second annotation rendering layer locked in the current viewport are sequentially drawn to an off-screen composition canvas corresponding to the size parameters to generate initial screenshot data;

[0009] According to the viewport parameters and the size parameters, scale data is superimposed on the initial screenshot data to generate target screenshot data.

[0010] In an embodiment, before the step of sequentially drawing the section image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport to the off-screen composition canvas corresponding to the viewport parameters according to the viewport parameters to generate the initial screenshot data, the method further comprises the following steps:

[0011] A unified annotation protocol is constructed to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data having the same coordinate system;

[0012] The first protocol data is mapped to a first annotation rendering layer and the second protocol data is mapped to a second annotation rendering layer based on spatial coordinates of a view engine corresponding to a slice image layer, and a display state of the first annotation rendering layer and the second annotation rendering layer is synchronously updated in a case where a viewport change event is listened to.

[0013] In an embodiment, the step of constructing the unified annotation protocol comprises:

[0014] A first JSON object of a first annotation source output updated based on a user interaction event is received, and the first JSON object carries key fields including at least coordinates, a graphic type and attribute information;

[0015] A second JSON object of a second annotation source output updated based on slice tissue density of the target slice image is received, and the second JSON object carries the same key fields as the first JSON object;

[0016] The first JSON object and the second JSON object are respectively converted into first protocol data and second protocol data both including source identification and rendering priority.

[0017] In an embodiment, the second annotation source has at least one slice analysis data source, and actual graphic data sources of the second annotation source are selected from the at least one slice analysis data source according to the viewport parameter.

[0018] In an embodiment, the view engine registers a viewport change event listener to broadcast view parameters including a zoom coefficient, a rotation angle and view coordinates to a first annotation rendering layer and a second annotation rendering layer in a case where the target slice image is zoomed, translated or rotated;

[0019] The step of synchronously updating a display state of the first annotation rendering layer and the second annotation rendering layer in a case where a viewport change event is listened to comprises:

[0020] In a case where a viewport change event is listened to, a geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer is adjusted according to the view parameters;

[0021] A drawing interface of a rendering engine corresponding to the first annotation rendering layer and the second annotation rendering layer is called according to the adjusted geometric transformation matrix, so as to redraw the display state of the first annotation rendering layer and the second annotation rendering layer.

[0022] In an embodiment, the step of generating initial screenshot data by sequentially drawing the slice image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport to an off-screen composition canvas corresponding to the size parameter according to the view parameters comprises:

[0023] obtaining slice image data from a view engine corresponding to the slice image layer, and drawing on a bottom layer of an off-screen composite canvas corresponding to the size parameter;

[0024] clipping off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameter and the size parameter to obtain first rendering content;

[0025] clipping off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameter and the size parameter to obtain second rendering content;

[0026] converting the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule;

[0027] drawing the first optimized annotations to a middle layer of the off-screen composite canvas and drawing the second optimized annotations to an upper layer of the off-screen composite canvas to generate initial screenshot data.

[0028] In an embodiment, the step of superimposing scale data on the initial screenshot data according to the viewport parameter and the size parameter to generate target screenshot data comprises:

[0029] generating a target scale length according to a preset scale factor and the size parameter;

[0030] generating an optimized scale length based on the target scale length and the viewport parameter;

[0031] obtaining an intelligent unit corresponding to the optimized scale length;

[0032] drawing scale data on the initial screenshot data based on the optimized scale length and the intelligent unit according to a superimposable region analysis result of the slice image to generate target screenshot data.

[0033] In addition, to achieve the above-mentioned purpose, the present application further provides a digital pathology slice intelligent screenshot device, which comprises:

[0034] a parameter acquisition module configured to capture a viewport parameter of a target slice image in a current viewport and a size parameter of a screenshot visible region in response to a slice screenshot instruction;

[0035] a preliminary rendering module configured to draw slice image layers, first annotation rendering layers and second annotation rendering layers locked in the current viewport in sequence to an off-screen composite canvas corresponding to the size parameter according to the viewport parameter to generate initial screenshot data;

[0036] A target rendering module is configured to superimpose scale data on the initial screenshot data according to the viewport parameter and the size parameter, and generate target screenshot data.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a digital pathology slice intelligent screenshot device, which comprises a memory, a processor, and a digital pathology slice intelligent screenshot program stored in the memory and executable on the processor. The digital pathology slice intelligent screenshot program is configured to implement the steps of the digital pathology slice intelligent screenshot method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium. The storage medium stores a digital pathology slice intelligent screenshot program. When the digital pathology slice intelligent screenshot program is executed by a processor, the steps of the digital pathology slice intelligent screenshot method as described above are implemented.

[0039] The present application provides a digital pathology slice intelligent screenshot method, device, equipment and storage medium. The method is applied to a front-end device, which comprises: in response to a slice screenshot instruction, capturing the viewport parameter of a target slice image in a current viewport and the size parameter of a screenshot visible area; according to the viewport parameter, sequentially drawing the slice image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport to an off-screen composition canvas corresponding to the size parameter, to generate initial screenshot data; and superimposing scale data on the initial screenshot data according to the viewport parameter and the size parameter, to generate target screenshot data. The present application can automatically synthesize slice images, multi-source annotation information and dynamic scales when performing digital pathology slice screenshot, without manual splicing, reducing errors, and realizing pure front-end real-time screenshot without the participation of the back-end, with fast response speed and meeting real-time interaction requirements. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0042] Figure 1 The first flowchart of the first embodiment of the digital pathology slice intelligent screenshot method of the present application;

[0043] Figure 2 The second flowchart of the first embodiment of the digital pathology slice intelligent screenshot method of the present application;

[0044] Figure 3 Figure 1 is a first flowchart of the digital pathology slice intelligent screenshot method according to an embodiment of the present application;

[0045] Figure 4 Figure 2 is a second flowchart of the digital pathology slice intelligent screenshot method according to an embodiment of the present application;

[0046] Figure 5 Figure 3 is a schematic diagram of the digital pathology slice intelligent screenshot method according to an embodiment of the present application;

[0047] Figure 6 Figure 4 is a schematic diagram of the module structure of the digital pathology slice intelligent screenshot device according to an embodiment of the present application;

[0048] Figure 7 Figure 5 is a schematic diagram of the device structure of the hardware running environment involved in the digital pathology slice intelligent screenshot method according to an embodiment of the present application.

[0049] The object, function features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0051] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings and specific embodiments.

[0052] The main solution of the present application is: in response to a slice screenshot instruction, capturing the viewport parameters of a target slice image in a current viewport and the size parameters of a screenshot visible area; according to the viewport parameters, sequentially rendering a slice image layer, a first annotation rendering layer and a second annotation rendering layer locked in the current viewport to an off-screen composition canvas corresponding to the size parameters, to generate initial screenshot data; and superimposing scale data on the initial screenshot data according to the viewport parameters and the size parameters, to generate target screenshot data.

[0053] The existing digital pathology slice screenshot only saves original image data, and cannot retain dynamic information (such as scales changing with zooming and superimposed annotation layers), needs to be manually spliced later, is low in efficiency and prone to errors, and cannot meet the real-time interaction requirements.

[0054] To solve the above problems, the application provides an intelligent screenshot method realized by pure front end, which can automatically synthesize the current dynamic scale of the slice (real-time calculation with zoom level), multi-source annotation layer, i.e., the annotation of the first annotation rendering layer (user-annotated area) and the annotation of the second annotation rendering layer (suspicious area analyzed by AI), and the digital pathology slice image, without manual splicing, reducing errors, preserving complete diagnostic context information, and improving user communication, sharing, and archiving efficiency. Therefore, the application can realize pure front-end real-time screenshot without the participation of the back end, fast response, ensure the integrity, layout aesthetics, and smoothness of the information, and meet the real-time interaction requirements.

[0055] It should be noted that the execution subject of the embodiment can be a digital pathology slice intelligent screenshot system, or a front-end device with data processing, network communication, and program running functions, such as a user workstation, tablet computer, personal computer, mobile phone, etc. installed with a browser, or a digital pathology slice intelligent screenshot device capable of realizing the above functions, etc. The embodiment does not make specific limitations on this. The following takes the front-end device as the execution subject to illustrate the embodiment and the following embodiments.

[0056] Based on this, the application embodiment provides a digital pathology slice intelligent screenshot method, referring to Figure 1 , Figure 1 The first flowchart of the first embodiment of the digital pathology slice intelligent screenshot method of the application is shown in FIG. 1.

[0057] In the embodiment, the digital pathology slice intelligent screenshot method includes steps S10-S30:

[0058] Step S10, in response to a slice screenshot instruction, capturing the viewport parameters of the target slice image in the current viewport and the size parameters of the screenshot visible area;

[0059] It should be understood that the slice screenshot instruction can be a screenshot operation signal triggered by the user through the front-end interface, which can be triggered by, for example, clicking the “screenshot” button, pressing a pre-set shortcut key, etc. The above-mentioned target slice image can be an original digital pathology slice (such as a DICOM, SDPC format digital pathology slice) that needs to be screenshot currently.

[0060] The current viewport can be a slice image region currently visible to the user in the display interface of the front-end device (i.e., the visual window region of the browser or application). Accordingly, the viewport parameter can be key data describing the current viewport state, including but not limited to viewport coordinates (i.e., the pixel position of the top-left corner of the current visual region in the slice coordinate system of the entire target slice image, such as {x: 1200, y: 800}, unit: pixel (px)), a zoom factor (i.e., the magnification of the current digital pathology slice corresponding to the original target slice image, such as 5x, 20x), and pixel size information (the physical size corresponding to each pixel in the slice data, such as 0.25 pm / pixel (micrometers per pixel)). The size parameter of the screenshot visual region can be the width and height (unit: pixel) of the user's screenshot slice region, which is usually consistent with the display size of the current viewport (e.g., 1024px x 768px).

[0061] For example, when the user clicks the "screenshot" button of the front-end interface (triggering a slice screenshot instruction), the front-end device can pause the interactive behavior of the slice and capture the viewport parameters and size parameters in real time through the API of the view engine (such as OpenSeaDragon) or Konva.

[0062] For example, the zoom factor can be obtained through "viewer.viewport.getZoom()" of OpenSeaDragon or Konva's bufferCanvas, the rotation angle can be obtained through "viewer.viewport.getRotation()", and the size parameter can be obtained through "viewer.container.clientWidth" and "viewer.container.clientHeight".

[0063] In step S20, according to the viewport parameter, the slice image layer, the first annotation rendering layer, and the second annotation rendering layer locked in the current viewport are sequentially rendered to the off-screen composition canvas corresponding to the size parameter to generate initial screenshot data.

[0064] It is easy to understand that, in the case of detecting a slice screenshot instruction, the front-end device can pause the interactive operation (such as zooming, panning, and rotating) of the current viewport, freeze the current state of the slice image layer, the first annotation rendering layer, and the second annotation rendering layer, and ensure that the layer content does not change during the screenshot process.

[0065] The slice image layer can be a layer displaying an original digital pathology slice image, and the data is derived from a slice source file (such as an sdpc or dcm format file) of the target slice image. The first annotation rendering layer and the second annotation rendering layer can be used to display annotation information. In this embodiment, the first annotation rendering layer can correspond to a user manual annotation layer, such as a rectangle, a circle, an arrow, and the like drawn by the user through an interactive tool. The second annotation rendering layer can correspond to a layer of image analysis results of the target slice image obtained by pre-setting a lightweight AI model (such as an ONNX Runtime front-end inference) or a pre-trained neural network structure, such as a suspicious area polygon, a cell dot, or a text note (such as “high-risk area, biopsy recommended”) identified by an AI or a large model.

[0066] It should be noted that the off-screen composition canvas can be a Canvas element in the front-end device that is not directly displayed on the interface, and is used for temporarily synthesizing the data of each layer (to avoid visual flicker caused by directly operating the display canvas). The size of the off-screen composition canvas is consistent with the size parameter of the screenshot visible area. The image data of each layer in the off-screen composition canvas after being synthesized in sequence without the scale bar is the initial screenshot data.

[0067] For example, in this embodiment, the front-end device can draw the locked slice image layer, the first annotation rendering layer, and the second annotation rendering layer in the order of “bottom layer-middle layer-top layer” to the off-screen composition canvas according to the captured viewport parameter, to form the initial screenshot data.

[0068] 1) The slice image layer is drawn on the bottom layer. The slice image data of the current viewport can be obtained through “viewer.drawer.canvas” of the view engine, and the “drawImage()” method of the Canvas API is called to draw the slice image data on the bottom layer of the off-screen composition canvas.

[0069] 2) The first annotation rendering layer is drawn on the middle layer. The first annotation rendering layer is drawn on the middle layer of the off-screen composition canvas by cropping the current viewport corresponding area from the off-screen canvas corresponding to the first annotation rendering layer (such as the bufferCanvas of Konva) according to the viewport parameter.

[0070] 3) The second annotation rendering layer is drawn on the top layer. The second annotation rendering layer is drawn on the top layer of the off-screen composition canvas by cropping the current viewport corresponding area from the off-screen canvas corresponding to the second annotation rendering layer (such as the renderer.view of PixiJS) according to the viewport parameter. Finally, the initial screenshot data containing the slice image and the annotation information is generated.

[0071] It is easy to understand that when the multiple layers (slice image layer, first annotation rendering layer, second annotation rendering layer) are directly synthesized, the screenshot may be blurred or the synthesis efficiency may be low due to data format differences and too much redundant information. Therefore, in a feasible implementation manner, referring to Figure 2 , Figure 2 FIG. 2 is a second flowchart of the first embodiment of the digital pathology slice intelligent screenshot method of the present application. In this embodiment, step S20 can include steps A1-A5:

[0072] Step A1, obtaining slice image data from the view engine corresponding to the slice image layer and drawing on the bottom layer of the off-screen synthesis canvas corresponding to the size parameter;

[0073] Step A2, cutting the off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameter and the size parameter to obtain the first rendering content;

[0074] Step A3, cutting the off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameter and the size parameter to obtain the second rendering content;

[0075] Step A4, converting the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule;

[0076] Step A5, drawing the first optimized annotations to the middle layer of the off-screen synthesis canvas and drawing the second optimized annotations to the upper layer of the off-screen synthesis canvas to generate initial screenshot data.

[0077] It is understood that the front-end device can obtain the slice image data (including pixel color, resolution, etc.) of the current viewport from the “viewer.drawer.canvas” of the view engine (such as OpenSeaDragon) corresponding to the slice image layer, and then call the “drawImage()” of the Canvas API to draw it to the bottom layer (coordinate origin (0, 0)) of the off-screen synthesis canvas as the basic layer of the screenshot.

[0078] Then, the area data matching the current viewport is cut out from the image data (such as Konva's “bufferCanvas”, PixiJS's “renderer.view”) stored in the off-screen canvas content corresponding to the first annotation rendering layer and the second annotation rendering layer, i.e., the off-screen Canvas of the first annotation rendering layer and the second annotation rendering layer, respectively, to obtain the first rendering content and the second rendering content.

[0079] Exemplarily, the embodiment can calculate the clipping area (e.g., the area of {x: 1200, y: 800, width: 1024, height: 768} clipped from the off-screen canvas of the first annotation rendering layer) based on the viewport parameter (e.g., the viewport coordinates {x: 1200, y: 800}) and the size parameter (e.g., 1024px x 768px), to obtain the first rendering content (user annotation) and the second rendering content (AI annotation).

[0080] It can be understood that, in an implementable manner, the preset canvas optimization rule described above can be a rule for simplifying the annotation data, which can be used to filter redundant annotation graphics beyond the current viewport, adjust the clarity of overlapping annotation graphics, convert the first rendering content into first optimized annotation and the second rendering content into second optimized annotation, so as to reduce the data amount and improve the drawing efficiency.

[0081] Exemplarily, the front-end device can simplify the polygon of 100 vertices into a polygon of 20 vertices, and retain the key contour; or when the user annotation and the AI annotation are spatially overlapped, if the first rendering content is a "strikeout" type, the user can override the AI analysis result, at this time, the AI annotation can be covered and marked as invalid, and the corresponding overlapping part in the second rendering content can be deleted; or if the first rendering content is a supplementary annotation, the user annotation and the AI annotation can be displayed simultaneously by using a semi-transparent mixing mode (i.e., the first rendering content remains unchanged, and the transparency of the overlapping part of the second rendering content is adjusted to 50%).

[0082] In another implementable manner, in order to meet the requirement of pathological diagnosis on the completeness of the screenshot information, the preset canvas optimization rule described above can also be a rule for guaranteeing the completeness of the annotation data, which can be used to guarantee the completeness of the annotation graphics beyond the current viewport, adjust the clarity of the overlapping annotation graphics, convert the first rendering content into first optimized annotation and the second rendering content into second optimized annotation, so as to guarantee the image completeness and improve the screenshot accuracy.

[0083] Exemplarily, the front-end device can check the completeness of the first rendering content and the second rendering content locked in the current viewport by using OCR or shape detection, and in the case that the completeness of any annotation is less than 95%, the size of the current viewport and the off-screen composition canvas can be expanded to a range meeting 95% annotation, and the step of capturing the viewport parameter of the target slice image in the current viewport and the size parameter of the screenshot visible area can be returned, and then the subsequent multi-layer data drawing in sequence can be performed again. When the user annotation and the AI annotation are spatially overlapped, the specific judgment rule is the same as that in the first embodiment, which will not be described herein again.

[0084] It can be understood that the above is only two implementable manners of the preset canvas optimization rule proposed by the embodiment, and does not mean a limitation thereon, and the user can select as needed.

[0085] Then, the front-end device can continue to draw the initial screenshot data in sequence, i.e., draw the first optimized annotation (user annotation) to the middle layer of the off-screen composite canvas, and draw the second optimized annotation (AI annotation) to the upper layer (because the user annotation has a higher priority), and finally generate initial screenshot data containing the slice image and the optimized annotation information.

[0086] Therefore, the embodiment can reduce redundant data and improve the efficiency of layer composition through cropping and optimization processing; and ensure the display priority of the annotation layer by drawing in sequence according to the preset canvas optimization rules, so as to ensure that the user annotation is not blocked by the AI annotation.

[0087] In step S30, scale data is superimposed on the initial screenshot data according to the viewport parameter and the size parameter, and target screenshot data is generated.

[0088] It should be noted that the scale data described above can be graphical and textual information (such as a line segment of 250 μm + “250 μm” text) reflecting the correspondence between pixels in the screenshot and actual physical size (such as microns μm, millimeters mm). In the prior art, such data needs to be manually spliced to the screenshot slice by the user after the screenshot, which is low in efficiency and high in error rate. However, the embodiment can automatically generate and merge the scale data into the initial screenshot data according to the viewport parameter and the size parameter. After superimposing the scale data, the front-end device generates the final image data of the picture (such as PNG, JPEG format) containing the slice image, the annotation information and the dynamic scale, i.e., the target screenshot data, and the front-end device can take the slice name, the patient and other key information as the name of the target screenshot data, and write the key information into the picture metadata through EXIF. After superimposition is completed, further, the scale of the existing screenshot is fixed (such as only showing “100 μm”), which cannot be dynamically adjusted with the viewport scaling, resulting in that the user cannot accurately determine the actual size at different scaling levels, which affects the diagnosis accuracy.

[0089] In a feasible implementation, in the embodiment, step S30 can include steps B1-B4:

[0090] In step B1, a target scale length is generated according to a preset scale factor and the size parameter;

[0091] In step B2, an optimized scale length is generated based on the target scale length and the viewport parameter;

[0092] In step B3, an intelligent unit corresponding to the optimized scale length is obtained;

[0093] In step B4, scale data is drawn on the initial screenshot data based on the optimized scale length and the intelligent unit according to the analysis result of the superimposable area of the slice image, and target screenshot data is generated.

[0094] It is easy to understand that the above preset scale factor can be a coefficient (such as 0.15-0.20, which can be configured according to actual conditions, and the specific value is not limited in the embodiment) for calculating the scale length and the screenshot width ratio, so as to ensure that the scale length is moderate (neither occupies too much screenshot space, nor is not clear and visible). Then the front-end device can calculate the initial scale pixel length based on the screenshot width contained in the size parameter and the preset scale factor, that is, the above target scale length. Exemplarily, assuming that the preset scale factor is 0.17, if the width in the size parameter of the screenshot visible area is 1000px, then the target scale length = 1000px x 0.17 = 170px; meanwhile, the front-end device can further set that the length constraint range of the target scale length is between 30px (minimum) and 200px (maximum), and if the calculation result exceeds the range, the boundary value is taken; the width constraint range can be set to not more than 15% of the screenshot width.

[0095] It is easy to understand that the above optimized scale length can be the actual scale length (such as 42.5μm rounded to 40μm) obtained by converting the target scale length into a physical length after combining the pixel physical size in the viewport parameter, and then processing by an intelligent rounding strategy. Exemplarily, assuming that the pixel physical size contained in the viewport parameter is 0.25μm / px, and the target scale length is 170px, then the target physical length = 170px x 0.25μm / px = 42.5μm. Then, the front-end device can adopt a user-friendly intelligent rounding strategy (preferentially rounding to 1, 2, 2.5, 5 and times, for example, the calculation value is 237μm, which can be rounded to 250μm; the calculation value is 0.48mm, which can be rounded to 0.5mm; the calculation value is 1.23mm, which can be rounded to 1.2mm or 1.25mm (the specific strategy can be selected)), and the 42.5μm is optimized to 40μm, that is, the optimized scale length is 40μm.

[0096] It needs to be understood that the embodiment can automatically switch units (pm->nm) at an ultra-high zoom level, and set a switching threshold in combination with medical standards (such as ASCO / CAP guidelines) to facilitate quick reading and understanding of users. Accordingly, the above-mentioned intelligent unit can be a physical unit (pm or mm) automatically selected by the front-end device according to the optimized scale length, to ensure that the unit is adapted to the length size (such as pm for small length and mm for large length). In the embodiment, the threshold logic of unit switching can be: when switching to mm unit, keep 1 decimal place (avoid 0.25mm and other non-intuitive values). For example: when the scale calculation value is greater than or equal to 1000pm, convert to mm unit and round (such as 1500pm is optimized to 1.5mm); otherwise, use pm (such as 40pm). In addition, if the converted value is in (0.1, 1), keep 1 decimal place; if it is greater than or equal to 1, round to 0.5mm step.

[0097] It can be understood that the above-mentioned superimposable area analysis result can be an analysis result of the initial screenshot data, for example, the analysis result of the distribution density (or cancerous area aggregation degree) of pathological tissues in the slice, which determines the blank (i.e. no user annotation and AI annotation) or non-key area (such as an area without important tissues or low tissue density) that is suitable for placing a scale. In addition, in another implementation, the embodiment can also directly superimpose and draw scale graphics and unit text in the initial screenshot data according to the preset configuration items (such as top left, bottom left, top right, and bottom right) by the user.

[0098] At the same time, in the embodiment, the line thickness, color, text font / size / color of the scale graphics can be customized and configured by the user to ensure clear readability under different backgrounds (dark / light). For example, after the front-end device analyzes the initial screenshot data and determines the superimposable area (such as the lower right corner), the scale can be drawn through the Canvas API: first, draw a horizontal line segment (length corresponding to 40pm, line width 2px, color black); then, draw a vertical short scale (length 5px) at both ends of the line segment; finally, draw the text "40pm" above the line segment (font 12px, color black, background semi-transparent white color to ensure readability).

[0099] Therefore, in the embodiment, the front-end device can dynamically calculate the length and unit (such as μm or mm) of the scale bar based on the viewport parameters and the size parameters at the time of triggering the slice screenshot instruction, and draw the scale bar graphics and text at the specified position (such as the lower right corner) of the initial screenshot data through the Canvas API. For example, if the screenshot width is 1000px and the scale factor is 0.17, the scale bar length is 170px; combined with the pixel physical size of 0.25 μm / px, the physical length is calculated to be 42.5 μm, which is rounded to 40 μm, and finally the scale bar of “40 μm” is drawn. That is, the embodiment can break through the mechanical setting of the existing scale bar, dynamically generate a scale bar matching the current viewport, and ensure that the user can accurately determine the actual size of the slice; and by introducing a pathology prior knowledge base (such as tumor cell size distribution), combined with intelligent rounding and unit switching, the readability of the scale bar is improved, and the diagnostic error is reduced.

[0100] In summary, the embodiment can realize pure front-end real-time screenshot, without the participation of the back-end, fast response speed, meet the real-time interaction demand; and automatically synthesize the slice image, annotation information (user annotation, AI result) and dynamic scale bar, without manual splicing, reduce the error; while retaining the complete diagnostic context information, improve the user communication, sharing and archiving efficiency.

[0101] The embodiment provides a digital pathology slice intelligent screenshot method, which is applied to a front-end device and includes the following steps: in response to a slice screenshot instruction, capturing viewport parameters of a target slice image in a current viewport and size parameters of a screenshot visible area; obtaining slice image data from a view engine corresponding to a slice image layer and drawing the slice image data on a bottom layer of an off-screen composition canvas corresponding to the size parameters; cutting off-screen canvas content corresponding to a first annotation rendering layer according to the viewport parameters and the size parameters to obtain first rendering content; cutting off-screen canvas content corresponding to a second annotation rendering layer according to the viewport parameters and the size parameters to obtain second rendering content; converting the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule; drawing the first optimized annotations to a middle layer of the off-screen composition canvas and drawing the second optimized annotations to an upper layer of the off-screen composition canvas to generate initial screenshot data; generating a target scale bar length according to a preset scale factor and the size parameters; generating an optimized scale bar length based on the target scale bar length and the viewport parameters; obtaining an intelligent unit corresponding to the optimized scale bar length; and drawing scale bar data on the initial screenshot data based on the optimized scale bar length and the intelligent unit according to a slice image superimposable area analysis result to generate target screenshot data. The embodiment can realize pure front-end real-time screenshot, without the participation of the back-end, fast response speed, meet the real-time interaction demand; and automatically synthesize the slice image, annotation information (user annotation, AI result) and dynamic scale bar, without manual splicing, reduce the error; while retaining the complete diagnostic context information, improve the user communication, sharing and archiving efficiency.

[0102] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be described in detail.

[0103] It is easy to understand that in the prior art, the coordinate systems of user labeling (such as manual labeling by doctors) and AI analysis results and other multi-source labeling data are not unified, resulting in misplacement of labeling information and slice images when taking screenshots, and unable to accurately superimpose.

[0104] Therefore, on the basis of the first embodiment, please refer to Figure 3 , Figure 3 The first flowchart of the second embodiment of the digital pathology slice intelligent screenshot method of the present application, in this embodiment, before step S20, the digital pathology slice intelligent screenshot method further includes steps C1-C2:

[0105] Step C1, a unified labeling protocol is constructed to convert the graphic data of the first labeling source and the second labeling source into first protocol data and second protocol data with the same coordinate system;

[0106] It can be understood that the above-mentioned unified labeling protocol can be a pre-set unified protocol middleware, which is a unified format specification of multi-source labeling data, and ensures that labeling data from different sources can be processed under the same coordinate system. The first labeling source can be a user interaction generated labeling data source (such as user drawing labeling through Konva tool). The second labeling source can be a labeling data source generated by AI, large model or algorithm analysis (such as suspicious areas output by AI through slice analysis). The first protocol data and the second protocol data can be data obtained by converting the original data of the first labeling source and the second labeling source into the same labeling protocol respectively.

[0107] Therefore, this embodiment can pre-prepare a protocol specification containing key fields such as "coordinates, graphic types, attribute information, source identification, rendering priority", and convert the original data of the first labeling source (such as Konva graphic data) and the original data of the second labeling source (such as AI output analysis results) into first protocol data and second protocol data with the same format. For example, the user's labeling rectangle and the AI's recognized polygon are both converted into protocol data containing "coordinate array, type (rect / polygon), color, source (user / ai), priority (high / low)".

[0108] In a feasible implementation manner, please refer to Figure 4 , Figure 4 The second flowchart of the second embodiment of the digital pathology slice intelligent screenshot method of the present application, in this embodiment, step C1 can include steps C11-C13:

[0109] Step C11, receiving a first JSON object of the first annotation source output updated based on the user interaction event, the first JSON object carrying key fields including at least coordinates, graphic type and attribute information;

[0110] Step C12, receiving a second JSON object of the second annotation source output updated based on the slice tissue density of the target slice image, the second JSON object carrying the same key fields as the first JSON object;

[0111] Step C13, converting the first JSON object and the second JSON object into first protocol data and second protocol data respectively, both of which include source identification and rendering priority.

[0112] It can be understood that the user interaction event can refer to the operation of the user on the annotation tool, such as drawing, modifying and deleting annotations, driving Konva by the event, and performing dirty rectangle update. The JSON format data output by the first annotation source (user annotation) based on the user interaction event, i.e. the first JSON object described above, can include: coordinates (such as the coordinates of the upper left corner and the lower right corner of a rectangle; or the coordinates of the three vertices of a triangle), graphic type (such as rect (rectangle), circle (circle), arrow (arrow), polygon (polygon)), attribute information (such as color (Color), line width (LineWidth), text description (Description)) and the like.

[0113] Exemplarily, when the user draws a rectangular annotation through the annotation tool (such as the drawing component provided by Konva) of the front-end interface, the tool triggers an interaction event, and the first annotation source outputs a first JSON object in real time, for example:

[0114] {

[0115] "Coordinates": [{"x":100,"y":100},{"x":200,"y":200}];

[0116] "Type":"rect" / / rect with upper left corner coordinates (100, 100) and lower right corner coordinates (200, 200);

[0117] "Color":"#ff0000";

[0118] "LineWidth":2;

[0119] }

[0120] The front-end device receives the first JSON object and temporarily stores it.

[0121] It is easy to understand that the aforementioned tissue density can be the degree of cell or tissue density in the slice image (e.g., high cell density areas, low cell density areas). In this embodiment, the JSON format data (output by a lightweight AI model, such as ONNXRuntime front-end inference) from the second annotation source, i.e., the second JSON object, can be dynamically updated based on the tissue density of the slice. For example, areas with normal cell density are low-frequency (≤5fps), while areas with high cell density are increased to 10fps to improve the visual continuity of key diagnostic areas.

[0122] In addition, the second JSON object can contain the same key fields (coordinates, graphic type, attribute information) as the first JSON object, which facilitates subsequent data conversion within the same coordinate system.

[0123] For example, the AI ​​analysis engine detects the tissue density of the slide in real time, outputting more refined analysis results in high-cell-density areas (such as tumor areas), and generating a second JSON object, for example:

[0124] {

[0125] "Coordinates": [{"x":150,"y":150},{"x":180,"y":160},{"x":170,"y":190}];

[0126] "Type":"polygon";

[0127] "Color":"#00ff00";

[0128] "Description": "Suspicious cell cluster";

[0129] }

[0130] The front-end device receives the second JSON object and stores it temporarily.

[0131] It should be noted that, to facilitate sequential rendering, the front-end device needs to convert the temporarily stored first and second JSON objects into first and second protocol data, respectively, containing source identifiers and rendering priorities. The source identifier can be an identifier distinguishing the source of the annotation data; for example, "user" could represent the first annotation source, and "ai" could represent the second annotation source. The rendering priority (layerPriority) indicates the rendering order of the annotation data, such as "high" and "low," or "001" and "002," etc. In this embodiment, the first annotation source can be set to have a higher priority than the second annotation source to ensure that user annotations corresponding to the first annotation source are not obscured by AI annotations corresponding to the second annotation source.

[0132] Exemplarily, the front-end device can convert the first JSON object into first protocol data (adding source identification and rendering priority), for example:

[0133] {

[0134] "Coordinates":[{"x":100,"y":100},{"x":200,"y":200}];

[0135] "Type":"rect";

[0136] "Color":"#ff0000";

[0137] "LineWidth":2;

[0138] "Source":"user";

[0139] "layerPriority":"high";

[0140] }

[0141] Similarly, the second JSON object is converted into second protocol data (source identification is "ai", and priority is "low"), ensuring that the formats of the two are unified.

[0142] Therefore, in view of the problem that the original formats of multi-source annotation data are not unified (for example, user annotation output XML format and AI analysis output custom JSON format), it is difficult to directly convert into unified protocol data, affecting the efficiency of layer synthesis, the embodiment can unify the formats of multi-source annotation data, reduce the complexity of data processing; and through source identification and rendering priority, the rationality of annotation display is ensured.

[0143] It is easy to understand that the AI analysis data source can contain a large amount of data (such as all suspicious areas of the whole slice), and if all of them are directly rendered, it will cause the performance of the front end to decline (such as lag, delay), affecting the user experience. Therefore, in a feasible implementation manner, in the embodiment, the second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

[0144] It can be understood that the above-mentioned slice analysis data source can be a raw data set of the second annotation source (AI analysis), and can include a plurality of sub-data sources such as low-precision analysis data of the whole slice, high-precision analysis data of the local area, etc. Exemplarily, in the embodiment, the at least one slice analysis data source corresponding to the second annotation source includes but is not limited to: whole slice low-precision data, 20x zoom level high-precision data, high cell density area detail data, etc.

[0145] The actual graphics data source can be AI analysis data currently required to be rendered to the second annotation rendering layer. The front-end device can filter data matching the current viewport from at least one slice analysis data source according to viewport parameters (such as a zoom level, a viewport coordinate, etc.). For example, in the case where the viewport parameters of the current viewport are that the current zoom level is 20x and the viewport coordinate corresponds to a high cell density area, the front-end device can select the matching "20x zoom level high-precision data" and "high cell density area detail data" from multiple data sources as the actual graphics data source, render only the AI annotation information required by the current viewport, and reduce the data processing amount.

[0146] Therefore, the embodiment can dynamically select required AI analysis data, avoid redundant data rendering, improve front-end performance, and ensure accurate and efficient display of AI annotation information of the current viewport.

[0147] Step C2, mapping the first protocol data to the first annotation rendering layer and the second protocol data to the second annotation rendering layer based on the spatial coordinates of the view engine corresponding to the slice image layer, and synchronously updating the display states of the first annotation rendering layer and the second annotation rendering layer in the case where a viewport change event is listened to.

[0148] It can be understood that the view engine can be a basic engine (such as OpenSeaDragon) used by the front-end device to render a digital pathology slice, and the spatial coordinates can be a global coordinate system of the entire slice. For example, the front-end device can map the coordinates of the first protocol data to the local coordinate system of the first annotation rendering layer based on the spatial coordinates of the view engine, so as to drive the first annotation rendering layer Konva to perform dirty rectangle updating (Dirty Rectangle) by a user interaction event, convert the first protocol data into an interactive graphic or text through a Konva API, and map the coordinates of the second protocol data to the local coordinate system of the second annotation rendering layer, perform graphic rendering through a PixiAPI of the second annotation rendering layer PixiJS, and ensure that the annotation data corresponds to the position of the slice image.

[0149] In a feasible implementation, the view engine registers a viewport change event listener to broadcast view parameters including a zoom coefficient, a rotation angle, and a view coordinate to the first annotation rendering layer and the second annotation rendering layer in the case where the target slice image is zoomed, translated, or rotated. In the embodiment, step C2 can include steps C21-C22:

[0150] Step C21, adjusting the geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameters in the case where the viewport change event is listened to.

[0151] Step C22, calling the drawing interface of the rendering engine corresponding to the first and second annotation rendering layers according to the adjusted geometric transformation matrix to redraw the display state of the first and second annotation rendering layers.

[0152] It can be understood that the above-mentioned viewport change event can be an event detected by the front-end device due to user operation (such as zooming, panning, and rotating) causing the current viewport state of the target slice image to change. The viewport change event listener can be a program module in the view engine for monitoring the change of the viewport state, such as the callback function registered by the "addHandler" interface of OpenSeaDragon. The front-end device can register a viewport change event listener (such as the "viewport-change" event of OpenSeaDragon) for the view engine to listen to the viewport change event in real time. The rendering frequency is real-time (such as 60fps), which can be achieved by OpenSeaDrag dynamic tile loading (LOD). When the viewport changes (such as zooming, panning, and rotating), if the annotation layer (the first and second annotation rendering layers) cannot be updated synchronously, it will cause the annotation information to be misaligned with the slice image, affecting the accuracy of the screenshot.

[0153] It can be understood that the above-mentioned view parameters can be a set of zoom coefficients, rotation angles, and view coordinates (i.e. the pixel position of the upper left corner of the changed image in the entire slice coordinate system), which are used to describe the state of the changed viewport. Therefore, in the case of a change in the viewport of the target slice image, the front-end device can broadcast the view parameters to the first and second annotation rendering layers through a custom event bus (such as EventEmitter).

[0154] For example, the front-end device can register a listener in advance in the view engine (such as OpenSeaDragon) through "viewer.addHandler("viewport-change",callback)". When the user zooms (such as a scroll wheel operation), pans (such as a mouse drag), or rotates the slice, the listener is triggered and the view parameters (such as a zoom coefficient of 5x, a rotation angle of 30°, and a view coordinate of {x:1000,y:500}) are calculated, and then broadcast to the first and second annotation rendering layers through a custom event bus. The parameter transmission can be as follows:

[0155] {

[0156] zoom:viewer.viewport.getZoom(), / / current zoom factor (such as 5x);

[0157] rotation:viewer.viewport.getRotation(), / / rotation angle (degrees);

[0158] position: viewer.viewport.pixelFromPoint(newPoint(0, 0)), / / view coordinate

[0159] }

[0160] It should be noted that the above geometric transformation matrix can be a mathematical matrix (such as a 3xn matrix, containing scaling factor, rotation angle, translation amount parameter (containing n coordinates)) used to describe scaling, rotation, translation and other transformations of the first annotation rendering layer and the second annotation rendering layer. The drawing interface can be a method provided by the rendering engine for redrawing the layer (such as Konva's "draw()", PixiJS's "render()").

[0161] Therefore, in the embodiment, after the first annotation rendering layer and the second annotation rendering layer receive the parameters, the scaling, rotation, and position of the first annotation rendering layer and the second annotation rendering layer can be adjusted synchronously to ensure that the annotation information is always aligned with the slice image. For example, the Konva layer can be adjusted by the "scale()" and "rotation()" methods, and the PixiJS layer can be adjusted by the "stage.scale" and "stage.rotation".

[0162] Subsequently, the first annotation rendering layer calls the "Konva.layer.draw()" interface, and the second annotation rendering layer calls the "Pixijs.app.render()" interface to redraw the layer based on the adjusted matrix, so as to ensure that the annotation information is completely matched with the position, angle, and scaling ratio of the slice image.

[0163] Exemplarily, after the first annotation rendering layer (Konva) and the second annotation rendering layer (PixiJS) receive the view parameters, the respective geometric transformation matrices are adjusted according to the parameters. The geometric transformation matrix can include a scaling factor (zoom), a rotation angle (rotation), and a view coordinate (position).

[0164] Further, the dynamic redrawing of the first annotation rendering layer (Konva) can be as follows:

[0165] Synchronous scaling by Konva.layer.scale({x: zoom, y: zoom});

[0166] Synchronous offset by Konva.layer.position({x: position.x, y: position.y});

[0167] Synchronous rotation by Konva.layer.rotation = rotation;

[0168] Re-draw rendering by Konva.layer.draw().

[0169] Dynamic re-drawing of the second annotation rendering layer (PixiJS) can be as follows:

[0170] Synchronous scaling by Pixijs.app.stage.scale(zoom);

[0171] Synchronous offset by Pixijs.app.stage.position.set(position.x, position.y);

[0172] Synchronous rotation by Pixijs.app.stage.rotation = rotation * (Math.PI / 180);

[0173] Re-draw rendering by Pixijs.app.render().

[0174] Further, to solve the coordinate system floating-point number error of OpenSeaDragon and Konva, the front-end device can add a coordinate correction function in the re-drawing process.

[0175] In this embodiment, real-time synchronization of the annotation layer when the viewport changes can be achieved, ensuring that the annotation information is always aligned with the slice image; the transformation parameters are accurately calculated through the geometric transformation matrix, improving the accuracy of annotation display.

[0176] In summary, the present embodiment can solve the coordinate system difference problem of multi-source annotation data, ensure accurate alignment of annotation information and slice images; and achieve real-time synchronization of the annotation layer when the viewport changes, ensuring the coherence of annotation display during user operation.

[0177] The embodiment discloses receiving a first JSON object of a first labeling source output updated based on a user interaction event, the first JSON object carrying key fields including at least coordinates, a graph type and attribute information; receiving a second JSON object of a second labeling source output updated based on a slice image of a target slice; the second JSON object carries the same key fields as the first JSON object; the second labeling source has at least one slice analysis data source, and an actual graph data source of the second labeling source is selected from the at least one slice analysis data source according to a viewport parameter. The first JSON object and the second JSON object are respectively converted into first protocol data and second protocol data each containing a source identifier and a rendering priority, so as to convert graph data of the first labeling source and the second labeling source into the first protocol data and the second protocol data with the same coordinate system; the first protocol data is mapped to a first labeling rendering layer, and the second protocol data is mapped to a second labeling rendering layer based on a space coordinate of a view engine corresponding to a slice image layer; in the case that a viewport change event is listened to, a geometric transformation matrix corresponding to the first labeling rendering layer and the second labeling rendering layer is adjusted according to the view parameter; the drawing interface of a rendering engine corresponding to the first labeling rendering layer and the second labeling rendering layer is called according to the adjusted geometric transformation matrix, so as to redraw the display state of the first labeling rendering layer and the second labeling rendering layer. The embodiment can solve the coordinate system difference problem of multi-source labeling data, ensure accurate alignment of labeling information and slice images, and realize real-time synchronization of labeling layers when the viewport changes, thereby ensuring the continuity of labeling display in the user operation process.

[0178] By way of example, in order to facilitate understanding of the technical concept or technical principle of the digital pathology slice intelligent screenshot method combined with the above embodiment one and embodiment two, please refer to Figure 5 , Figure 5 The following is a brief flowchart of the digital pathology slice intelligent screenshot method of the present application:

[0179] The present application provides a specific method for realizing digital pathology slice intelligent screenshot in the Web browser environment of a front-end device. The method aims to solve the problem that the traditional screenshot method cannot preserve key information such as dynamic scale, user labeling layer and AI analysis result, ensure that the shared pathology screenshot contains complete diagnostic context information, and has real-time and smooth interaction. The method is purely front-end implemented, without the need for backend participation in rendering, which can significantly reduce response delay. As shown in Figure 5 , the implementation process of the present application can be as follows:

[0180] Step 1, initialize view and labeling layer:

[0181] Load target slice image (e.g. DICOM or SDPC format) in front-end device (e.g. browser of user workstation computer). Use OpenSeaDragon (OSD) as base view engine to render slice and initialize two independent annotation canvas layers:

[0182] First annotation rendering layer, i.e. user annotation layer: render using Konva library, support user interactive annotation (rectangle, circle, arrow, text).

[0183] Second annotation rendering layer, i.e. AI / model analysis layer: render using PixiJS library, used to display suspicious areas identified by AI algorithm (e.g. polygon, dot).

[0184] Register viewport change event listener for OSD view's viewport-change event (zoom, pan, rotate). When viewport changes, event handler gets view parameters: zoom (zoom factor, e.g. 20x), rotation (rotation angle), position (pixel coordinates of top-left corner of viewable area in whole image coordinate system). And broadcast view parameters through custom event bus (e.g. EventEmitter).

[0185] Perform coordinate dynamic mapping:

[0186] First annotation rendering layer (Konva): receive viewport parameters, adjust geometric transformation matrix, set KonvaLayer's size to OSD container's clientWidth and clientHeight. Call konvaLayer.scale({x:zoom,y:zoom}), konvaLayer.position({x:position.x,y:position.y}), konvaLayer.rotation(rotation), and finally call konvaLayer.draw() to redraw. Ensure user annotation graphics keep consistent with slice view.

[0187] Second annotation rendering layer (PixiJS): receive viewport parameters, adjust geometric transformation matrix, and call pixiApp.stage.scale.set(zoom,zoom), pixiApp.stage.position.set(position.x,position.y), pixiApp.stage.rotation=rotation*(Math.PI / 180), and finally call pixiApp.render() to redraw. Ensure AI annotation graphics move, zoom, and rotate synchronously with slice.

[0188] Step 2, Trigger screenshot and state capture:

[0189] The user can click the "screenshot" button on the web interface of the front-end device. The system immediately suspends all interactive events of the OSD (zoom, pan, rotate). Freeze the graphical state of the current AI layer (PixiJS) and user annotation layer (Konva) to prevent annotations from being modified during the screenshot process.

[0190] Capture key parameters, namely the viewport parameters of the current viewport and the size parameters of the screenshot visible area:

[0191] 1) Viewport coordinates: Call OSD API "viewer.viewport.pixelFromPoint(newPoint(0,0))" to get the pixel coordinates of the top-left corner of the current visible area in the entire slice (for example, {x:1500,y:800}).

[0192] 2) Zoom level: Call OSD API (viewer.viewport.getZoom()) to get the current magnification (for example, 20x).

[0193] 3) Pixel physical size: Read from slice metadata (for example, 0.25μm / pixel).

[0194] 4) Size parameters: Get the clientWidth and clientHeight of the OSD container element as the width and height (px) of the screenshot canvas.

[0195] Step 3, Multi-canvas layer composition:

[0196] Create an off-screen Canvas element (i.e. off-screen composition canvas) with size parameters corresponding to the size of the screenshot area captured in step 2. Then compose the layers in order from bottom to top:

[0197] 1) Bottom layer - slice image layer: Directly get the slice image rendered by the current view from the OSD's DrawerCanvas (viewer.drawer.canvas). Draw it to the corresponding position of the composition canvas (usually coordinates (0,0)).

[0198] 2) Middle layer - AI annotation layer: Get the current frozen PixiJS canvas (pixiApp.renderer.view). According to the viewport coordinates and screenshot size captured in step 2, use Canvas API (ctx.drawImage()) to draw the corresponding visible area in the PixiJS canvas to the composition canvas. Preserve the original style of AI annotations (color, line width).

[0199] 3) Upper layer - user annotation layer: Get the current frozen KonvaStageCanvas (konvaStage.toCanvas()). According to the parameters captured in step 2, use the Canvas API to draw the visible area of the Konva canvas to the top layer of the composite canvas. Preserve the original style of user annotations (color, line width, font). The user annotation layer is overlaid on the AI annotation layer by default.

[0200] In this process, redundant annotation graphics beyond the current viewport in the AI annotation layer or the user annotation layer can be filtered through preset canvas optimization rules, and the clarity of overlapping annotation graphics can be adjusted to reduce data volume and improve drawing efficiency; or the completeness of annotation graphics beyond the current viewport can be guaranteed, and the clarity of overlapping annotation graphics can be adjusted to guarantee image integrity and improve screenshot accuracy. Specific requirements can be set according to actual conditions.

[0201] Step 4, dynamic scale intelligent calculation and drawing:

[0202] Overlay and draw dynamic scale on composite canvas:

[0203] 1) Target length calculation: scale length = screenshot width * scale factor (configuration item, default 0.17, range 0.15-0.20). The result is constrained between the minimum length (30px) and the maximum length (200px). Example: screenshot width 1000px, target length = 1000*0.17 = 170px.

[0204] 2) Actual physical length calculation: physical length = target length (px) * pixel physical size (μm / px). Example: 170px * 0.25μm / px = 42.5μm.

[0205] 3) Intelligent rounding: round the physical length for user-friendly rounding. Preferentially round to 1, 2, 2.5, 5, etc. or their 10^n times. Example: 42.5μm rounded to 40μm; 237μm rounded to 250μm; 1.23mm rounded to 1.2mm or 1.25mm (optional strategy).

[0206] 4) Intelligent unit switching: if the rounded physical length >= 1000μm (1mm), automatically switch the unit to millimeters (mm) and display the text as 1.5mm; otherwise, use microns (μm) and display the text as 250μm. Avoid values that are not easy to estimate (such as 233μm).

[0207] 5) Scale drawing: using Canvas API to draw a horizontal line segment (thickness, color configurable) at the specified position (configurable: top-left, bottom-left, top-right, bottom-right, default bottom-right) of the composite canvas or the overlayable area analysis result of the slice image, and draw vertical short scale lines at both ends of the line segment. Then draw a text label (content: physical length after rounding + unit, such as 40 μm; font, size, color configurable, ensure clear contrast with background) above or below the scale. In addition, according to user requirements, draw the zoom level text (such as 20x) corresponding to the current viewport near the scale data.

[0208] Step 5, output integrated picture:

[0209] Call the toDataURL() or toBlob() method of the composite canvas to convert the final synthesized image data into a picture (such as PNG, JPEG format). Further, a picture file name can be generated for easy user identification or retrieval: it can contain slice name, patient ID, timestamp, etc. (for example, Patient123_LiverBiopsy_20230806_1600.png). Optionally, use EXIF or other libraries to write key information (such as zoom level 20x, pixel size 0.25 μm / px, screenshot time) into the metadata of the picture file.

[0210] Step 6, restore interaction:

[0211] After completing the screenshot, re-enable the interactive events of the OSD, unfreeze the AI layer and user annotation layer, and the user can continue to operate.

[0212] Therefore, the present application can realize pure front-end digital pathology slice intelligent screenshot, ensuring that the output picture contains complete diagnostic information (original slice, dynamic scale, user annotation, AI result), eliminating the tediousness and errors of manual splicing. Through precise coordinate mapping and layer synthesis mechanism, the position accuracy of the annotation information in the screenshot is ensured. The intelligent calculation and drawing of the dynamic scale provide accurate and easy-to-read scale reference. The whole process responds quickly, the user experience is smooth, and it has good browser compatibility, significantly improving the efficiency and accuracy of user sharing, communication and archiving of diagnostic information.

[0213] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the digital pathology slice intelligent screenshot method of the present application. Based on this technical concept, more forms of simple transformation are within the protection scope of the present application.

[0214] The present application also provides a digital pathology slice intelligent screenshot device, please refer to Figure 6 , Figure 6As a module structure schematic diagram of the digital pathology slice intelligent screenshot device of the embodiment of the present application, in the embodiment, the digital pathology slice intelligent screenshot device comprises:

[0215] The parameter acquisition module 601 is configured to capture the viewport parameter of the target slice image in the current viewport and the size parameter of the screenshot visible area in response to a slice screenshot instruction.

[0216] The preliminary rendering module 602 is configured to draw the slice image layer, the first annotation rendering layer and the second annotation rendering layer in the current viewport in sequence to an off-screen composition canvas corresponding to the size parameter according to the viewport parameter, to generate initial screenshot data.

[0217] The target rendering module 603 is configured to superimpose scale data on the initial screenshot data according to the viewport parameter and the size parameter, to generate target screenshot data.

[0218] As an implementable manner, in the embodiment, the preliminary rendering module 602 is further configured to construct a unified annotation protocol, to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data having the same coordinate system; to map the first protocol data to the first annotation rendering layer and the second protocol data to the second annotation rendering layer based on the spatial coordinates of the view engine corresponding to the slice image layer, and to synchronously update the display states of the first annotation rendering layer and the second annotation rendering layer in the case of listening to a viewport change event.

[0219] As an implementable manner, in the embodiment, the preliminary rendering module is further configured to receive a first JSON object output by the first annotation source based on a user interaction event update, the first JSON object carrying key fields including at least coordinates, graphic types and attribute information; to receive a second JSON object output by the second annotation source based on a slice tissue density update of the target slice image, the second JSON object carrying the same key fields as the first JSON object; and to convert the first JSON object and the second JSON object into first protocol data and second protocol data respectively, both of which include source identification and rendering priority.

[0220] As an implementable manner, in the embodiment, the second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameter.

[0221] As an implementable manner, the view engine registers a viewport change event listener to broadcast view parameters including a scaling factor, a rotation angle and a view coordinate to the first annotation rendering layer and the second annotation rendering layer in the case that the target slice image is scaled, translated or rotated; in the embodiment, the preliminary rendering module is further configured to adjust a geometric transformation matrix corresponding to the first annotation rendering layer and the second annotation rendering layer according to the view parameters in the case that the viewport change event is listened to; and the preliminary rendering module is further configured to call a drawing interface of a rendering engine corresponding to the first annotation rendering layer and the second annotation rendering layer according to the adjusted geometric transformation matrix, so as to redraw the display state of the first annotation rendering layer and the second annotation rendering layer.

[0222] As an implementable manner, in the embodiment, the preliminary rendering module is further configured to obtain slice image data from a view engine corresponding to the slice image layer and draw the slice image data on a bottom layer of an off-screen composite canvas corresponding to the size parameter; the preliminary rendering module is further configured to crop off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameter and the size parameter to obtain first rendering content; the preliminary rendering module is further configured to crop off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameter and the size parameter to obtain second rendering content; the preliminary rendering module is further configured to convert the first rendering content and the second rendering content into first optimized annotations and second optimized annotations according to a preset canvas optimization rule; the preliminary rendering module is further configured to draw the first optimized annotations to a middle layer of the off-screen composite canvas and draw the second optimized annotations to an upper layer of the off-screen composite canvas to generate initial screenshot data.

[0223] As an implementable manner, in the embodiment, the target rendering module is further configured to generate a target scale length according to a preset scale factor and the size parameter; the target rendering module is further configured to generate an optimized scale length based on the target scale length and the viewport parameter; the target rendering module is further configured to obtain an intelligent unit corresponding to the optimized scale length; and the target rendering module is further configured to draw scale data on the initial screenshot data based on the optimized scale length and the intelligent unit according to a superimposable area analysis result of the slice image to generate target screenshot data.

[0224] The digital pathology slice intelligent screenshot device provided in the application adopts the intelligent screenshot method in the above embodiments, and can solve the technical problems in the prior art that the digital pathology slice screenshot cannot save dynamic information, needs manual splicing and has poor real-time performance, and cannot meet the real-time screenshot interaction requirements of the digital pathology slice. Compared with the prior art, the device provided in the application realizes millisecond-level response of complete information screenshot by implementing multi-source annotation protocol construction, dynamic coordinate mapping and intelligent scale generation through a pure front end, eliminates human error and improves diagnosis efficiency, and meets the real-time screenshot interaction requirements of the digital pathology slice. Other technical features in the device are the same as the features disclosed in the above method embodiments, and will not be described here.

[0225] The application provides a digital pathological section intelligent screenshot device, which comprises at least one processor and a memory in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the digital pathological section intelligent screenshot method in the above embodiment I.

[0226] Reference will be made to the following description Figure 7 which shows a structural schematic diagram of the digital pathological section intelligent screenshot device suitable for being used to implement the embodiments of the application. The digital pathological section intelligent screenshot device in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones loaded with browsers, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 7 The shown digital pathological section intelligent screenshot device is only an example, and should not bring any limitation to the functions and use ranges of the embodiments of the application.

[0227] As Figure 7As shown, the digital pathology slide intelligent cropping device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the operation of the digital pathology slide intelligent cropping device are also stored in the random access memory 1004. The processing device 1001, the read only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the digital pathology slide intelligent cropping device to communicate wirelessly or wired with other devices to exchange data. Although the digital pathology slide intelligent cropping device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0228] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a digital pathology slide intelligent cropping program product including a digital pathology slide intelligent cropping program carried on a computer readable medium, the digital pathology slide intelligent cropping program containing program codes for performing the methods shown in the flowcharts. In such embodiments, the digital pathology slide intelligent cropping program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the read only memory 1002. When the digital pathology slide intelligent cropping program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0229] The digital pathology section intelligent screenshot device provided by the present application adopts the digital pathology section intelligent screenshot method in the above embodiment, and can solve the technical problem of how to meet the real-time screenshot interaction requirement of the digital pathology section. Compared with the prior art, the digital pathology section intelligent screenshot device provided by the present application has the same beneficial effects as the digital pathology section intelligent screenshot method provided by the above embodiment, and other technical features in the digital pathology section intelligent screenshot device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0230] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0231] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0232] The present application provides a storage medium having computer readable program instructions (i.e. digital pathology section intelligent screenshot program) stored thereon, the computer readable program instructions being used to execute the digital pathology section intelligent screenshot method in the above embodiment.

[0233] The storage medium provided by the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of storage media can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the storage medium can be transmitted by any suitable medium, including but not limited to: electrical wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0234] The storage medium can be included in the digital pathology slice intelligent screenshot device, or can exist separately and not be assembled into the digital pathology slice intelligent screenshot device.

[0235] The storage medium carries one or more programs, which, when executed by the digital pathology slice intelligent screenshot device, cause the digital pathology slice intelligent screenshot device to: digitally take a pathology slice intelligent screenshot.

[0236] The digital pathology slice intelligent screenshot program code for performing the operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as C or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0237] The flow diagrams and block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and digital pathology slice intelligent screenshot program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0238] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0239] The readable storage medium provided by the application is a storage medium, and the storage medium stores computer readable program instructions (i.e., a digital pathology section intelligent screenshot program) for executing the above-mentioned digital pathology section intelligent screenshot method, which can solve the technical problem of how to meet the real-time screenshot interaction requirement of the digital pathology section. Compared with the prior art, the beneficial effects of the storage medium provided by the application are the same as those of the digital pathology section intelligent screenshot method provided by the above-mentioned embodiments, and are not described here.

[0240] The above is only some embodiments of the application, and does not limit the scope of the application scheme, and any equivalent structural transformation made by using the content of the application specification and drawings, or direct / indirect application in other related technical fields is included in the protection scope of the application.

Claims

1. A method for intelligent screenshotting of digital pathological slides, characterized in that, The method is applied to a front-end device, and the method includes: In response to a slice capture command, capture the viewport parameters and size parameters of the target slice image in the current viewport of the visible area of ​​the slice. Based on the viewport parameters, the slice image layer, the first annotation rendering layer and the second annotation rendering layer locked in the current viewport are sequentially drawn to the off-screen composite canvas corresponding to the size parameters to generate initial screenshot data; Based on the viewport parameters and the size parameters, scale data is superimposed on the initial screenshot data to generate target screenshot data; Before generating the initial screenshot data, the process of sequentially drawing the slice image layer, the first annotation rendering layer, and the second annotation rendering layer locked at the current viewport onto the off-screen composite canvas corresponding to the size parameters, based on the viewport parameters, includes: A unified annotation protocol is constructed to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data with the same coordinate system; Based on the spatial coordinates of the view engine corresponding to the sliced ​​image layer, the first protocol data is mapped to the first annotation rendering layer, and the second protocol data is mapped to the second annotation rendering layer. When a viewport change event is detected, the display status of the first annotation rendering layer and the second annotation rendering layer are updated synchronously. The step of sequentially drawing the slice image layer, the first annotation rendering layer, and the second annotation rendering layer locked at the current viewport to the off-screen composite canvas corresponding to the size parameters, based on the viewport parameters, to generate initial screenshot data includes: The slice image data is obtained from the view engine corresponding to the slice image layer and drawn on the bottom layer of the off-screen composite canvas corresponding to the size parameter; The off-screen canvas content corresponding to the first annotation rendering layer is cropped according to the viewport parameters and the size parameters to obtain the first rendering content; The off-screen canvas content corresponding to the second annotation rendering layer is cropped according to the viewport parameters and the size parameters to obtain the second rendering content; According to the preset canvas optimization rules, the first rendered content and the second rendered content are converted into the first optimized annotation and the second optimized annotation; The first optimized annotation is drawn to the middle layer of the off-screen composite canvas, and the second optimized annotation is drawn to the upper layer of the off-screen composite canvas to generate initial screenshot data.

2. The intelligent screenshot method for digital pathological slides as described in claim 1, characterized in that, The steps for constructing the unified annotation protocol include: Receive a first JSON object output from a first annotation source that is updated based on user interaction events. The first JSON object carries key fields that include at least coordinates, graphic type, and attribute information. Receive a second JSON object output by a second annotation source based on the slice tissue density update of the target slice image, wherein the second JSON object carries the same key fields as the first JSON object; The first JSON object and the second JSON object are respectively converted into first protocol data and second protocol data, both of which contain source identifier and rendering priority.

3. The intelligent screenshot method for digital pathological slides as described in claim 2, characterized in that, The second annotation source has at least one slice analysis data source, and the actual graphic data source of the second annotation source is selected from the at least one slice analysis data source according to the viewport parameters.

4. The intelligent screenshot method for digital pathological slides as described in claim 2, characterized in that, The view engine is registered with a viewport change event listener to broadcast view parameters, including scaling factor, rotation angle and view coordinates, to the first annotation rendering layer and the second annotation rendering layer when the target slice image is scaled, translated or rotated. The step of synchronously updating the display states of the first annotation rendering layer and the second annotation rendering layer upon detecting a viewport change event includes: Upon detecting a viewport change event, adjust the geometric transformation matrices corresponding to the first and second annotation rendering layers based on the view parameters. The drawing interfaces of the rendering engines corresponding to the first and second annotation rendering layers are called according to the adjusted geometric transformation matrix to redraw the display state of the first and second annotation rendering layers.

5. The intelligent screenshot method for digital pathological slides as described in claim 1, characterized in that, The step of generating target screenshot data by overlaying scale data onto the initial screenshot data based on the viewport parameters and the size parameters includes: Generate the target scale length based on the preset scale factor and the size parameters; An optimized scale length is generated based on the target scale length and the viewport parameters; Obtain the intelligent unit corresponding to the length of the optimized scale; Based on the analysis results of the overlayable areas of the sliced ​​images, target screenshot data is generated by drawing scale data on the initial screenshot data based on the optimized scale length and the intelligent unit.

6. A digital pathological slide intelligent screenshot device, characterized in that, The digital pathology slide intelligent screenshot device includes: The parameter acquisition module is used to capture the viewport parameters of the target slice image in the current viewport and the size parameters of the visible area of ​​the screenshot in response to the slice screenshot command; The initial rendering module, used for the viewport parameters, sequentially draws the slice image layer, the first annotation rendering layer, and the second annotation rendering layer locked in the current viewport to the off-screen composite canvas corresponding to the size parameters, generating initial screenshot data; The target rendering module is used to generate target screenshot data by overlaying scale data onto the initial screenshot data according to the viewport parameters and the size parameters; The preliminary rendering module is also used to construct a unified annotation protocol to convert the graphic data of the first annotation source and the second annotation source into first protocol data and second protocol data with the same coordinate system; based on the spatial coordinates of the view engine corresponding to the slice image layer, the first protocol data is mapped to the first annotation rendering layer, the second protocol data is mapped to the second annotation rendering layer, and the display status of the first annotation rendering layer and the second annotation rendering layer is updated synchronously when a viewport change event is detected. The preliminary rendering module is further configured to obtain slice image data from the view engine corresponding to the slice image layer and draw it on the bottom layer of the off-screen composite canvas corresponding to the size parameters; crop the off-screen canvas content corresponding to the first annotation rendering layer according to the viewport parameters and the size parameters to obtain the first rendering content; crop the off-screen canvas content corresponding to the second annotation rendering layer according to the viewport parameters and the size parameters to obtain the second rendering content; convert the first rendering content and the second rendering content into the first optimized annotation and the second optimized annotation according to the preset canvas optimization rules; draw the first optimized annotation to the middle layer of the off-screen composite canvas and draw the second optimized annotation to the upper layer of the off-screen composite canvas to generate initial screenshot data.

7. A digital pathological slide intelligent screenshot device, characterized in that, The device includes: a memory, a processor, and a digital pathology slide intelligent screenshot program stored in the memory and executable on the processor, the digital pathology slide intelligent screenshot program being configured to implement the steps of the digital pathology slide intelligent screenshot method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a digital pathology slide intelligent screenshot program, which, when executed by a processor, implements the steps of the digital pathology slide intelligent screenshot method as described in any one of claims 1 to 5.

Images