A method, system, device and storage medium for fast acquisition of virtual IHC staining images in breast surgery

Abstract

The application relates to the technical field of digital pathology, and particularly provides a method, system and device for quickly obtaining a virtual IHC staining image during breast surgery and a storage medium, the method comprising the following steps: obtaining a breast lesion tissue sample of a patient, preparing an HE staining section, inputting a pre-trained virtual immunohistochemical staining generation model after scanning, and obtaining a virtual IHC image. According to the method, only the HE staining digital whole section image needs to be input to the pre-trained virtual immunohistochemical staining generation model, and the virtual IHC image equivalent to the traditional intraoperative IHC staining can be output within several minutes, so that the time limit requirement of 30 minutes for providing a diagnosis report in intraoperative rapid pathology can be met; the method does not need to perform the experimental operation of traditional intraoperative IHC staining, does not need to consume IHC staining related reagents, simultaneously reduces the artificial operation cost, and effectively reduces the diagnosis and treatment economic burden of the patient.

Description

Technical Field

[0001] This application relates to the field of digital pathology technology, specifically to a method, system, device, and storage medium for rapidly acquiring virtual IHC staining images during breast surgery. Background Technology

[0002] Pathological tissue sections are a method used in medical diagnosis that involves cutting tissue samples into thin sections and then staining them to reveal specific parts of the cells and tissue structure.

[0003] Hematoxylin-eosin (HE) staining is a staining method used in histology. It uses hematoxylin dye to stain the basophilic structures of cells and tissues blue-purple, and eosin dye to stain the eosinophilic structures pink. HE staining helps pathologists assess the overall structure and morphology of tissues, including cell morphology, size, arrangement, and structural features, thereby diagnosing diseases and assessing the health status of tissues.

[0004] Immunohistochemical staining (IHC), also known as immunohistochemistry, is an immunostaining technique that utilizes specific antibodies to bind to antigens in tissue sections or smears, selectively detecting the presence of target antigens. This staining method can determine the expression level and location of antigens, improving the sensitivity and specificity of pathological diagnosis. Immunohistochemical staining includes steps such as antigen retrieval, background blocking, primary antibody application, secondary antibody application, staining, and mounting. The entire process typically takes several hours to more than ten hours.

[0005] HE-stained frozen sections can provide rapid histological diagnostic results during surgery, but HE staining can only provide basic information about tissue morphology, cell structure and tissue distribution. In some cases, it is necessary to observe IHC-stained sections to further confirm or rule out possible lesion types.

[0006] However, existing intraoperative IHC staining techniques have the following drawbacks: 1) Intraoperative IHC staining is time-consuming, which cannot meet the requirement of issuing a diagnostic report within 30 minutes for intraoperative rapid pathology, making it difficult or impossible to obtain IHC staining results during surgery. The timeliness of intraoperative rapid pathology diagnosis is one of the thirteen quality control indicators for pathology as defined by the pathology quality control center; 2) Intraoperative IHC staining reagents and labor costs are high, which increases the economic burden of diagnosis and treatment for patients; 3) The number of technical personnel in the hospital's pathology department is limited, and intraoperative IHC staining requires additional workload for technical personnel, which significantly increases human and time costs.

[0007] Therefore, how to design a method, system, device, and storage medium for rapidly acquiring virtual IHC staining images during breast surgery that can shorten the intraoperative IHC staining time and reduce the detection cost is a technical problem that has not yet been solved in the existing technology. Summary of the Invention

[0008] Therefore, this application proposes a method, system, device, and storage medium for rapidly acquiring virtual IHC staining images during breast surgery, which can shorten the intraoperative IHC staining time and reduce the detection cost, in order to improve or solve at least one of the above-mentioned technical problems.

[0009] This application mainly includes the following aspects: In a first aspect, embodiments of this application provide a method for rapidly acquiring virtual IHC staining images during breast surgery, the method comprising: Obtain breast lesion tissue specimens from the patient; The breast lesion tissue specimen was subjected to frozen section preparation and modified intraoperative rapid HE staining to obtain HE-stained sections; The HE-stained section was scanned in its entirety to obtain a digital full-section image of HE staining corresponding to the HE-stained section. The HE-stained digital whole-slice image is input into a pre-trained virtual immunohistochemical staining generation model; the virtual immunohistochemical staining generation model includes a generator G, which includes an SGMS geometric registration subnetwork and a CAR virtual staining synthesis subnetwork. The SGMS geometric registration subnetwork is used to extract multi-level structural features of breast pathology and perform spatial geometric registration on the HE-stained digital whole-section images, outputting a registration-optimized HE feature sequence. ; The CAR virtual staining synthesis subnetwork is registered with the optimized HE feature sequence. For time-matched dynamic structural conditions, perform conditional modified stream synthesis processing to output a virtual IHC image. .

[0010] According to one embodiment of the present invention, the steps of the modified intraoperative rapid HE staining include: The slices prepared by the frozen sectioning were rinsed with distilled water for 20 seconds. The cell nuclei of the slide were stained with hematoxylin solution for 20 seconds. Rinse the slices with running water for 20 seconds to remove residual hematoxylin solution; The slides were soaked in the blueing solution for 10 seconds to complete the blueing of the cell nuclei; Rinse the slices again with running water for 20 seconds to remove any residual blueing solution. The slide was stained with 0.5% eosin solution for 20 seconds; The slices were soaked in 95% ethanol for 10 seconds to complete the initial dehydration. The sections were soaked in anhydrous ethanol for 10 seconds to complete the secondary dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the three-stage dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the fourth stage of deep dehydration. The slices were soaked in xylene for 10 seconds to achieve initial transparency. The slices were soaked in xylene for 10 seconds to achieve secondary transparency. The slices were soaked in xylene for 10 seconds to achieve three levels of deep transparency. The HE-stained sections are obtained by mounting with distilled water. This step includes: dropping an appropriate amount of distilled water onto the tissue section that has been cleared to a third depth; covering the entire tissue with a coverslip along one side of the tissue to avoid air bubbles; absorbing excess water with absorbent paper; and completing the mounting.

[0011] According to one embodiment of the present invention, the training of the virtual immunohistochemical staining generation model includes the following steps: Several pathological specimens from breast surgeries were collected, and the modified intraoperative rapid HE staining and IHC staining were performed on the same tissue section. After scanning, paired HE images and IHC images to be processed were obtained. The core region of interest (ROI) of the pathological tumor in the HE image and the IHC image to be processed is manually extracted and divided into training, validation, and test sets according to a preset ratio. The training set undergoes data augmentation and standardization preprocessing to obtain the HE image. and real IHC images The dataset; The virtual immunohistochemical staining generation model is constructed, including the generator G, the discriminator D, and the double-fidelity constraint module; The dataset is input into the virtual immunohistochemistry staining generation model, wherein the generator G adopts an encoder-decoder architecture to extract the HE images. and the actual IHC image Based on the multi-level structural features and pathological semantic features, the corresponding virtual IHC image is output. and identity reconstruction images ; The discriminator D employs a convolutional integral class network structure to distinguish the input image from the real IHC image. Or the virtual IHC image The generator G and the discriminator D form an adversarial network architecture. Through adversarial competition and iterative optimization, the generator G is forced to generate the virtual IHC image. Fitting the real IHC image The pixel distribution and overall style; The dual-fidelity constraint module includes a texture constraint module and a pathological information constraint module, wherein the texture constraint module acts on the CAR virtual staining synthesis subnetwork of the generator G to ensure the virtual IHC image With the HE image Structural alignment, and with the actual IHC image The staining style is consistent; the pathological information constraint module acts on the SGMS geometric registration subnetwork of the generator G to lock in core diagnostic features and ensure the virtual IHC image. The structural accuracy and effectiveness of pathological diagnosis.

[0012] According to one embodiment of the present invention, the SGMS geometric registration subnetwork includes a differential homeomorphism and outward flow unit coupled with a semi-Lagrangian composite differential homeomorphism integration unit. The coupled differential homeomorphism and external flow unit introduces a geometric velocity field and an image velocity field into the HE image. To the real IHC image Spatial geometric registration, and the HE image To the virtual IHC image Synthetic joint modeling; The geometric velocity field is: ; in, The pixel-level two-dimensional deformation velocity field output by the geometric network, with subscripts... These are the learnable weight parameters of the geometric network; For a pathological slide image, it is a two-dimensional spatial domain representing the set of all pixel coordinates; It is a continuous time interval, representing the complete time evolution process from the initial no variation to the final complete registration; In a two-dimensional real number space, the x and y axis two-dimensional deformation offsets corresponding to each pixel; The image velocity field is: ; in, The image pixel value change rate field output by the generator network, subscript These are the learnable weight parameters of the generator; It is a three-dimensional real number space, corresponding to the pixel value space of an RGB three-channel digital image; For the image domain The total number of pixels; For the HE image The feature space provides the generator with pathological structural condition inputs. To achieve the HE image To the real IHC image Smooth, wrinkle-free registration, while simultaneously integrating the spatial geometric registration and the virtual IHC image. The synthesis process is deeply coupled, defining a deformation sampling map with a time index. ,in, The two-dimensional pixel coordinate space of the pathological slide image; , is a continuous time variable, representing the complete evolution process from the initial unchanging state to the final fully registered state; Based on the geometric velocity field, the deformation sampling map is defined by the deformation mapping evolution ordinary differential equation. The continuous evolution rule, the ordinary differential equation of the deformation mapping evolution is: ; in, yes Time coordinates The new coordinates after deformation mapping ; for The HE feature map after dynamic registration at any time, its pixel coordinates The eigenvalue at that location is ,in For the HE image A pixel mapping function is used to output the HE image. At the new coordinates The pixel feature values ​​at the location are used to obtain the registered HE features; It is related to the HE image. Paired real IHC images ; As initial conditions, The time-deformation mapping is a unit mapping with no spatial transformation, and the coordinates retain their original values.

[0013] According to one embodiment of the present invention, the semi-Lagrangian composite differential homeomorphic integral unit is used to transform the deformation mapping evolution ordinary differential equation into an iteratively optimized discrete numerical computation scheme, the steps of which include: The continuous time interval Uniformly discretized The iteration step, the... The discrete time points corresponding to each step are: The corresponding discrete time step is ; Based on the discretized time steps, a semi-Lagrangian update is performed through a mapping composition operation to iteratively optimize the deformation mapping. The deformation mapping update formula is as follows: ; in, , These are the deformation sampling maps for the (k+1)th and kth discrete-time steps, respectively; Unit mapping; This represents the predicted deformation velocity field after the k-th step, which is transformed from the pixel coordinate system to the normalized coordinate system. This is a mapping compound operator; During the iteration process, the HE feature sequence corresponding to the registration optimization for each deformation mapping step is generated. Its pixel coordinates The eigenvalues ​​at that location satisfy ;in For the HE image The pixel mapping function is used to output the pixel feature value at the corresponding coordinates; These are the pixel coordinates of the image. This represents the number of discrete iteration steps.

[0014] According to one embodiment of the present invention, the CAR virtual staining synthesis subnetwork includes a modified flow matching basic definition unit, a time matching dynamic condition control unit, and a generation loss calculation unit; The modified flow matching basic definition unit is used to construct the true IHC image from Gaussian noise. The linear path probability flow provides path constraints and supervision labels for flow matching training. The steps include: Define the transformation from standard Gaussian noise to the true IHC image. Probability flow of straight path: ; in, The intermediate image stream at time t is the transition from Gaussian noise to the actual IHC image. The transitional state; For continuous time variables, ;in For interval Uniform distribution on This is the preset minimum positive value; Standard Gaussian noise, ;in It follows a multivariate standard normal distribution; Based on the probability flow of the straight path, the analytical solution of the true velocity field corresponding to this path is derived: ; in, The actual velocity field corresponding to this path represents the intermediate image stream. To the real IHC image The ideal speed of evolution; The time-matching dynamic condition control unit is used to progressively register and optimize the HE feature sequence with each iteration step. As a dynamic structural condition for image synthesis, the temporal matching of the flow evolution process with structural cues is achieved, and the steps include: During the discrete iterative training process, the HE feature sequence output by the SGMS geometric registration sub-network corresponding to the k-th iteration is used. As structural condition input, the HE feature map is dynamically registered in the corresponding continuous domain. ; As the iteration step k progresses, the HE feature sequence The registration accuracy is gradually improved, and the structural clues received by the generator G are optimized synchronously. Based on the above coupling architecture, the generator G supports two input modes: when the input is the HE image. At that time, the virtual IHC image is output. When the input is the actual IHC image At that time, the identity reconstruction image is output. , used for loss calculation of the texture constraint module; The generation loss calculation unit is used to constrain the fit between the image velocity field predicted by the generator G and the real velocity field, and to construct the flow matching loss. : ; in, For time variables and Gaussian noise Find the expected value; It is a function of the image velocity field; The intermediate image stream at time t; The HE feature map at time t; The actual IHC image; It is the square of the L2 norm, i.e., the mean square error.

[0015] According to one embodiment of the present invention, the texture constraint module includes a common parameter support unit, a texture alignment constraint link, and an identity style constraint link; the texture alignment constraint link and the identity style constraint link are two links executed in parallel. The common parameter support unit is used to provide a unified feature extraction and mapping basis for the texture alignment constraint link and the identity style constraint link, and the steps include: Define the set of network layers of the generator G. The network layer set It contains multi-layered networks that can capture low-level texture details, mid-level structural features, and high-level semantic information; Build with the generator G Shared multilayer perceptron corresponding to layer features The texture alignment constraint link and the identity style constraint link fully reuse the shared multilayer perceptron. To ensure that the feature mapping space is completely consistent; The texture alignment constraint link is used to ensure the virtual IHC image With the HE image The structural alignment process includes the following steps: A1: Obtain the HE image and the virtual IHC image output by the generator G And extract the set of network layers. The dimensions corresponding to each network layer are HE image multi-level feature map Multi-level feature maps of virtual IHC images Where C is the number of feature channels, H is the feature map height, and W is the feature map width; A2: In the multi-level feature map of the HE image With the virtual IHC image multi-level feature map At the same coordinate position, synchronous random sampling Each spatial location is used to extract the HE feature vector set corresponding to that spatial location. and virtual IHC feature vector set ; A3: The HE feature vector set and the virtual IHC feature vector set Input the shared multilayer perceptron Feature space mapping is performed to obtain the HE feature set. and virtual IHC feature set ; With each virtual IHC feature As the core, HE features at the same spatial location are set as positive sample HE features. All other HE features not located in the same spatial location are designated as negative sample HE features. ,in ,Right now For anchor point location All sampling locations outside; A4: Constructing the InfoNCE loss function for texture alignment The formula is as follows: ; in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; This represents the temperature coefficient; the superscript T is an abbreviation for "Text," indicating that the loss function focuses on texture alignment. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; The identity style constraint link is used to protect the virtual IHC image. Compared with the real IHC image The coloring style is consistent with the texture alignment constraint link, and the process is executed synchronously and in parallel throughout. The steps include: B1: Obtain the actual IHC image and the identity reconstruction image output by the generator G And extract the set of network layers. The dimensions corresponding to each network layer are Real IHC image multi-level feature map Multi-level feature maps of identity reconstruction images ; B2: In the multi-level feature map of the real IHC image With the multi-level feature map of the identity reconstruction image At the same coordinate position, synchronous random sampling Each spatial location is used to extract the true IHC feature vector set for that spatial location. Image feature vector set for identity reconstruction ; B3: The actual IHC feature vector set and the identity reconstruction image feature vector set Input the shared multilayer perceptron We perform feature space mapping to obtain the true IHC feature set. Image feature sets for identity reconstruction ; Reconstruct image features for each identity As the core, the true IHC features of the same spatial location are set as the true IHC features of the positive samples. All other true IHC features not from the same spatial location are set as negative sample true IHC features. ,in That is, i represents all sampling positions except for the anchor position k; B4: Constructing the Identity Style InfoNCE Loss Function The formula is as follows: ; in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; The value represents the temperature coefficient; the superscript I is an abbreviation for "ID", indicating that the loss function focuses on matching identity style with coloring style. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; The texture alignment InfoNCE loss function and the aforementioned identity style InfoNCE loss function After the calculation is completed, the total loss is calculated using the texture constraint total loss function. This completes the backpropagation and iterative optimization of the generator G; the total loss function for texture constraints is: ; in , These are weights that can be adjusted based on training results.

[0016] Secondly, embodiments of this application also provide a system for rapidly acquiring virtual IHC staining images during breast surgery, the system comprising: The acquisition module is used to acquire HE-stained digital whole-section images; An image conversion module is used to convert the HE-stained digital whole-slice image into a virtual IHC image. The image conversion module includes an SGMS network unit and a CAR network unit. The SGMS network unit is used to extract multi-level structural features of breast pathology and perform spatial geometric registration on the HE-stained digital whole-slice image, outputting a registration-optimized HE feature sequence. The CAR network unit is used to perform conditionally modified flow synthesis processing with the registration-optimized HE feature sequence as a time-matched dynamic structural condition to obtain the virtual IHC image.

[0017] The output module is used to output the virtual IHC image.

[0018] Thirdly, embodiments of this application also provide a computer device, the computer device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the steps of the method described above when executing the instructions.

[0019] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0020] The technical solution provided in this application has the following advantages: 1. The method for rapidly acquiring virtual IHC staining images during breast surgery in this application only requires inputting HE-stained digital whole-slice images into a pre-trained virtual immunohistochemical staining generation model, which can output virtual IHC images equivalent to traditional intraoperative IHC staining within minutes. This meets the time limit requirement of issuing diagnostic reports within 30 minutes for rapid intraoperative pathology, ensuring that the timeliness rate of rapid intraoperative pathology diagnosis meets the standards and conforms to the relevant assessment standards for tertiary hospital accreditation.

[0021] 2. The method for rapidly acquiring virtual IHC staining images during breast surgery described in this application eliminates the need for traditional intraoperative IHC staining procedures and the consumption of IHC staining-related reagents, thereby reducing manual operation costs and effectively alleviating the economic burden of diagnosis and treatment for patients.

[0022] 3. The method for rapidly acquiring virtual IHC staining images during breast surgery in this application only requires pathology technicians to complete the modified intraoperative rapid HE staining, scan the HE-stained sections, and input them into the model. It eliminates the need for the cumbersome experimental procedures of traditional intraoperative IHC staining, reducing the workload of technicians and significantly reducing manpower and time costs.

[0023] 4. The virtual immunohistochemical staining generation model of this application constructs a coupled continuous time frame through the SGMS geometric registration sub-network and CAR virtual staining synthesis sub-network of the generator G, jointly modeling differential homeomorphic geometric alignment and conditionally modified flow synthesis. Specifically, this application introduces corresponding driven geometric supervision through the SGMS geometric registration sub-network, thereby achieving smooth, wrinkle-free, and pixel-level accurate registration of HE and IHC images without the need for densely deformed labels. Secondly, this application achieves dynamic conditional control of time matching through the CAR virtual staining synthesis sub-network. During the flow evolution process, the generator G is guided by gradually aligned structural cues, ensuring that the generated virtual IHC image is structurally accurately aligned with the input HE image and highly fits the real IHC staining effect. This effectively solves the problems of tissue misalignment, structural distortion, and loss of pathological details in traditional virtual staining techniques, improving the pathological diagnostic effectiveness and clinical usability of virtual IHC images.

[0024] 5. The virtual immunohistochemical staining generation model of this application has a texture constraint module and a pathological information constraint module. The texture constraint module can ensure the structural alignment of the virtual IHC image with the HE image and the consistency of the staining style with the real IHC image. The pathological information constraint module can lock the core diagnostic features and ensure the structural accuracy of the virtual IHC image and the effectiveness of pathological diagnosis.

[0025] 6. The final step of the modified intraoperative rapid HE staining procedure in this application uses distilled water for mounting, which has the following advantages compared to the traditional HE staining method that uses neutral resin for mounting: 1) Using distilled water for mounting allows for easy and complete removal of coverslips after HE staining and scanning, without tearing or damaging the tissue, thus ensuring the structural integrity of the tissue sections. This solves the technical problem of traditional neutral resin mounting, which makes it difficult to remove coverslips and easily damages the tissue. 2) Relying on the characteristic that coverslips can be removed non-destructively with distilled water, this application can perform HE staining and IHC staining sequentially on the same tissue section during the model training stage. This allows the input real HE image, real IHC image and the virtual IHC image generated by the model to achieve precise correspondence at the same cell and the same pixel position, improving the fidelity and accuracy of virtual IHC image generation. It solves the technical problem in traditional pathological artificial intelligence research that HE staining and IHC staining can only be completed on adjacent sections, inevitably resulting in spatial misalignment and tissue loss. This lays a solid underlying data foundation for the clinical pathological diagnostic effectiveness of virtual IHC images.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an overall flowchart of the method for rapidly acquiring virtual IHC staining images during breast surgery according to this application.

[0029] Figure 2 This is a flowchart of the training process for the virtual immunohistochemical staining generation model in this application.

[0030] Figure 3 This is an architecture diagram of the system for rapidly acquiring virtual IHC staining images during breast surgery, as described in this application.

[0031] Figure 4 These are comparison images of stained images. (a), (b), (c), and (d) are the actual HE staining images at the first, second, third, and fourth positions, respectively. (e), (f), (g), and (h) are the actual IHC staining images at the first, second, third, and fourth positions, respectively. (i), (j), (k), and (l) are the virtual IHC images generated by this model at the first, second, third, and fourth positions, respectively. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0033] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0034] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Example like Figure 1 As shown, this embodiment provides a method for rapidly acquiring virtual IHC staining images during breast surgery, including the following steps: S1: Obtain breast lesion tissue specimens from the patient; S2: The breast lesion tissue specimen was frozen sectioned and subjected to modified intraoperative rapid HE staining to obtain HE-stained sections; S3: Perform a full-field scan on the HE-stained section to obtain a digital full-section image of HE staining corresponding to the HE-stained section; S4: Input the HE-stained digital whole-slice image into a pre-trained virtual immunohistochemical staining generation model; the virtual immunohistochemical staining generation model includes a generator G, which includes an SGMS geometric registration subnetwork and a CAR virtual staining synthesis subnetwork; The SGMS geometric registration subnetwork is used to extract multi-level structural features of breast pathology and perform spatial geometric registration on the HE-stained digital whole-section images, outputting a registration-optimized HE feature sequence. ; The CAR virtual staining synthesis subnetwork is registered with the optimized HE feature sequence. For time-matched dynamic structural conditions, perform conditional modified stream synthesis processing to output a virtual IHC image. .

[0036] The steps of a method for rapidly acquiring virtual IHC staining images during breast surgery, as described in this application, are detailed below for clinical use: The virtual immunohistochemical staining generation model, which has been trained and clinically validated, will be deployed to the dedicated workstation of the digital pathology system in the pathology department. Software compatibility debugging and operating environment configuration will be completed to ensure compliance with the clinical use guidelines for medical software.

[0037] Complete the deep matching between the slide scanner and the model system to ensure that the optical magnification, image format, resolution, color space, and white balance parameters of the clinical scan are completely consistent with the input data during the model training phase, thus eliminating input deviations.

[0038] S1: Obtain breast lesion tissue specimens from the patient; S2: The breast lesion tissue specimen was frozen sectioned and subjected to modified intraoperative rapid HE staining to obtain HE-stained sections; Specifically, the surgeon removes a breast lesion tissue specimen from the patient during the operation and immediately transfers it to the pathology department's frozen section after removal from the body. After the pathologist verifies that the patient information and specimen information are correct, the specimen is immediately embedded in OCT glue and frozen at -20°C. Then, a modified intraoperative rapid HE staining is performed to obtain HE-stained sections. The preparation of frozen sections here is the same as the modified intraoperative rapid HE staining and model training phase.

[0039] S3: Perform a full-field scan on the HE-stained section to obtain a digital full-section image of HE staining corresponding to the HE-stained section; Specifically, the pathologist places the HE-stained slides into a pre-configured slide scanner and performs a full-field scan with the same parameters as in the model training phase to prepare digital whole-slide images (WSIs), namely the HE-stained digital whole-slide images.

[0040] S4: Input the HE-stained digital whole-slice image into a pre-trained virtual immunohistochemical staining generation model; the virtual immunohistochemical staining generation model includes a generator G, which includes an SGMS geometric registration subnetwork and a CAR virtual staining synthesis subnetwork; the SGMS geometric registration subnetwork performs multi-level structural feature extraction and spatial geometric registration of the HE-stained digital whole-slice image, and outputs a registration-optimized HE feature sequence. The CAR virtual staining synthesis subnetwork is registered with the optimized HE feature sequence. For time-matched dynamic structural conditions, perform conditional modified stream synthesis processing to output a virtual IHC image. .

[0041] Based on real HE staining images and generated virtual IHC images, pathologists can quickly differentiate between benign and malignant breast lesions, determine the pathological type of the lesion, and issue an intraoperative rapid pathological diagnosis report.

[0042] Based on the intraoperative rapid pathology report, the surgeon communicates with the patient's family to obtain informed consent before immediately developing and implementing the corresponding surgical plan.

[0043] This application's method for rapidly acquiring virtual IHC staining images during breast surgery only requires inputting HE-stained digital whole-slice images into a pre-trained virtual immunohistochemical staining generation model. It can output virtual IHC images equivalent to traditional intraoperative IHC staining within minutes, thus meeting the 30-minute time limit requirement for rapid intraoperative pathology reports and ensuring timely intraoperative pathology diagnosis, aligning with the relevant assessment standards for tertiary hospital accreditation. This method eliminates the need for traditional intraoperative IHC staining experiments and reagents, reducing labor costs and effectively alleviating the economic burden on patients. Furthermore, this method only requires pathology technicians to perform a modified intraoperative rapid HE staining procedure, scanning the HE-stained slides and inputting the data into the model, eliminating the need for cumbersome traditional intraoperative IHC staining procedures, thus reducing the workload of technicians and significantly decreasing manpower and time costs.

[0044] The above describes a method for rapidly acquiring virtual IHC staining images during breast surgery. The following section details the training of the virtual immunohistochemical staining generation model in this method (e.g., Figure 2 (As shown), including the following steps: S101: Collect several pathological specimens from breast surgery, complete the modified intraoperative rapid HE staining and IHC staining of the same tissue section, and obtain a one-to-one paired HE image and IHC image to be processed after scanning. S102: Manually extract the core region of interest (ROI) of the pathological tumor from the HE image and the IHC image to be processed, and divide them into training set, validation set, and test set according to a preset ratio. Perform data augmentation and standardization preprocessing on the training set to obtain the HE image. and real IHC images The dataset; S103: Construct the virtual immunohistochemical staining generation model, including the generator G, discriminator D, and double-fidelity constraint module; S104: Input the dataset into the virtual immunohistochemical staining generation model, wherein the generator G adopts an encoder-decoder main structure to extract the HE image. and the actual IHC image Based on the multi-level structural features and pathological semantic features, the corresponding virtual IHC image is output. and identity reconstruction images The discriminator D employs a convolutional class network structure to distinguish the input image as the real IHC image. Or the virtual IHC image The generator G and the discriminator D form an adversarial network architecture. Through adversarial competition and iterative optimization, the generator G is forced to generate the virtual IHC image. Fitting the real IHC image The pixel distribution and overall style; the dual-fidelity constraint module includes a texture constraint module and a pathological information constraint module, wherein the texture constraint module acts on the CAR virtual staining synthesis subnetwork of the generator G to ensure the virtual IHC image With the HE image Structural alignment, and with the actual IHC image The staining style is consistent; the pathological information constraint module acts on the SGMS geometric registration subnetwork of the generator G to lock in core diagnostic features and ensure the virtual IHC image. The structural accuracy and effectiveness of pathological diagnosis.

[0045] The following details the training steps of the virtual immunohistochemical staining generation model: S101: Collect several pathological specimens from breast surgery, complete the modified intraoperative rapid HE staining and IHC staining of the same tissue section, and obtain a one-to-one paired HE image and IHC image to be processed after scanning. This step specifically includes the following steps: Step 1011: Collect approximately 200 rapid intraoperative specimens from breast surgeries, covering the full spectrum of lesions including breast adenosis, fibroadenoma, intraductal papilloma, intraductal papillary carcinoma, ductal carcinoma in situ, and invasive carcinoma, to ensure sample diversity; all fresh surgical specimens were immediately embedded in OCT gel after excision and frozen at -20°C in preparation for subsequent sectioning.

[0046] Step 1012: Cut the frozen-fixed specimen into tissue sections with a thickness of about 3 μm in a cryostat, and then immediately fix them in 95% ethanol solution to avoid tissue degradation.

[0047] Step 1013: Perform modified intraoperative rapid HE staining to obtain HE-stained sections; In one possible implementation, the modified intraoperative rapid HE staining step includes the following steps: The slices prepared by the frozen sectioning were rinsed with distilled water for 20 seconds. The cell nuclei of the slide were stained with hematoxylin solution for 20 seconds. Rinse the slices with running water for 20 seconds to remove residual hematoxylin solution; The slides were soaked in the blueing solution for 10 seconds to complete the blueing of the cell nuclei. Rinse the slices again with running water for 20 seconds to remove any residual blueing solution. The slide was stained with 0.5% eosin solution for 20 seconds. The slices were soaked in 95% ethanol for 10 seconds to complete the initial dehydration. The sections were soaked in anhydrous ethanol for 10 seconds to complete the secondary dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the three-stage dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the fourth stage of deep dehydration. The slices were soaked in xylene for 10 seconds to achieve initial transparency. The slices were soaked in xylene for 10 seconds to achieve secondary transparency. The slices were soaked in xylene for 10 seconds to achieve three levels of deep transparency. The HE-stained sections are obtained by mounting with distilled water. This step includes: dropping an appropriate amount of distilled water onto the tissue section that has been cleared to a third depth; covering the entire tissue with a coverslip along one side of the tissue to avoid air bubbles; absorbing excess water with absorbent paper; and completing the mounting.

[0048] The final step of this improved intraoperative rapid HE staining procedure uses distilled water for mounting, which has the following advantages compared to the traditional HE staining method that uses neutral resin for mounting: 1) Using distilled water for mounting allows for easy and complete removal of coverslips after HE staining and scanning, without tearing or damaging the tissue, thus ensuring the structural integrity of the tissue sections. This solves the technical problem of traditional neutral resin mounting, which makes it difficult to remove coverslips and easily damages the tissue. 2) Relying on the characteristic that coverslips can be removed non-destructively with distilled water, this application can perform HE staining and IHC staining sequentially on the same tissue section during the model training stage. This allows the input real HE image, real IHC image and the virtual IHC image generated by the model to achieve precise correspondence at the same cell and the same pixel position, improving the fidelity and accuracy of virtual IHC image generation. It solves the technical problem in traditional pathological artificial intelligence research that HE staining and IHC staining can only be completed on adjacent sections, inevitably resulting in spatial misalignment and tissue loss. This lays a solid underlying data foundation for the clinical pathological diagnostic effectiveness of virtual IHC images.

[0049] It should be noted that, due to the difficulty in removing coverslips after traditional neutral resin mounting and the risk of tissue damage, both routine intraoperative procedures in current clinical pathology diagnosis and related research in the field of traditional pathological artificial intelligence are performed on adjacent continuous tissue sections for HE staining and IHC staining. Even if adjacent sections are taken from the same tissue block and the section thickness is controlled at the micrometer level, it is impossible to guarantee that the tissue morphology, cell distribution, and spatial position of the two sections are completely consistent. Inevitably, problems such as tissue spatial misalignment and local tissue loss will occur, making it impossible to form an accurate pairing relationship between real HE images and real IHC images. This causes deviation in the supervision signal during the training of artificial intelligence models, leading to the model learning incorrect feature associations and reducing the fidelity and accuracy of the images generated by the model. This application solves this long-standing technical problem from the root of data preparation by using the above-mentioned improved distilled water mounting scheme.

[0050] Step 1014: Using a slide scanner, perform a full-field scan of the HE-stained slides at 40X optical magnification, and then prepare the scanned slides into digital whole-slide images (WSIs), i.e. the HE images to be processed, and then archive and store them according to the sample number.

[0051] Step 1015: Gently remove the coverslip from the HE slide, rinse the slide with distilled water to remove any possible impurities, and then place the rinsed slide in PBS buffer and let it stand at room temperature to complete the hydration preparation before immunohistochemical staining.

[0052] Step 1016: Perform IHC staining to obtain IHC stained sections; the specific steps include the following: At room temperature, incubate sections with a mixture of CK5 / 6 and P63 primary antibodies for 10 min; Rinse the sections three times with PBS buffer to remove unbound primary antibody; Incubate the sections with the corresponding secondary antibody for 10 minutes at room temperature; Rinse the slides three times with PBS buffer to remove unbound secondary antibody; At room temperature, incubate the sections with DAB chromogenic solution for 2 minutes to complete the specific labeling and color development; Soak the sections in 95% ethanol for 10 seconds to complete the initial dehydration; The sections were soaked in anhydrous ethanol for 10 seconds to complete the secondary dehydration. The sections were soaked in anhydrous ethanol for 10 seconds to complete the three-stage dehydration. The sections were soaked in anhydrous ethanol for 10 seconds to complete the fourth stage of deep dehydration. The section was soaked in xylene for 10 seconds to achieve initial clearing. The sections were soaked in xylene for 10 seconds to achieve secondary transparency. The sections were soaked in xylene for 10 seconds to achieve level 3 transparency. IHC staining was performed by mounting slides with neutral resin.

[0053] Step 1017: Using a slide scanner with the same 40X magnification and scanning parameters as the HE slides, perform a full-field scan on the IHC-stained slides, and then prepare IHC digital whole-slide images (WSIs) paired with the HE slides, i.e., the IHC images to be processed. These images are bound and archived one by one with the HE slides of the same number to ensure accurate pairing.

[0054] S102: Manually extract the core region of interest (ROI) of the pathological tumor from the HE image and the IHC image to be processed, and divide them into training set, validation set, and test set according to a preset ratio. Perform data augmentation and standardization preprocessing on the training set to obtain the HE image. and real IHC images The dataset; This step specifically includes the following steps: Step 1021: Performed by two or more senior pathologists, manually delineate the core region of interest (ROI) of the pathological tumor for each pair of bound HE and IHC slides.

[0055] Step 1022: For each case, extract multiple ROIs at different magnifications according to the field of view gradient from low magnification (2X) to high magnification (40X); Strictly ensure that the ROIs of HE-stained sections and IHC-stained sections are completely matched in terms of field of view, magnification, and spatial coordinates, and remove unqualified ROIs with tissue folds, uneven staining, detachment, or labeling deviations to complete the final effective ROI screening.

[0056] Step 1023: Stratify by case level to ensure a balanced distribution of lesion types in each subset. Randomly divide all paired ROIs into three sets: 50% training set, 25% validation set, and 25% test set. The training set is used for model weight training, the validation set is used for hyperparameter tuning and optimal model selection, and the test set is used for final model performance evaluation.

[0057] Step 1024: For the paired ROIs in the training set, first perform random color modification to simulate the color difference between different batches of coloring, and then scale them at a random ratio in the range of 0.95 to 1.05 to simulate the size difference between different scanning scenarios.

[0058] Step 1025: Extract non-overlapping 540×540 pixel image patches from the expanded ROI. Apply diagonal flip, horizontal or vertical mirroring, or random rotation of 90°, 180° or 270° to each patch in sequence. Then apply Gaussian blur and median blur to expand the training sample size and improve the model's generalization ability.

[0059] Step 1026: Scale the enhanced patches to 128×128 pixels to complete pixel value normalization, obtaining the final HE image. and real IHC images The dataset.

[0060] S103: Construct the virtual immunohistochemical staining generation model, including the generator G, discriminator D, and double-fidelity constraint module; S104: Input the dataset into the virtual immunohistochemical staining generation model, wherein the generator G adopts an encoder-decoder main structure to extract the HE image. and the actual IHC image Based on the multi-level structural features and pathological semantic features, the corresponding virtual IHC image is output. and identity reconstruction images The discriminator D employs a convolutional class network structure to distinguish the input image as the real IHC image. Or the virtual IHC image The generator G and the discriminator D form an adversarial network architecture. Through adversarial competition and iterative optimization, the generator G is forced to generate the virtual IHC image. Fitting the real IHC image The pixel distribution and overall style; the dual-fidelity constraint module includes a texture constraint module and a pathological information constraint module, wherein the texture constraint module acts on the CAR virtual staining synthesis subnetwork of the generator G to ensure the virtual IHC image With the HE image Structural alignment, and with the actual IHC image The staining style is consistent; the pathological information constraint module acts on the SGMS geometric registration subnetwork of the generator G to lock in core diagnostic features and ensure the virtual IHC image. The structural accuracy and effectiveness of pathological diagnosis.

[0061] It should be noted that the SGMS geometric registration sub-network plays a core registration role only during the training phase of the virtual immunohistochemistry staining generation model. During model training, we have real HE images and real IHC images paired with the same tissue slice. The SGMS geometric registration sub-network completes smooth, wrinkle-free, pixel-level accurate registration of these two sets of real images through corresponding driven geometric supervision, and outputs a registration-optimized HE feature sequence that is progressively optimized with the number of iterations. This provides precisely aligned dynamic structural cues for the subsequent generation of virtual IHC images by the CAR virtual staining synthesis subnetwork.

[0062] During the clinical inference phase, only HE-stained digital whole-slice images need to be input into the pre-trained virtual immunohistochemical staining generation model; no real IHC images are input. At this time, the SGMS geometric registration sub-network no longer performs registration operations with real IHC images. Instead, it reuses the weight parameters that have been solidified after training to complete the extraction of multi-level structural features of breast pathology. This provides the CAR virtual staining synthesis sub-network with suitable pathological structural features. Finally, the CAR virtual staining synthesis sub-network uses these structural features as time-matched dynamic structural conditions to perform conditional modified flow synthesis processing and output virtual IHC images.

[0063] The generator G is described below: The generator G includes an SGMS geometric registration subnetwork and a CAR virtual staining synthesis subnetwork; In one possible implementation, the SGMS geometric registration subnetwork includes a differential homeomorphism and outward flow unit coupled with a semi-Lagrangian composite differential homeomorphism integral unit. The coupled differential homeomorphism and external flow unit introduces a geometric velocity field and an image velocity field into the HE image. To the real IHC image Spatial geometric registration, and the HE image To the virtual IHC image Synthetic joint modeling; The geometric velocity field is: ; in, The pixel-level two-dimensional deformation velocity field output by the geometric network, with subscripts... These are the learnable weight parameters of the geometric network; For a pathological slide image, it is a two-dimensional spatial domain representing the set of all pixel coordinates; It is a continuous time interval, representing the complete time evolution process from the initial no deformation to the final complete registration; In a two-dimensional real number space, the x and y axis two-dimensional deformation offsets corresponding to each pixel; The geometric velocity field is used to predict pixel-level deformation offset.

[0064] The image velocity field is: ; in, The image pixel value change rate field output by the generator network, subscript These are the learnable weight parameters of the generator; It is a three-dimensional real number space, corresponding to the pixel value space of an RGB three-channel digital image; For the image domain The total number of pixels; For the HE image The feature space provides the generator with pathological structural condition inputs. The image velocity field is used to extract pathological semantic features based on the registered HE features.

[0065] To achieve the HE image To the real IHC image Smooth, wrinkle-free registration, while simultaneously integrating the spatial geometric registration and the virtual IHC image. The synthesis process is deeply coupled, defining a deformation sampling map with a time index. ,in, The two-dimensional pixel coordinate space of the pathological slide image; , is a continuous time variable, representing the complete evolution process from the initial unchanging state to the final fully registered state; The deformation sampling mapping is used to achieve spatial geometric registration from HE image to IHC image.

[0066] Based on the geometric velocity field, the deformation sampling map is defined by the deformation mapping evolution ordinary differential equation. The continuous evolution rule, the ordinary differential equation of the deformation mapping evolution is: ;in, yes Time coordinates The new coordinates after deformation mapping ; for The HE feature map after dynamic registration at any time, its pixel coordinates The eigenvalue at that location is ,in For the HE image A pixel mapping function is used to output the HE image. At the new coordinates The pixel feature values ​​at the location are used to obtain the registered HE features; It is related to the HE image. Paired real IHC images ; As initial conditions, The time-deformation mapping is a unit mapping with no spatial transformation, and the coordinates retain their original values.

[0067] The deformation mapping evolution ordinary differential equation realizes the continuous and controllable evolution of deformation mapping: the deformation rate of pixel coordinates is driven by the currently registered HE features and the real IHC image. Starting from the initial undeformed state, it gradually realizes the differential homeomorphic registration from the HE image to the real IHC image over time. Mathematically, it ensures the smoothness of the registration process and avoids problems such as tissue wrinkles, tears, and misalignments, providing accurately aligned structural features for subsequent virtual IHC image synthesis.

[0068] The aforementioned deformation mapping evolution ordinary differential equation defines the continuous evolution rule of deformation mapping. However, the training and inference of deep learning models are discrete iterative processes. In order to transform the continuous differential homeomorphic registration into a trainable and numerically stable implementation scheme, and at the same time ensure that the registration process is smooth and free of organizational wrinkles and tears from the iterative logic, this application adopts the semi-Lagrangian composite differential homeomorphic integral method to discretize and solve the continuous ordinary differential equation, so as to achieve wrinkle-free and pixel-level accurate registration from HE image to IHC image.

[0069] In one possible implementation, the semi-Lagrangian composite differential homeomorphic integral unit is used to transform the deformation mapping evolution ordinary differential equation into an iteratively optimizeable discrete numerical computation scheme, the steps of which include: D1: The continuous time interval Uniformly discretized The iteration step, the... The discrete time points corresponding to each step are: The corresponding discrete time step is ; D2: Based on the discretized time step, a semi-Lagrangian update is performed through a mapping composition operation to iteratively optimize the deformation mapping. The deformation mapping update formula is as follows: ; in, , These are the deformation sampling maps for the (k+1)th and kth discrete-time steps, respectively; Unit mapping; This represents the predicted deformation velocity field after the k-th step, which is transformed from the pixel coordinate system to the normalized coordinate system. This is a mapping compound operator; D3: During the iteration process, the HE feature sequence corresponding to the registration optimization of each deformation mapping step is generated. Its pixel coordinates The eigenvalues ​​at that location satisfy ;in For the HE image The pixel mapping function is used to output the pixel feature value at the corresponding coordinates; These are the pixel coordinates of the image; This represents the number of discrete iteration steps.

[0070] The aforementioned SGMS geometric registration subnetwork achieves wrinkle-free pixel-level registration from HE images to IHC images, and outputs a registration-optimized HE feature sequence that is progressively optimized with the number of iterations. The CAR virtual staining synthesis subnetwork serves as the core of the generator's virtual staining generation, capable of generating HE feature sequences based on registration optimization. This enables end-to-end generation from HE images to IHC images. In one possible implementation, the CAR virtual staining synthesis subnetwork includes a modified flow matching basic definition unit, a time matching dynamic condition control unit, and a generation loss calculation unit; The modified flow matching basic definition unit is used to construct the true IHC image from Gaussian noise. The linear path probability flow provides path constraints and supervision labels for flow matching training. The steps include: E1: Defines the transition from standard Gaussian noise to the true IHC image. Probability flow of straight path: in, The intermediate image stream at time t is the transition from Gaussian noise to the actual IHC image. The transitional state; For continuous time variables, ;in For interval Uniform distribution on This is the preset minimum positive value; Standard Gaussian noise, ;in It follows a multivariate standard normal distribution; This formula defines the linear path probability flow of the generation process, providing explicit generation path constraints for subsequent flow matching training, and is the fundamental definition for conditional modified flow synthesis.

[0071] E2: Based on the probability flow of the straight path, the analytical solution of the true velocity field corresponding to this path is derived: ; in, The actual velocity field corresponding to this path represents the intermediate image stream. To the real IHC image The ideal speed of evolution; This formula provides the analytical solution for the true velocity field of the probabilistic flow evolution, providing accurate supervision labels for the generator's image velocity field prediction, and is the core foundation of the subsequent flow matching loss function.

[0072] The time-matching dynamic condition control unit is used to progressively register and optimize the HE feature sequence with each iteration step. As a dynamic structural condition for image synthesis, the temporal matching of the flow evolution process with structural cues is achieved, and the steps include: F1: During the discrete iterative training process, the HE feature sequence output by the SGMS geometric registration sub-network corresponding to the k-th iteration. As structural condition input, the HE feature map is dynamically registered in the corresponding continuous domain. ; F2: As the iteration step k progresses, the HE feature sequence... The registration accuracy is gradually improved, and the structural clues received by the generator G are optimized synchronously. F3: Based on the above coupling architecture, the generator G supports two input modes: when the input is the HE image... At that time, the virtual IHC image is output. When the input is the actual IHC image At that time, the identity reconstruction image is output. , used for loss calculation of the texture constraint module; Traditional conditional generation models often use fixed structural features as generation conditions, which cannot solve the structural misalignment problem when pairing HE and IHC images. This results in poor structural alignment between the generated virtual IHC image and the original HE image, failing to meet the accuracy requirements for pathological diagnosis. This application achieves temporal matching between the flow evolution process and structural cues, solving the structural misalignment problem caused by traditional fixed-condition generation.

[0073] The generation loss calculation unit is used to constrain the fit between the image velocity field predicted by the generator G and the real velocity field, and to construct the flow matching loss. : Among them, For time variables and Gaussian noise Find the expected value; It is a function of the image velocity field; The intermediate image stream at time t; The HE feature map at time t; The actual IHC image; It is the square of the L2 norm, i.e., the mean square error.

[0074] This loss function ensures the consistency between the generated virtual IHC image and the real IHC image by constraining the fit between the image velocity field predicted by the generator and the real velocity field. At the same time, combined with dynamic condition control of time matching, it achieves accurate structural alignment between the generated image and the original HE image, providing reliable virtual staining results for subsequent pathological diagnosis.

[0075] It should be noted that in the actual discrete iterative training of the model, the continuous-time variable in the formula... Corresponding discrete iteration steps HE feature map The output of the corresponding SGMS sub-network This achieves a complete mapping from theoretical definition to engineering implementation.

[0076] The virtual immunohistochemical staining generation model of this application constructs a coupled continuous time frame through the SGMS geometric registration subnetwork and the CAR virtual staining synthesis subnetwork of the generator G, jointly modeling differential homeomorphic geometric alignment and conditionally modified flow synthesis. Specifically, this application introduces corresponding driven geometric supervision through the SGMS geometric registration subnetwork, thereby achieving smooth, wrinkle-free, and pixel-level accurate registration of HE and IHC images without the need for densely deformed labels. Secondly, this application achieves dynamic conditional control of time matching through the CAR virtual staining synthesis subnetwork. During the flow evolution process, the generator G is guided by gradually aligned structural cues, ensuring that the generated virtual IHC image is structurally accurately aligned with the input HE image and highly fits the real IHC staining effect. This effectively solves the problems of tissue misalignment, structural distortion, and loss of pathological details in traditional virtual staining techniques, improving the pathological diagnostic effectiveness and clinical usability of virtual IHC images.

[0077] The discriminant D is described below: The discriminator D employs a convolutional integral class network structure to distinguish the input image from the real IHC image. Or the virtual IHC image The generator G and the discriminator D form an adversarial network architecture. Through adversarial competition and iterative optimization, the generator G is forced to generate the virtual IHC image. Fitting the real IHC image The pixel distribution and overall style; The following describes the double-fidelity constraint module: The dual-fidelity constraint module includes a texture constraint module and a pathological information constraint module.

[0078] In one possible implementation, the texture constraint module includes a common parameter support unit, a texture alignment constraint link, and an identity style constraint link; the texture alignment constraint link and the identity style constraint link are two links executed in parallel. The common parameter support unit is used to provide a unified feature extraction and mapping basis for the texture alignment constraint link and the identity style constraint link, and the steps include: C1: Define the set of network layers for the generator G. The network layer set It contains multi-layered networks that can capture low-level texture details, mid-level structural features, and high-level semantic information; C2: Constructing the generator G. Shared multilayer perceptron corresponding to layer features The texture alignment constraint link and the identity style constraint link fully reuse the shared multilayer perceptron. To ensure that the feature mapping space is completely consistent; The texture alignment constraint link is used to ensure the virtual IHC image With the HE image The structural alignment process includes the following steps: A1: Obtain the HE image and the virtual IHC image output by the generator G And extract the network layer set The dimensions corresponding to each network layer are HE image multi-level feature map Multi-level feature maps of virtual IHC images Where C is the number of feature channels, H is the feature map height, and W is the feature map width; A2: In the multi-level feature map of the HE image With the virtual IHC image multi-level feature map At the same coordinate position, synchronous random sampling Each spatial location is used to extract the HE feature vector set corresponding to that spatial location. and virtual IHC feature vector set ; A3: The HE feature vector set and the virtual IHC feature vector set Input the shared multilayer perceptron Feature space mapping is performed to obtain the HE feature set. and virtual IHC feature set ; With each virtual IHC feature As the core, HE features at the same spatial location are set as positive sample HE features. All other HE features not located in the same spatial location are designated as negative sample HE features. ,in ,Right now For anchor point location All sampling locations outside; A4: Constructing the InfoNCE loss function for texture alignment The formula is as follows: ; in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; This represents the temperature coefficient; the superscript T is an abbreviation for "Text," indicating that the loss function focuses on texture alignment. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; This is the main constraint loss for generator training. The core objective is to force the generated virtual IHC image to be precisely aligned with the input original HE image in terms of tissue, cell, spatial location, and texture structure.

[0079] The identity style constraint link is used to protect the virtual IHC image. Compared with the real IHC image The coloring style is consistent with the texture alignment constraint link, and the process is executed synchronously and in parallel throughout. The steps include: B1: Obtain the actual IHC image and the identity reconstruction image output by the generator G And extract the network layer set The dimensions corresponding to each network layer are Real IHC image multi-level feature map Multi-level feature maps of identity reconstruction images ; B2: In the multi-level feature map of the real IHC image With the multi-level feature map of the identity reconstruction image At the same coordinate position, synchronous random sampling Each spatial location is used to extract the true IHC feature vector set for that spatial location. Image feature vector set for identity reconstruction ; B3: The actual IHC feature vector set and the identity reconstruction image feature vector set Input the shared multilayer perceptron Feature space mapping is performed to obtain the true IHC feature set. Image feature sets for identity reconstruction ; Reconstruct image features for each identity As the core, the true IHC features of the same spatial location are set as the true IHC features of the positive samples. All other true IHC features not from the same spatial location are set as negative sample true IHC features. ,in That is, i represents all sampling positions except for the anchor position k; B4: Constructing the Identity Style InfoNCE Loss Function The formula is as follows: ; in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; The value represents the temperature coefficient; the superscript I is an abbreviation for "ID", indicating that the loss function focuses on matching identity style with coloring style. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; This is the auxiliary constraint loss for generator training. The core objective is to force the generator to learn the style features, grayscale distribution, and color development rules of real IHC staining, so that the generated virtual IHC images are completely consistent with the real IHC staining effects routinely used in pathology clinics, and conform to the reading habits of pathologists.

[0080] The texture alignment InfoNCE loss function and the aforementioned identity style InfoNCE loss function After the calculation is completed, the total loss is calculated using the texture constraint total loss function. This completes the backpropagation and iterative optimization of the generator G; the total loss function for texture constraints is: ; in , These are weights that can be adjusted based on training results.

[0081] Ultimately, the total loss, texture alignment loss, and identity style loss together form the generator's full loss function, completing the backpropagation and iterative optimization of the generator weights.

[0082] The texture constraint module acts on the CAR virtual staining synthesis subnetwork of the generator G to ensure the quality of the virtual IHC image. With the HE image Structural alignment, and with the actual IHC image The dyeing style is consistent.

[0083] The pathological information constraint module acts on the SGMS geometric registration subnetwork of the generator G to lock in core diagnostic features and ensure the virtual IHC image. The structural accuracy and effectiveness of pathological diagnosis.

[0084] It should be noted that the pathological information constraint module is based on existing technology, including spatial alignment constraint technology for non-rigid registration of pathological images and supervised constraint technology for semantic consistency matching of pathological features. Specifically, a pathological information-specific loss is set to ensure that the semantic correspondence of pathological features at the same spatial coordinates is complete during feature extraction and image generation, with a focus on ensuring that core diagnostic features related to myoepithelial cells are not lost, misplaced, or distorted. An image spatial continuity regularization term is added to avoid tissue texture breaks and structural distortions in the generated image, ensuring the spatial continuity and morphological integrity of pathological tissues, and ensuring that the generated virtual IHC image conforms to the morphological standards of clinical pathological diagnosis.

[0085] The following describes the environment configuration and training phases for model training: Training was performed on a computing device equipped with a single Nvidia RTX 3090 GPU, using the PyTorch deep learning framework and the Adam optimizer.

[0086] The first stage of pre-training involves training the SGMS module of the model, setting the total number of iterations to 250,000 and the initial learning rate to 2e-4. During training, the model performance is evaluated using a validation set, and the model weights that perform best on the validation set are saved. The second stage of fine-tuning training: Based on the optimal model weights selected in the first stage of validation set, the CAR module of the model is fine-tuned and trained. The total number of iterations is set to 100,000, and the initial learning rate is set to 5e-5. The performance of the validation set is monitored synchronously during the training process, and the optimal model weights of the entire process are finally saved.

[0087] like Figure 4 As shown, Figure 4 In the diagram, (a), (b), (c), and (d) are the actual HE staining images at the first, second, third, and fourth positions, respectively; (e), (f), (g), and (h) are the actual IHC staining images at the first, second, third, and fourth positions, respectively; and (i), (j), (k), and (l) are the virtual IHC images generated by the model at the first, second, third, and fourth positions, respectively. It can be seen that the virtual IHC images are highly similar to the corresponding actual IHC staining images in terms of overall staining style, color hue, and positive expression distribution, and achieve pixel-level precise alignment with the corresponding actual HE staining images in terms of tissue structure, cell morphology, and spatial layout.

[0088] like Figure 3 As shown, based on the same application concept, this application embodiment also provides a system for rapidly acquiring virtual IHC stained images during breast surgery, including an acquisition module, an image conversion module, and an output module; wherein, the acquisition module is used to acquire HE stained digital whole-slice images; the image conversion module is used to convert the HE stained digital whole-slice images into virtual IHC images; the image conversion module includes an SGMS network unit and a CAR network unit; wherein, the SGMS network unit is used to perform breast pathology multi-level structural feature extraction and spatial geometric registration on the HE stained digital whole-slice images, and output a registration-optimized HE feature sequence; the CAR network unit is used to perform conditionally modified flow synthesis processing with the registration-optimized HE feature sequence as a time-matched dynamic structural condition to obtain the virtual IHC image. The output module is used to output the virtual IHC image.

[0089] Based on the same concept, embodiments of this application also provide a computer device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by a processor, the instructions being executed by the at least one processor to cause the at least one processor to implement the above-described method when executing the instructions.

[0090] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method.

[0091] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0092] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0097] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for rapidly acquiring virtual IHC staining images during breast surgery, characterized in that, include: Obtain breast lesion tissue specimens from the patient; The breast lesion tissue specimen was subjected to frozen section preparation and modified intraoperative rapid HE staining to obtain HE-stained sections; The HE-stained section was scanned in its entirety to obtain a digital full-section image of HE staining corresponding to the HE-stained section. The HE-stained digital whole-slice image is input into a pre-trained virtual immunohistochemical staining generation model; the virtual immunohistochemical staining generation model includes a generator G, which includes an SGMS geometric registration subnetwork and a CAR virtual staining synthesis subnetwork. The SGMS geometric registration subnetwork is used to extract multi-level structural features of breast pathology and perform spatial geometric registration on the HE-stained digital whole-section images, outputting a registration-optimized HE feature sequence. ; The CAR virtual staining synthesis subnetwork is registered with the optimized HE feature sequence. For time-matched dynamic structural conditions, perform conditional modified stream synthesis processing to output a virtual IHC image. ; The SGMS geometric registration subnetwork includes a differential homeomorphism and outward flow unit coupled with a semi-Lagrangian composite differential homeomorphism integration unit. The coupled differential homeomorphism and external flow unit introduces geometric velocity field and image velocity field into the HE image. To the real IHC image Spatial geometric registration, and the HE image To the virtual IHC image Synthetic joint modeling; The geometric velocity field is: ； in, The pixel-level two-dimensional deformation velocity field output by the geometric network, with subscripts... These are the learnable weight parameters of the geometric network; For a pathological slide image, it is a two-dimensional spatial domain representing the set of all pixel coordinates; It is a continuous time interval, representing the complete time evolution process from the initial no deformation to the final complete registration; In a two-dimensional real number space, the x and y axis two-dimensional deformation offsets corresponding to each pixel; The image velocity field is: ； in, The image pixel value change rate field output by the generator network, subscript These are the learnable weight parameters of the generator; It is a three-dimensional real number space, corresponding to the pixel value space of an RGB three-channel digital image; For the image domain The total number of pixels; For the HE image The feature space provides the generator with pathological structural condition inputs. To achieve the HE image To the real IHC image Smooth, wrinkle-free registration, while simultaneously integrating the spatial geometric registration and the virtual IHC image. The synthesis process is deeply coupled, defining a deformation sampling map with a time index. ,in, The two-dimensional pixel coordinate space of the pathological slide image; , is a continuous time variable, representing the complete evolution process from the initial unchanging state to the final fully registered state; Based on the geometric velocity field, the deformation sampling map is defined by the deformation mapping evolution ordinary differential equation. The continuous evolution rule, the ordinary differential equation of the deformation mapping evolution is: ； in, yes Time coordinates The new coordinates after deformation mapping ; for The HE feature map after dynamic registration at any time, its pixel coordinates The eigenvalue at that location is ,in For the HE image A pixel mapping function is used to output the HE image. At the new coordinates The pixel feature values ​​at the location are used to obtain the registered HE features; It is related to the HE image. The paired real IHC images ; As initial conditions, The time-deformation mapping is a unit mapping with no spatial transformation, and the coordinates retain their original values. The semi-Lagrangian composite differential homeomorphic integral unit is used to transform the deformation mapping evolution ordinary differential equation into an iteratively optimized discrete numerical computation scheme, the steps of which include: The continuous time interval Uniformly discretized The iteration step, the... The discrete time points corresponding to each step are: The corresponding discrete time step is ; Based on the discretized time steps, a semi-Lagrangian update is performed through a mapping composition operation to iteratively optimize the deformation mapping. The deformation mapping update formula is as follows: ； in, , These are the deformation sampling maps for the (k+1)th and kth discrete-time steps, respectively; Unit mapping; This is the predicted deformation velocity field after the k-th step is converted from the pixel coordinate system to the normalized coordinate system; This is a mapping compound operator; During the iteration process, the HE feature sequence corresponding to the registration optimization for each deformation mapping step is generated. Its pixel coordinates The eigenvalues ​​at that location satisfy ;in For the HE image The pixel mapping function is used to output the pixel feature value at the corresponding coordinates; These are the pixel coordinates of the image; The number of discrete iteration steps; The CAR virtual staining synthesis subnetwork includes a modified flow matching basic definition unit, a time matching dynamic condition control unit, and a generation loss calculation unit; The modified flow matching basic definition unit is used to construct the true IHC image from Gaussian noise. The linear path probability flow provides path constraints and supervision labels for flow matching training. The steps include: Define the transformation from standard Gaussian noise to the true IHC image. Probability flow of straight path: ； in, The intermediate image stream at time t is the transition from Gaussian noise to the actual IHC image. The transitional state; For continuous time variables, ;in For interval Uniform distribution on This is the preset minimum positive value; Standard Gaussian noise, ;in It follows a multivariate standard normal distribution; Based on the probability flow of the straight path, the analytical solution of the true velocity field corresponding to this path is derived: ； in, The actual velocity field corresponding to this path represents the intermediate image stream. To the real IHC image The ideal speed of evolution; The time-matching dynamic condition control unit is used to progressively register and optimize the HE feature sequence with each iteration step. As a dynamic structural condition for image synthesis, the temporal matching of the flow evolution process with structural cues is achieved, and the steps include: During the discrete iterative training process, the HE feature sequence output by the SGMS geometric registration sub-network corresponding to the k-th iteration is used. As structural condition input, the HE feature map is dynamically registered in the corresponding continuous domain. ; As the iteration step k progresses, the HE feature sequence The registration accuracy is gradually improved, and the structural cues received by the generator G are optimized synchronously. Based on the above coupling architecture, the generator G supports two input modes: when the input is an HE image. At that time, the virtual IHC image is output. When the input is a real IHC image At that time, output the identity reconstruction image. , used for loss calculation in the texture constraint module; The generation loss calculation unit is used to constrain the fit between the image velocity field predicted by the generator G and the real velocity field, and to construct the flow matching loss. : ； in, For time variables and Gaussian noise Find the expected value; It is a function of the image velocity field; The intermediate image stream at time t; The HE feature map at time t; The actual IHC image; It is the square of the L2 norm, i.e., the mean square error.

2. The method for rapidly acquiring virtual IHC staining images during breast surgery according to claim 1, characterized in that, The steps of the modified intraoperative rapid HE staining include: The slices prepared by the frozen sectioning were rinsed with distilled water for 20 seconds. The cell nuclei of the slide were stained with hematoxylin solution for 20 seconds. Rinse the slices with running water for 20 seconds to remove residual hematoxylin solution; The slides were soaked in the blueing solution for 10 seconds to complete the blueing of the cell nuclei. Rinse the slices again with running water for 20 seconds to remove any residual blueing solution. The slide was stained with 0.5% eosin solution for 20 seconds. The slices were soaked in 95% ethanol for 10 seconds to complete the initial dehydration. The sections were soaked in anhydrous ethanol for 10 seconds to complete the secondary dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the three-stage dehydration. The slices were soaked in anhydrous ethanol for 10 seconds to complete the fourth stage of deep dehydration. The slices were soaked in xylene for 10 seconds to achieve initial transparency. The slices were soaked in xylene for 10 seconds to achieve secondary transparency. The slices were soaked in xylene for 10 seconds to achieve three levels of deep transparency. The HE-stained sections are obtained by mounting with distilled water. This step includes: dropping an appropriate amount of distilled water onto the tissue section that has been cleared to a third depth; covering the entire tissue with a coverslip along one side of the tissue to avoid air bubbles; absorbing excess water with absorbent paper; and completing the mounting.

3. The method for rapidly acquiring virtual IHC staining images during breast surgery according to claim 1, characterized in that, The training of the virtual immunohistochemical staining generation model includes the following steps: Several pathological specimens from breast surgeries were collected, and the modified intraoperative rapid HE staining and IHC staining were performed on the same tissue section. After scanning, paired HE images and IHC images to be processed were obtained. The core region of interest (ROI) of the pathological tumor in the HE image and the IHC image to be processed is manually extracted and divided into training, validation, and test sets according to a preset ratio. The training set undergoes data augmentation and standardization preprocessing to obtain the HE image. and real IHC images The dataset; The virtual immunohistochemical staining generation model is constructed, including the generator G, the discriminator D, and the double-fidelity constraint module; The dataset is input into the virtual immunohistochemistry staining generation model, wherein the generator G adopts an encoder-decoder architecture to extract the HE images. and the actual IHC image Based on the multi-level structural features and pathological semantic features, the corresponding virtual IHC image is output. and identity reconstruction images ; The discriminator D employs a convolutional integral class network structure to distinguish the input image from the real IHC image. Or the virtual IHC image The generator G and the discriminator D form an adversarial network architecture. Through adversarial competition and iterative optimization, the generator G is forced to generate the virtual IHC image. Fitting the real IHC image The pixel distribution and overall style; The dual-fidelity constraint module includes a texture constraint module and a pathological information constraint module, wherein the texture constraint module acts on the CAR virtual staining synthesis subnetwork of the generator G to ensure the virtual IHC image With the HE image Structural alignment, and with the actual IHC image The staining style is consistent; the pathological information constraint module acts on the SGMS geometric registration subnetwork of the generator G to lock in core diagnostic features and ensure the virtual IHC image. The structural accuracy and effectiveness of pathological diagnosis.

4. The method for rapidly acquiring virtual IHC staining images during breast surgery according to claim 1, characterized in that: The texture constraint module includes a common parameter support unit, a texture alignment constraint link, and an identity style constraint link; the texture alignment constraint link and the identity style constraint link are two links that are executed in parallel. The common parameter support unit is used to provide a unified feature extraction and mapping basis for the texture alignment constraint link and the identity style constraint link, and the steps include: Define the set of network layers of the generator G. The network layer set It contains multi-layered networks that can capture low-level texture details, mid-level structural features, and high-level semantic information; Build with the generator G Shared multilayer perceptron corresponding to layer features The texture alignment constraint link and the identity style constraint link fully reuse the shared multilayer perceptron. To ensure that the feature mapping space is completely consistent; The texture alignment constraint link is used to ensure the virtual IHC image With the HE image The structural alignment process includes the following steps: A1: Obtain the HE image and the virtual IHC image output by the generator G And extract the network layer set The dimensions corresponding to each network layer are HE image multi-level feature map Multi-level feature maps of virtual IHC images Where C is the number of feature channels, H is the feature map height, and W is the feature map width; A2: In the multi-level feature map of the HE image With the virtual IHC image multi-level feature map At the same coordinate position, synchronous random sampling Each spatial location is used to extract the HE feature vector set corresponding to that spatial location. and virtual IHC feature vector set ; A3: The HE feature vector set and the virtual IHC feature vector set Input the shared multilayer perceptron Feature space mapping is performed to obtain the HE feature set. and virtual IHC feature set ; With each virtual IHC feature As the core, HE features at the same spatial location are set as positive sample HE features. All other HE features not located in the same spatial location are designated as negative sample HE features. ,in ,Right now For anchor point location All sampling locations outside; A4: Constructing the InfoNCE loss function for texture alignment The formula is as follows: ； in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; This represents the temperature coefficient; the superscript T is an abbreviation for "Text," indicating that the loss function focuses on texture alignment. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; The identity style constraint link is used to protect the virtual IHC image. Compared with the real IHC image The coloring style is consistent with the texture alignment constraint link, and the process is executed synchronously and in parallel throughout. The steps include: B1: Obtain the actual IHC image and the identity reconstruction image output by the generator G And extract the network layer set The dimensions corresponding to each network layer are Real IHC image multi-level feature map Multi-level feature maps of identity reconstruction images ; B2: In the multi-level feature map of the real IHC image With the multi-level feature map of the identity reconstruction image At the same coordinate position, synchronous random sampling Each spatial location is used to extract the true IHC feature vector set for that spatial location. Image feature vector set for identity reconstruction ; B3: The actual IHC feature vector set and the identity reconstruction image feature vector set Input the shared multilayer perceptron We perform feature space mapping to obtain the true IHC feature set. Image feature sets for identity reconstruction ; Reconstruct image features for each identity As the core, the true IHC features of the same spatial location are set as the true IHC features of the positive samples. All other true IHC features not from the same spatial location are set as negative sample true IHC features. ,in That is, i represents all sampling positions except for the anchor position k; B4: Constructing the Identity Style InfoNCE Loss Function The formula is as follows: ； in, ; ; ; This represents the normalized cosine similarity between two feature vectors. The set of network layers in the generator G used for feature extraction ; The value represents the temperature coefficient; the superscript I is an abbreviation for "ID", indicating that the loss function focuses on matching identity style with coloring style. The total number of layers in the generator G and The ratio of the number of layers used is used to balance the loss weights of features from different layers; The texture alignment InfoNCE loss function and the aforementioned identity style InfoNCE loss function After the calculation is completed, the total loss is calculated using the texture constraint total loss function. This completes the backpropagation and iterative optimization of the generator G; the total loss function for texture constraints is: ； in , These are weights that can be adjusted based on training results.

5. A system for rapidly acquiring virtual IHC staining images during breast surgery, characterized in that: Used to implement the method described in claim 1.

6. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the steps of the method according to any one of claims 1 to 4 when executing the instructions.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images