Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment, and relates to the technical field of recommendation artificial intelligence. The method comprises the following steps: fusing spatial layout information and context image semantic information of a plurality of full-slice images through a three-dimensional position coding and vector fusion mechanism to obtain a fusion vector; dynamically aggregating the visual lexical elements formed by the fusion vectors into a compressed representation with a fixed length through a learnable lexical element compression mechanism; and inputting the compressed representation into a multi-modal large model to obtain a structured analysis report. According to the method, the technical problems that the calculation amount is too large, spatial information fusion is insufficient, key information is easy to lose, and report accuracy and reliability are poor when a full-slice image is processed in an existing mode can be solved. Therefore, the diagnosis report generated by using the method can meet clinical precise diagnosis requirements.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, storage medium and electronic device. Background Technology

[0002] With the deepening penetration of artificial intelligence in medical diagnosis, pathological diagnosis, as a crucial link in disease confirmation, is increasingly demanding in terms of accuracy and comprehensiveness. Whole-slide images (WSI), as core data for pathological diagnosis, can reach gigabyte (GB) data sizes per image, generating numerous image patches after segmentation. When generating pathology reports based on WSI using a multimodal large-scale model, directly inputting a large number of patches as visual lexical inputs to the model would lead to an exponential increase in computational load, far exceeding the lexical processing capabilities of existing large-scale models and the hardware's computational capacity.

[0003] Current methods for processing whole-slice pathology images (WSIs) generally employ a single-dimensional approach to encoding image patch locations, simple text segmentation for visual lexical processing, and isolated filtering of unrelated visual information. However, single-dimensional image patch location encoding only characterizes the planar position of image patches within a single WSI, failing to effectively integrate the spatial layout relationships across multiple WSIs from the same patient, thus limiting the comprehensiveness of multi-image joint analysis. Simple text segmentation for visual lexical processing lacks differentiation of the importance of pathological information, easily leading to the dilution of key diagnostic information, residual invalid noise, and difficulty in maintaining long-distance dependencies between image patches. Isolated lexical filtering ignores the spatial relationships and contextual semantics between pathological image patches, failing to support clinical scenarios requiring comprehensive multi-regional information assessment, such as neural invasion and cancer depth of invasion. Ultimately, this results in insufficient accuracy and reliability of the generated diagnostic reports, failing to meet the needs of precise clinical diagnosis. Summary of the Invention

[0004] In view of this, this application provides a data processing method, apparatus, storage medium, and electronic device. It aims to solve the technical problems existing in current methods of processing WSI, such as excessive computational load, insufficient fusion of spatial information from multiple WSIs, easy loss of key information, and lack of accuracy and reliability in reports.

[0005] In a first aspect, this application provides a data processing method, including: By using a three-dimensional position encoding and vector fusion mechanism, spatial layout information and contextual image semantic information from multiple full-slice images are fused to obtain a fused vector.

[0006] Through a learnable lexical compression mechanism, visual lexical units composed of fused vectors are dynamically aggregated into a fixed-length compressed representation.

[0007] The compressed representation is input into the multimodal large model to obtain a structured analysis report.

[0008] Secondly, this application provides a data processing apparatus, comprising: The 3D image encoding module is configured to fuse the spatial layout information and contextual image semantic information of multiple full-slice images through a 3D position encoding and vector fusion mechanism to obtain a fused vector.

[0009] The lexical compression module is configured to dynamically aggregate visual lexical units composed of fused vectors into a fixed-length compressed representation through a learnable lexical compression mechanism.

[0010] The prediction module is configured to input compressed representations into a large multimodal model to obtain a structured analysis report.

[0011] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.

[0012] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0013] In view of the above embodiments, this application provides a data processing method, apparatus, storage medium, and electronic device. This application first employs a three-dimensional positional encoding and vector fusion mechanism to deeply fuse the spatial layout association and contextual semantic information of multiple WSIs from the same patient, generating a fusion vector with complete association logic. Subsequently, a learnable lexical compression mechanism is introduced to dynamically aggregate visual lexical units composed of the fusion vector, forming a fixed-length compressed representation. Due to the use of the three-dimensional positional encoding and vector fusion mechanism, the spatial layout logic and contextual semantics of multiple WSIs are deeply associated, integrating scattered image patch information into a holistic representation with both spatial association and semantic coherence. The learnable lexical compression mechanism has adaptive information weight allocation capabilities, assigning higher weights to high-value diagnostic information, thereby effectively suppressing redundant noise information. With the synergistic effect of both, key diagnostic information is accurately preserved, invalid noise is effectively filtered, and the fixed-length aggregation avoids computational redundancy caused by direct input of massive lexical units, significantly reducing the computational load and perfectly adapting to the lexical processing capabilities and hardware computing power of existing large-scale models. At the terminal stage of information processing, the multimodal large model outputs a standardized structured analysis report based on the optimized compressed representation. This report not only accurately supports complex clinical scenarios requiring comprehensive assessment of information from multiple regions, such as nerve invasion and cancer depth, but also significantly improves the reliability and clinical applicability of the report, effectively reducing the burden on pathologists in reading slides and writing reports. It provides an efficient, accurate, and practical technical path for the large-scale and standardized implementation of AI-assisted pathology diagnosis.

[0014] It should be noted that the above content is only a general overview of the technical solution of this application. In order to enable those skilled in the art to clearly understand the core technical means of this application and to accurately implement this solution based on the content disclosed in the specification, the technical details of this application will be described in detail below in conjunction with specific embodiments. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 2This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 3 An example diagram of a full-slice image provided in an embodiment of this application is shown; Figure 4 This illustration shows an example of full-slice image segmentation provided in an embodiment of this application; Figure 5 This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 6 An example diagram of extracting semantic vectors from a whole-slice image provided in an embodiment of this application is shown; Figure 7 An example diagram of vector fusion provided in an embodiment of this application is shown; Figure 8 This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 9 This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application; Figure 10 This illustration shows an example diagram of a visual lexical compression process based on multi-head attention, provided in an embodiment of this application. Figure 11 This illustration shows an example of a system architecture for optimizing a large-scale pathology report generation model that incorporates a visual compression mechanism, as provided in an embodiment of this application. Figure 12 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application is shown. Detailed Implementation

[0018] The following is combined with Figure 1 The flowchart shown illustrates the execution process of the data processing method. This data processing method includes the following steps S101 to S103: S101. By using a three-dimensional position encoding and vector fusion mechanism, the spatial layout information and contextual image semantic information of multiple full-slice images are fused to obtain a fused vector.

[0019] The aforementioned three-dimensional (3D) positional encoding and vector fusion mechanism is a fusion technique that captures the spatial correlation and semantic features of multiple WSIs. Its core is to represent the spatial correlation logic of multiple WSIs in 3D dimension and extract the core pathological features contained in the WSIs through semantic encoding. The fusion technique then generates a fused vector that combines both spatial correlation logic and core pathological features.

[0020] The spatial layout information includes the planar position (x, y dimensions) of each image patch within a single WSI, as well as the sequential position (z dimension) of the WSI to which the image patch belongs in multiple WSIs, comprehensively reflecting the spatial association logic between multiple WSIs.

[0021] The contextual image semantic information refers to the core pathological features contained in each image block, including semantic information directly related to diagnosis such as cell morphology, tissue structure, and lesion area.

[0022] For example, the planar position of an image patch within its respective WSI is first characterized by the x and y dimensions, and then the sequence affiliation of the WSI to which the image patch belongs in multiple images is characterized by the z dimension (e.g., the z value of the image patch in the second WSI is set to 2). Through the collaborative representation of the three-dimensional parameters, a 3D position vector that fully reflects the spatial association of multiple WSIs is formed.

[0023] Next, pathological semantic information such as cell morphology, lesion area, and tissue structure is extracted from WSI, and the pathological semantic information is quantified and transformed into standardized semantic vectors to realize the computable expression of pathological features.

[0024] Finally, the 3D location vector and semantic vector are concatenated dimensionally, so that the fused vector simultaneously carries the spatial correlation logic and the core pathological features. That is, the spatial positional relationship of image patches in multiple WSIs is clearly defined through 3D location encoding and vector fusion mechanism, and key pathological diagnostic information is fully preserved, providing coherent and comprehensive information support for subsequent multi-image joint analysis.

[0025] S102. Through a learnable lexical compression mechanism, visual lexical units composed of fused vectors are dynamically aggregated into a fixed-length compressed representation.

[0026] The learnable lexical compression mechanism is a dynamic aggregation technique based on multi-head attention iteration. Its core is to use learnable compressed lexical units as carriers of information aggregation, engaging in full information interaction with visual lexical units composed of fused vectors. It distinguishes the importance of visual lexical units according to their pathological diagnostic value, adaptively aggregates core pathological semantics and spatial association features, continuously optimizes the information content of compressed lexical units, and ultimately condenses a massive amount of scattered visual lexical units into a fixed-length compressed representation. Specifically: The visual lexicon refers to the basic data unit composed of the fusion vector generated in step S101. Each visual lexicon corresponds to an image patch and carries the 3D spatial location information and pathological semantic features of the patch.

[0027] The learnable compressed lexical unit is a set of trainable vectors of a preset number (e.g., k=640), used to extract key information from massive visual lexical units and dynamically aggregate them.

[0028] The fixed-length compressed representation refers to a set of vectors of length k obtained after multiple rounds of attention iteration optimization. It condenses the core pathological information and spatial associations of multiple WSIs and can directly adapt to the lexical input requirements of multimodal large models (referred to as large models).

[0029] For example, a learnable compression benchmark with a fixed length and matching the dimensions of visual lexical units is first established, allowing it to interact comprehensively with all visual lexical units. Then, by distinguishing the importance of each visual lexical unit, key information such as cell morphology, lesion regions, and spatial correlations across multiple images are aggregated in a weighted manner. Once the information condensation of the learnable compression benchmark reaches a stable state (e.g., the gain of information aggregation is less than a preset threshold, and no new core features are added or updated), it is output as a compressed representation of fixed length.

[0030] S103. Input the compressed representation into the multimodal large model to obtain the structured analysis report.

[0031] The aforementioned multimodal large model is an intelligent model with cross-modal information understanding and generation capabilities. It can simultaneously process visual features and textual semantics, accurately associating pathological image information with clinical diagnostic descriptions. Its core is based on a condensed compressed representation to generate structured pathology reports that conform to clinical standards.

[0032] For example, the fixed-length compressed representation obtained in step S102 is first input into a multimodal large model. The large model, through its pre-trained cross-modal association capabilities, parses the condensed WSI core pathological features and spatial layout associations in the compressed representation. Then, combined with the professional knowledge system of clinical pathology diagnosis, the image features are transformed into precise text descriptions, automatically constructing a structured report containing modules such as diagnostic conclusions, lesion area features, and clinical reference suggestions.

[0033] The data processing method provided in the embodiments of this application will now be described in detail.

[0034] In some scenarios, multimodal large models are optimized models that have undergone joint fine-tuning for classification and report generation. This aims to enhance the model's ability to identify pathological disease categories, improve the accuracy and reliability of structured report generation, and suppress the common "illusion" problem of large models, ensuring that the output results meet clinical diagnostic needs. The "illusion" problem of large models refers to the phenomenon where, during report generation, the model deviates from the actual characteristics of the input pathological image, fabricating non-existent pathological descriptions, diagnostic conclusions, or key indicators. For example, it may fabricate the location of lesions, falsify cell morphology features, or fabricate undetected pathological phenomena, resulting in reports that do not match the actual clinical situation and lose their diagnostic reference value.

[0035] This application also provides an implementation method for training large multimodal models.

[0036] In some embodiments, the training dataset is used to drive the multimodal large model to be trained to perform classification and report generation tasks, and the joint loss is calculated based on the execution results of the classification and report generation tasks. The network parameters of the multimodal large model to be trained are optimized using the joint loss to obtain the multimodal large model.

[0037] The training dataset includes training data, which is obtained in the same way as the compressed representation.

[0038] For example, the execution result of a classification task performed by the multimodal large model to be trained is, for instance, a pathological disease category determination result. The execution result of a report generation task performed by the multimodal large model to be trained is, for instance, a text report containing diagnostic conclusions, lesion feature descriptions, and image correlation analysis.

[0039] For example, in the data preparation stage before training a multimodal large model, the compressed representation is obtained by first performing image block segmentation, 3D position encoding and vector fusion, and learnable word compression on multiple whole slice images (WSI) used for training in the manner described in steps S101~S102 above, and finally obtaining training data (compressed representation), ensuring that the training data is consistent with the data distribution and feature dimensions of the actual application scenario.

[0040] After obtaining the training dataset, it is used to drive the multimodal large model to be trained to perform classification and report generation tasks. For example, the training data is input into the multimodal large model to be trained, yielding disease classification results for corresponding pathological images and generating pathological report text based on the training data. Subsequently, based on the joint loss function, the classification loss between the classification results and the real disease category, and the generation loss between the generated report and the real clinical report are calculated separately. The two types of losses are then weighted and fused using the balance coefficient included in the joint loss function to obtain the joint loss. Using the joint loss as the optimization objective, the network parameters of the multimodal large model to be trained are iteratively updated through backpropagation, continuously optimizing the model's ability to capture pathological features and its cross-modal conversion accuracy. This process continues until the large model's ability to capture pathological features and its cross-modal conversion accuracy meet the requirements, resulting in the optimized multimodal large model.

[0041] In some examples, configurations related to the classification task are removed from the large model, and only configurations related to the report generation task are retained for subsequent output of standardized structured analysis reports based on compressed representations.

[0042] By removing configurations related to classification tasks from the large model, redundant functions are simplified and the operational load is reduced, allowing the large model to focus on the accurate conversion of pathological image features into clinical diagnostic text. Simultaneously, leveraging the disease category recognition capabilities accumulated during training, key knowledge point constraints are provided for report generation, effectively reducing unfounded false descriptions and significantly suppressing the "illusion" problem of the large model. This ensures that the output structured analysis report not only conforms to clinical standards but also fully restores the core pathological information and spatial correlation logic of multiple whole-slice images, improving the accuracy and reliability of the report.

[0043] In some embodiments, the joint loss function includes a first loss function, a second loss function, and a balance coefficient. The first loss function is a cross-entropy loss function, used to accurately measure the difference between the classification result and the actual disease category, adapting to multi-class or binary classification scenarios in pathological diagnosis. The second loss function is a cross-entropy loss function or a negative log-likelihood loss function, used to quantify the semantic consistency between the generated report and the actual clinical report. By calculating the degree of matching between the predicted probability of each word in the generated text and the actual word, it optimizes the accuracy and standardization of report generation. The balance coefficient can be set to a value between 0 and 1 (e.g., 0.3, 0.5, etc.), used to dynamically adjust the weight ratio of the two types of losses. For example, when it is necessary to enhance the model's ability to identify disease categories, the weight of the first loss function can be appropriately increased. When it is necessary to prioritize the clinical adaptability of the report text, the weight of the second loss function can be increased to ensure that the model simultaneously considers classification accuracy and report generation quality.

[0044] In some implementations, the joint loss function is expressed as follows:

[0045] Where L is the joint loss, which is used to comprehensively measure the overall performance of the model in classification and report generation tasks.

[0046] L1 is the classification loss, used to quantify the difference between the disease classification results output by the model and the true disease categories. It is calculated using the cross-entropy loss function.

[0047] L2 is the generation loss, used to measure the semantic consistency between the pathology reports generated by the model and the real clinical reports. It is calculated using the cross-entropy loss function or the negative log-likelihood loss function.

[0048] λ is a balancing coefficient, ranging from 0 to 1 (e.g., 0.3, 0.5, etc.), used to dynamically adjust the weight ratio of classification loss and generation loss in the joint loss. It can be flexibly set according to actual training needs. When it is necessary to enhance the model's disease category recognition ability, the value of λ can be increased; when it is necessary to prioritize the accuracy and standardization of report generation, the value of λ can be decreased to ensure that the model simultaneously considers the optimization effects of both types of tasks.

[0049] In some embodiments, this application also provides an implementation method that uses a training dataset to drive a large multimodal model to be trained to perform classification and report generation tasks, and calculates the joint loss based on the execution results of the classification and report generation tasks using a joint loss function. Figure 2 As shown, the process includes the following steps S201 to S203: S201. Input the training data into the classification branch of the multimodal large model to be trained, obtain the classification result, and calculate the classification loss based on the first loss function.

[0050] Among them, the classification branch of the multimodal large model to be trained is a functional module specifically used for pathological disease category determination. Its core is to output the disease category label of the corresponding image based on the pathological features condensed in the training data (compressed representation).

[0051] For example, first determine the real disease category corresponding to the training data, and then quantify the deviation between the classification result output by the model and the real category through the first loss function. The larger the deviation, the higher the classification loss value, and vice versa.

[0052] In some implementations, the expression for the first loss function L1 is as follows:

[0053] Where N is the total number of samples (training data) in the training data; y i Let i be the true class label of the i-th sample; This is the classification result.

[0054] S202. Input the training data into the report generation branch of the multimodal large model to be trained, obtain the generation result, and calculate the generation loss based on the second loss function.

[0055] Among them, the report generation branch of the multimodal large model to be trained is a functional module focused on the generation of pathological text reports. It can output text reports that conform to clinical standards (such as complete report content including lesion description, diagnosis conclusion, and feature analysis) based on the pathological features and spatial correlation information in the training data.

[0056] For example, using real clinical pathology reports as a reference standard, the semantic differences between the model-generated report and the real report are quantified word-by-word by a second loss function. For instance, for each word in the generated report, the model calculates the probability predicted by that word and the degree of matching between that word and the real word. The differences of all words are accumulated to form the final generation loss. The lower the degree of matching between the generated report and the real report, the higher the generation loss value.

[0057] In some implementations, the expression for the second loss function L2 is as follows:

[0058] Where T is the text length (number of lexical units) of the actual pathology report; w t The t-th term in the actual report; Given the context of the first t-1 words, the model predicts the t-th word as w. t The probability of.

[0059] S203. The joint loss is obtained by balancing the classification loss and the generation loss using the weighted average of the balancing coefficients.

[0060] For example, the balance coefficient λ has been introduced in the above example and will not be elaborated on further here. By weighting the classification loss L1 with λ, the generation loss L2 with 1-λ, and fusing the weighted classification loss and generation loss, the joint loss L is obtained.

[0061] In some embodiments, after optimizing the network parameters of the multimodal large model to be trained using joint loss to obtain the trained multimodal large model, the classification branch of the multimodal large model is removed. The classification branch provides disease category constraints during the joint fine-tuning stage, helping the model to enhance the accurate capture of pathological features, thereby suppressing the "illusion" problem in report generation. In actual deployment for generating structured analysis reports, only the report generation-related functions need to be retained to output pathology reports conforming to clinical standards based on compressed representation, ensuring both report generation efficiency and the accuracy and reliability of the output results.

[0062] After obtaining the trained multimodal large model, this application also provides an implementation method for data preprocessing during the stage of generating a structured analysis report of multiple whole slice images (WSI) in actual clinical scenarios.

[0063] In some embodiments, multiple full-slice images of the same object are divided into image blocks to obtain multiple image blocks.

[0064] like Figure 3 The image shown is an example of a whole-slice image. It presents the complete view of a single pathological whole-slice of an object, which includes different pathological information such as lesion areas and normal tissues.

[0065] When segmenting image blocks, a fixed-size sliding window (e.g., 512×512 pixels) can be used to uniformly divide the image. During segmentation, the window should slide continuously while preserving a small overlap between adjacent blocks (e.g., 10% overlap) to avoid losing edge pathological information. For areas where the image edge is smaller than the window size, zero-padding is used to bring it to the standard size. Finally, a single full-slice image is decomposed into multiple image blocks of uniform size.

[0066] like Figure 4 As shown, there are two full-slice images, 1 and 2, which are divided into several image blocks after image block segmentation.

[0067] After dividing multiple full-slice images into image blocks to obtain multiple image blocks, this application also provides an implementation method that uses a 3D position encoding and vector fusion mechanism to fuse the spatial layout information and contextual image semantic information of multiple full-slice images to obtain a fused vector. For example... Figure 5 As shown, this embodiment includes the following steps S301 to S303: S301. Perform three-dimensional position encoding on each image block to obtain a position vector representing the spatial position of the image block in multiple full-slice images.

[0068] For example, for each image patch, its three-dimensional spatial location parameters (x, y, z) are first defined. The x and y parameters correspond to the horizontal and vertical positions of the image patch within its respective whole-slice image (WSI). The z parameter corresponds to the sequence number of the WSI to which the image patch belongs among multiple WSIs (e.g., based on clinical sampling order or slice depth, the z value of the 3rd WSI is set to 3). Then, the location parameters in the x, y, and z dimensions are encoded using a preset encoding rule to obtain the corresponding encoded components for each dimension. Finally, the three encoded components are fused according to a preset method to generate the 3D location vector Di of the image patch. This vector Di represents the planar position of the image patch within a single WSI and its sequence association among multiple WSIs, providing accurate spatial location support for subsequent multi-WSI joint analysis.

[0069] S302. Semantically encode each image block to obtain a semantic vector representing the semantic features of the image block.

[0070] For example, a dedicated image patch coding model (such as ViT or ResNet series models) pre-trained on pathological image data is used to extract semantics from each image patch. The image patch coding model extracts pathologically relevant semantic information layer by layer from the image patch through a multi-scale feature capture mechanism. Specifically, the bottom layer of the image patch coding model captures detailed features such as cell edges and staining intensity. The middle layer extracts local features such as cell arrangement patterns and glandular structures. The top layer condenses core diagnostic features such as lesion areas and abnormal tissues. Then, through the fully connected layers and feature normalization processing of the image patch coding model, this abstract pathological semantic information is transformed into a standardized vector with a fixed dimension (consistent with the dimension of the 3D position vector), namely the semantic vector Vi, giving the pathological features of the image patch computable and fusionable attributes. Figure 6 As shown, after segmenting multiple full-slice images into image blocks, the image blocks are input into the image block coding model. The image block coding model outputs a semantic vector Vi (e.g., ...). Figure 6 (V1~V24 shown).

[0071] S303. The position vector and semantic vector are concatenated and fused to obtain a fused vector that integrates 3D position information and contextual semantic information.

[0072] For example, for the same image patch, the location vector Di and semantic vector Vi are fused using a dimension-aligned direct concatenation method. The semantic vector Vi is used as the first half, and the location vector Di as the second half, concatenated in dimensional order to form a fusion vector Pi=[Vi,Di] with a dimension of 2d (e.g., 1024 dimensions). This fusion vector Pi retains the core pathological features carried by the semantic vector Vi (such as tumor cell morphology and lesion area distribution) while integrating the spatial correlation information represented by the location vector Di (such as cross-WSI sequence and single-WSI intraplane location), achieving a deep binding between pathological semantics and spatial location. This provides a complete and relevant foundational data for subsequent learnable lexical compression. Figure 7 As shown, after segmenting multiple full-slice images into image blocks, the image blocks are input into the image block coding model and the 3D position coding model respectively to obtain the semantic vector Vi and the position vector Di (e.g., ...). Figure 7 (D1~D16 shown). Subsequently, the semantic vector Vi and the position vector Di are concatenated to obtain the fused vector Pi.

[0073] This application also provides an implementation method that performs 3D position encoding on each image patch to obtain a position vector representing the spatial position of the image patch in multiple full-slice images. For example... Figure 8 As shown, the steps include S401 to S403 as follows: S401. Determine the three-dimensional position parameters of each image block.

[0074] The three-dimensional position parameters include a first position parameter and a second position parameter characterizing the planar position of the image patch within its corresponding whole-slice image, and a third position parameter characterizing the sequential position of the whole-slice image to which the image patch belongs among multiple whole-slice images. For example, the first position parameter is the horizontal pixel coordinate (x) described in the previous example. The second position parameter is the vertical pixel coordinate (y) described in the previous example. The third position parameter is the whole-slice image sequence number (z).

[0075] For example, taking the top-left pixel of the WSI to which the image patch belongs as the origin of the coordinate system, and setting the horizontal direction to the right as the positive x-axis, the top-left pixel of the image patch is selected as the position reference. The pixel number of this reference point in the horizontal direction of the WSI is read by an image processing tool (such as OpenCV or PIL), which is the first position parameter x. For example, if the pixel number corresponding to the top-left corner of an image patch in the horizontal direction of the WSI is 800, then x = 800, accurately locking its horizontal position within the WSI.

[0076] For example, reusing the aforementioned WSI planar coordinate system, with the top-left pixel (0,0) as the origin, and setting the vertical downward direction as the positive y-axis, the top-left pixel of the image block is used as the position reference. The pixel number of this reference point in the WSI vertical direction is read, which is the second position parameter y. For example, if the pixel number corresponding to the top-left corner of the image block in the WSI vertical direction is 600, then y=600, which, together with x=800, determines the unique planar position (800, 600) of the image block within its WSI.

[0077] For example, multiple WSIs of the same subject are sorted according to clinical diagnostic logic (such as sampling site priority, slice depth, and detection time order), and each sorted WSI is assigned a unique integer sequence number (incrementing from 1). This sequence number is the third positional parameter z of all image patches contained in it. For instance, if three WSIs of the same patient are sorted as "lesion center slice - lesion edge slice - normal tissue slice", with corresponding z values ​​of 1, 2, and 3 respectively, then the z parameters of all image patches in the three WSIs are 1, 2, and 3 respectively, clearly defining the cross-WSI affiliation of image patches.

[0078] S402. Perform cosine coding on the three-dimensional position parameters of each image block to obtain the coded components in three spatial dimensions.

[0079] For example, cosine coding is performed on the three-dimensional position parameters (x, y, z, corresponding to the horizontal, vertical, and WSI sequence positions of each image patch, respectively). By combining sine and cosine functions, the discrete position parameters are transformed into continuous and distinguishable vector representations, ultimately obtaining the coded components (Di,x, Di,y, Di,z) corresponding to the three spatial dimensions of x, y, and z.

[0080] S403. Add the encoded components of the three spatial dimensions one by one according to the index of the position vector to obtain the position vector of the image block in the spatial location of multiple full-slice images.

[0081] Specifically, the fusion of three-dimensional position information is achieved through the indexing of the position vector to obtain a 3D position vector. The index represents the sequence number of the element in the encoded component (j=0,1,2,...,d-1). The index provides an alignment reference for the encoded components in the x, y, and z dimensions, ensuring that elements corresponding to the same index j respectively carry position information in the same feature dimension of the three dimensions, avoiding confusion of position information in different dimensions, and laying the foundation for the effective fusion of three-dimensional information.

[0082] For example, for each index j, the element values ​​of the three encoded components Di,x,Di,y, and Di,z at that index are summed to obtain the element value of the position vector at that index. After traversing all indices and summing them one by one, the resulting vector is the 3D position vector of the image patch. This 3D position vector simultaneously carries the planar position of the image patch within a single WSI and the sequence association between multiple WSIs, providing accurate spatial location support for subsequent fusion with semantic vectors and joint analysis of multiple WSIs.

[0083] This application also provides an implementation method for cosine coding of the three-dimensional position parameters of each image block to obtain coded components in three spatial dimensions.

[0084] This embodiment first determines the fixed dimensions of the position vector.

[0085] The fixed dimension represents the range of values ​​for the index of the position vector, which is used to unify the length of the three spatial dimension encoded components and ensure the compatibility of subsequent vector fusion.

[0086] For example, combining the commonly used dimensions of semantic coding of pathological images and the input adaptation requirements of multimodal large models, the fixed dimension of the position vector is set to d (such as 512-dimensional or 1024-dimensional). Then the index value range of the position vector is j=0,1,2,...,d-1, that is, each encoded component contains d elements, providing a unified quantization carrier for the position information of the three dimensions.

[0087] For any one of the first, second, and third position parameters of each image block, perform the following encoding operations S501~S502: S501. For each index of the position vector, if the index is even, the position parameter is encoded using a sine function to obtain the encoded value of the index.

[0088] For example, for each index j of the position vector, if index j is even, a sine function is used to encode the position parameter to obtain the encoded value of the index. The sine function transforms discrete position parameters into continuous vector elements, while a scaling factor is introduced to balance the numerical range under different indices, ensuring that positional differences are accurately captured.

[0089] S502. If the index is odd, the position parameter is encoded using a cosine function to obtain the encoded value of the index.

[0090] For example, if the index of the position vector is odd, the position parameter is encoded using a cosine function to obtain the encoded value of the index. The core is to form a complementary encoding logic by using the cosine function and the sine function of S501 to further enhance the distinguishability of different position parameters. At the same time, the scaling factor is used to ensure the reasonable distribution of the encoded value and avoid feature confusion caused by single function encoding.

[0091] In some implementations, taking the encoding operation of the first position parameter of any image block as an example, the expression of its encoded component Di,x is as follows:

[0092] Where index j is even, the sin value is used as the encoding value for the corresponding index. When index j is odd, the cosine value is used as the encoding value for the corresponding index. 10000 2j / d This is a scaling factor used to balance the range of values ​​across different indices.

[0093] Similarly, the encoding operation on the second positional parameter yields the following expression for its encoded component Di,y:

[0094] The calculation process is similar to obtaining the expression for Di,x as described above, and will not be explained in detail here.

[0095] The encoding operation on the third positional parameter, and the expression for its encoded component Di,z, are shown below:

[0096] The calculation process is similar to obtaining the expression for Di,x as described above, and will not be explained in detail here.

[0097] Through encoding operations, the coded components of the three spatial dimensions of each image patch are obtained.

[0098] After encoding each dimension, the three encoded components are added one by one according to the index of the position vector to obtain the final 3D position encoding vector Di of the image patch.

[0099] The expression for its 3D position encoding vector Di is as follows:

[0100] After obtaining the fusion vector through the above embodiments, this application also provides an implementation method that dynamically aggregates visual lexical units composed of the fusion vector into a fixed-length compressed representation through a learnable lexical compression mechanism. For example... Figure 9 As shown, the steps include S601 to S604 as follows: S601. Determine a set of pre-defined numbers of learnable compressible lexical units.

[0101] For example, first, a set of learnable compressible lexical units, C = [c1, c2, ..., ck], is determined. Here, k is the fixed number of lexical units after compression (e.g., 640, 1024, etc., which needs to be adapted to the lexical processing capability and hardware computing power of the multimodal large model). This set of learnable compressible lexical units, C, is a set of trainable vectors. Initially, it can be generated using random initialization, and its vector dimension is consistent with the dimension of the fused vector to ensure compatibility with subsequent attention mechanism calculations. The set of learnable compressible lexical units, C, acts as a "query carrier" throughout the compression process. Through iterative learning, it extracts key pathological semantics and spatial association information from massive visual lexical units, ultimately forming a fixed-length compressed representation with information condensation capabilities.

[0102] S602. Connect the visual word set composed of fused vectors with the compressed word set to form a joint sequence.

[0103] For example, the visual word set P=[p1,p2,.......pm] obtained by 3D encoding multiple full-slice images of the same object (such as the same patient) (where m is the original number of visual words, i.e., the total number of image blocks after segmentation, and m) k) is connected sequentially with the compressed word set C determined in step S601 to form a joint sequence T=[C,P].

[0104] S603: Using the joint sequence as input, perform multiple rounds of iterative computation through a multi-head attention mechanism.

[0105] For example, the joint sequence T is used as input, the compressed word set C is used as the query vector matrix Q, and the joint sequence T is used as the key vector matrix K and the value vector matrix V, respectively. Multiple rounds of iterative computation are performed through a multi-head attention mechanism. Specifically, firstly, linear projection processing is performed on Q, K, and V to map them to feature dimensions suitable for multi-head attention computation. Then, they are split into multiple independent sub-vector matrices, each corresponding to an attention head, thereby achieving parallel capture of multi-dimensional information such as pathological semantic features and 3D spatial correlation features. Next, for each attention head, the similarity score between the query sub-vector and the key sub-vector is calculated. After normalization using the softmax function, the contribution weight of each original visual word to the compressed word is obtained, thus distinguishing key diagnostic information from redundant background information. Subsequently, the value sub-vectors are weighted and summed based on the normalized weights to complete the information aggregation of a single attention head. Then, the aggregation results of all attention heads are concatenated and feature fusion is performed through a linear transformation layer to generate the parameter update vector for this round of iteration. Finally, this parameter is used to update the vector and adjust the weight parameters of the compressed word set C, so that C gradually absorbs the core value information of high-weight visual words.

[0106] Each iteration uses the adjusted compressed word set C as the new query vector matrix Q, repeating the attention interaction and parameter update process described above to continuously optimize the information aggregation effect of compressed words.

[0107] S604. If the iteration reaches a preset number of iterations or the update magnitude of the weight parameters of the compressed tokens is less than a preset threshold, stop the iteration and obtain a compressed representation of a fixed length.

[0108] For example, during multiple iterations, two termination conditions are monitored in real time: first, whether the number of iterations reaches a preset threshold (e.g., 10 or 15 rounds, which can be adjusted according to model training efficiency and information aggregation requirements); and second, whether the update magnitude of the weight parameters of the compressed lexical set C is less than a preset threshold (e.g., the L2 norm of the parameter change is less than 1e-6). When either of these conditions is met, the iteration stops, resulting in a fixed-length compressed representation.

[0109] If the preset number of iterations is reached, it indicates that the compressed lexical units have completed sufficient information interaction and aggregation. If the update magnitude of the weight parameters is lower than the threshold, it indicates that the information content of the compressed lexical units tends to stabilize, and further iterations cannot significantly improve the aggregation effect. At this point, the compressed lexical unit set C is a fixed-length compressed representation of the original massive visual lexical units. This compressed representation not only condenses the core pathological semantics and 3D spatial association information, but also significantly reduces the computational cost of subsequent model processing.

[0110] In some implementations, the joint sequence T is used as input, and multiple rounds of iterative computation are performed through a multi-head attention mechanism, as shown in the following expression:

[0111] in, , represents a compressed representation of a fixed length obtained by satisfying the termination condition; Q=C indicates that the query vector is the compressed word set C of the current iteration; K=(C,P) represents the key vector as a joint sequence consisting of the compressed word set C of the current iteration and the original visual word set P. It is used to calculate similarity with the query vector to filter key information.

[0112] V=(C,P) represents a value vector consisting of a joint sequence T formed by the current iteration's compressed word set C and the original visual word set P, used to aggregate information based on attention weights.

[0113] m represents the number of attention heads in a multi-head attention mechanism.

[0114] W o , representing the learnable linear transformation parameters used to fuse the outputs of multiple attention heads.

[0115] Concat represents the concatenation operation, used to merge the outputs of multiple attention heads into a single vector.

[0116] head i , represents the calculation result of the i-th attention head, and each attention head corresponds to a set of independent information capture dimensions.

[0117] The above formula extracts key information from the joint sequence T of the current iteration through multi-head attention computation, and then performs a residual connection + C with the original compressed tokens. This updates the compressed tokens while avoiding the loss of already aggregated information.

[0118] For any attention head, its expression is as follows:

[0119] in, , represents the learnable projection parameter of the query vector corresponding to the i-th attention head; , represents the learnable projection parameter of the key vector corresponding to the i-th attention head; , represents the learnable projection parameter of the value vector corresponding to the i-th attention head; T represents a joint sequence; , represents the scaling factor, used to alleviate the softmax gradient vanishing problem caused by excessive attention scores; Softmax represents normalizing the attention score to obtain the contribution weight of visual words to compressed words, thereby enabling the filtering of key information.

[0120] The above formula maps query, key, and value vectors to the adaptation dimension through learnable projection parameters, calculates and normalizes attention scores, and finally aggregates value vector information based on weights. This process enables a single attention head to specifically capture key information in the joint sequence, laying the foundation for the subsequent fusion of results from multiple attention heads. The above formula allows each attention head to independently focus on a specific dimension of information in the joint sequence (such as the morphology of lesion areas in pathological images, 3D spatial relationships, etc.). Attention weights are used to filter out visual word information that contributes significantly to the compression of word units, thereby achieving the information condensation of the original visual word units.

[0121] like Figure 10 As shown, an example of a visual lexical compression process based on multi-head attention is introduced. Using a joint sequence T as input, visual lexical units P, composed of fused vectors, are dynamically condensed through learnable compressible lexical units.

[0122] The joint sequence T is used as the key vector matrix K and the value vector matrix V, respectively, and the compressed word set C is used as the query vector matrix Q, which are input into the multi-head attention module. The multi-head attention module first calculates the similarity score between the query and the key, and then normalizes the similarity score using the softmax function to obtain the attention weights of visual words to compressed words. Based on the normalized attention weights, the value vectors are weighted and summed to extract key pathological and 3D spatial information from the visual words, and the adjusted compressed word set is output.

[0123] Subsequently, the adjusted compressed word set will be used as the query vector matrix for the next iteration, and will again undergo multi-head attention interaction with the joint sequence T. Through multiple iterations, the compressed words gradually absorb core value information, ultimately forming a fixed-length compressed representation C. new .

[0124] This application also provides an embodiment that uses a joint sequence as input and performs multi-round iterative computation through a multi-head attention mechanism. In any round of iterative computation, this embodiment uses the learnable compressible lexicon set as the query vector matrix and the joint sequence as the key vector matrix and value vector matrix, and dynamically adjusts the weight parameters of the compressible lexicon set in the following manner: S701. Perform linear projection on the query vector matrix, key vector matrix, and value vector matrix and split them into multiple sets of sub-vector matrices.

[0125] Each set of subvector matrices corresponds to an independent attention head in a multi-head attention mechanism.

[0126] For example, through learnable projection parameters , , Dimension mapping is performed on the three types of vector matrices (query vector Q, key vector K, and value vector V) to adapt them to the computational requirements of multi-head attention. Then, the matrices are split according to the preset number of attention heads, forming multiple sets of parallel sub-vector matrices.

[0127] S702. For each set of subvector matrices, calculate the similarity score between the query vector in the subquery matrix and the key vector in the subkey matrix, and normalize the similarity score to obtain the attention weight of visual lexical units to compressed lexical units.

[0128] Attention weights characterize the degree of contribution of different visual lexical units to compressed lexical units.

[0129] For example, the similarity score between each group of subqueries and subkey vectors is calculated by dot product operation, and then the similarity score is normalized by the softmax function to obtain the attention weight with a value in the range of [0,1]. The higher the weight, the greater the aggregation value of the pathological semantics and 3D spatial information contained in the corresponding visual word for the compressed word.

[0130] S703. Based on the attention weights, the value vectors in the sub-value matrix are weighted and summed to obtain the information aggregation result of each independent attention head.

[0131] For example, each value vector in the sub-value matrix is ​​multiplied by the attention weight of the corresponding visual word, and then all weighted value vectors are summed to achieve the key aggregation of high-contribution information and obtain the target information captured by a single attention head.

[0132] S704. The aggregation results of all independent attention heads are concatenated and fused through a linear transformation layer to generate the parameter update vector for this round of iteration.

[0133] For example, the aggregation results of multiple attention heads are first concatenated into a unified vector according to their dimensions, and then the vector is transformed by a learnable linear transformation parameter W. o Feature fusion is performed to integrate multi-dimensional pathological and spatial information to generate vectors that can guide the updating of compressed lexical parameters.

[0134] S705. Adjust the weight parameters of the compressed word set using the parameter update vector to obtain the adjusted compressed word set.

[0135] For example, the generated parameter update vector is compared with the weight parameter W of the current compressed lexical set. o By superimposing these elements, the weights of compressed word units can be dynamically adjusted, allowing the compressed word units to gradually absorb the core information from the visual word units.

[0136] In some implementations, referring to the above embodiments, this application also introduces a large-scale optimization system for pathology report generation that integrates a visual compression mechanism. The system architecture is as follows: Figure 11 As shown, it includes a data preprocessing module, a 3D image encoding module, a lexical compression module, a joint fine-tuning module, and a report generation module. The implementation logic of each module is as follows: The data preprocessing module divides multiple whole-slice pathology (WSI) images of the same patient into several image blocks using the image block segmentation module, preparing them for subsequent encoding.

[0137] The 3D image encoding module first performs 3D position encoding (planar position + WSI sequence number) on each image block to obtain a position vector Di. Then, the pathological image encoder encodes the image block into a semantic vector Vi. Finally, the position vector Di and the semantic vector Vi are concatenated to obtain a fused vector Pi containing positional information.

[0138] The lexical compression module introduces a learnable set of compressible lexical units C, which is concatenated with a visual lexical unit set P formed by a fusion vector Pi to form a joint sequence T. Using C as the query vector, the weight parameters of C are adjusted through multiple rounds of multi-head attention to ultimately obtain the compressed representation Ct. new .

[0139] The joint fine-tuning module constructs a joint loss function L, which includes classification loss L1 and generation loss L2, during the training phase of the multimodal large model. The multimodal large model is then fine-tuned using the joint loss function L to obtain a well-trained multimodal large model.

[0140] In the report generation module, after the multimodal large model has been trained, its classification branch is removed. During the inference phase, the report generation branch of the large model is used to generate pathology reports.

[0141] Furthermore, this embodiment provides a data processing device, such as... Figure 12 As shown, the device includes a three-dimensional image encoding module 1210, a word compression module 1220, and a prediction module 1230, wherein: The 3D image encoding module 1210 is configured to fuse the spatial layout information and contextual image semantic information of multiple full-slice images through a 3D position encoding and vector fusion mechanism to obtain a fused vector. The lexical compression module 1220 is configured to dynamically aggregate visual lexical units composed of fused vectors into a fixed-length compressed representation through a learnable lexical compression mechanism. Prediction module 1230 is configured to input compressed representations into a multimodal large model to obtain a structured analysis report.

[0142] In some embodiments, the apparatus further includes a joint fine-tuning module configured to drive the multimodal large model to be trained to perform classification and report generation tasks using the training dataset, and to calculate the joint loss based on the execution results of the classification and report generation tasks using the joint loss function; wherein the training dataset includes training data, and the training data is obtained in the same way as the compressed representation.

[0143] The network parameters of the multimodal large model to be trained are optimized using joint loss to obtain the multimodal large model.

[0144] In some embodiments, the joint fine-tuning module is further configured to input training data into the classification branch of the multimodal large model to be trained, obtain classification results, and calculate classification loss based on a first loss function; the classification loss is used to measure the difference between the classification results and the true category.

[0145] The training data is input into the report generation branch of the multimodal large model to be trained to obtain the generation result, and the generation loss is calculated based on the second loss function; the generation loss is used to measure the difference between the generated result and the real result.

[0146] The joint loss is obtained by balancing the classification loss and generation loss using a weighted average of the balancing coefficients.

[0147] In some embodiments, the joint fine-tuning module is also configured to remove the classification branch of the multimodal large model.

[0148] In some embodiments, the apparatus further includes a data preprocessing module configured to perform image block segmentation on multiple full-slice images of the same object to obtain multiple image blocks.

[0149] In some embodiments, the three-dimensional image encoding module 1210 is further configured to perform 3D position encoding on each image block to obtain a position vector representing the spatial position of the image block in multiple full-slice images.

[0150] Semantic encoding is performed on each image patch to obtain a semantic vector representing the semantic features of the image patch.

[0151] The position vector and the semantic vector are concatenated and fused to obtain a fused vector that integrates 3D position information and contextual semantic information.

[0152] In some embodiments, the three-dimensional image encoding module 1210 is further configured to determine three-dimensional position parameters for each image block; the three-dimensional position parameters include a first position parameter and a second position parameter characterizing the planar position of the image block in its belonging whole slice image, and a third position parameter characterizing the sequential position of the whole slice image to which the image block belongs in multiple whole slice images.

[0153] Cosine coding is performed on the three-dimensional position parameters of each image block to obtain coded components in three spatial dimensions.

[0154] The encoded components of the three spatial dimensions are added one by one according to the index of the position vector to obtain the position vector of the image patch in the spatial location of multiple full-slice images.

[0155] In some embodiments, the three-dimensional image encoding module 1210 is further configured to determine a fixed dimension of the position vector; the fixed dimension represents the range of values ​​for the index of the position vector.

[0156] For any one of the first, second, and third position parameters of each image block, perform the following encoding operation: For each index of the position vector, if the index is even, the position parameter is encoded using a sine function to obtain the encoded value of the index.

[0157] If the index is odd, the position parameter is encoded using a cosine function to obtain the encoded value of the index.

[0158] Through encoding operations, the coded components of the three spatial dimensions of each image patch are obtained.

[0159] In some embodiments, the lexical compression module 1220 is further configured to determine a preset number of learnable compressible lexical sets.

[0160] The visual word set composed of fused vectors is connected with the compressed word set to form a joint sequence.

[0161] Using the joint sequence as input, multiple rounds of iterative computation are performed through a multi-head attention mechanism.

[0162] The iteration stops when the preset number of iterations is reached or the update magnitude of the weight parameters of the compressed tokens is less than the preset threshold, resulting in a compressed representation of fixed length.

[0163] In some embodiments, the lexical compression module 1220 is further configured to dynamically adjust the weight parameters of the compressed lexical set in any round of iterative computation, using the learnable compressible lexical set as the query vector matrix and the joint sequence as the key vector matrix and value vector matrix, in the following manner.

[0164] The query vector matrix, key vector matrix, and value vector matrix are linearly projected and split into multiple sets of sub-vector matrices; each set of sub-vector matrices corresponds to an independent attention head in the multi-head attention mechanism.

[0165] For each set of subvector matrices, the similarity score between the query vector in the subquery matrix and the key vector in the subkey matrix is ​​calculated, and the similarity score is normalized to obtain the attention weight of visual words to compressed words; the attention weight represents the degree of contribution of different visual words to compressed words.

[0166] The information aggregation result of each independent attention head is obtained by weighted summation of the value vectors in the sub-value matrix based on the attention weights.

[0167] The aggregation results of all independent attention heads are concatenated and fused through a linear transformation layer to generate the parameter update vector for this round of iteration.

[0168] The weight parameters of the compressed lexical set are adjusted using the parameter update vector to obtain the adjusted compressed lexical set; the adjusted compressed lexical set is used for the next iteration.

[0169] It should be noted that other corresponding descriptions of the functional units involved in the data processing apparatus provided in this embodiment can be found in the description of the data processing method in the above embodiments, and will not be repeated here.

[0170] Based on the data processing method shown in the above embodiments, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method shown in the above embodiments.

[0171] Based on the methods shown in the above embodiments, this embodiment also provides a computer program product on which a computer program is stored, and when the computer program product is executed by a processor, it implements the methods shown in the above embodiments.

[0172] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, portable hard drive, etc.). The storage medium includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0173] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0174] Based on the method shown in the above embodiments, and Figure 12To achieve the above objectives, as illustrated in the virtual device embodiment, this application also provides an electronic device. This electronic device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to implement the method shown in the above embodiment.

[0175] Optionally, the aforementioned electronic device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optionally, a USB interface, a card reader interface, etc. Optionally, the network interface may include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0176] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0177] Through the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. Compared with the current related technologies, this application first adopts a 3D position encoding and vector fusion mechanism to deeply fuse the spatial layout association and contextual semantic information of multiple WSIs of the same patient, generating a fusion vector with complete association logic. Subsequently, a learnable lexical compression mechanism is introduced to dynamically aggregate the visual lexicals composed of the fusion vector to form a fixed-length compressed representation. Due to the adoption of the 3D position encoding and vector fusion mechanism, the spatial layout logic and contextual semantics of multiple WSIs are deeply associated, so that the scattered image block information is integrated into a whole representation with both spatial association and semantic coherence. The learnable lexical compression mechanism has the ability to adaptively allocate information weights, which can give higher weights to high-value diagnostic information, thereby effectively suppressing redundant noise information. With the synergistic effect of the two, key diagnostic information is accurately preserved and invalid noise is effectively filtered out. At the same time, the fixed-length aggregation avoids the computational redundancy caused by the direct input of massive lexicals, greatly reducing the computational load and perfectly adapting to the lexical processing capabilities and hardware computing power of existing large models. At the terminal stage of information processing, the multimodal large model outputs a standardized structured analysis report based on the optimized compressed representation. This report not only accurately supports complex clinical scenarios requiring comprehensive judgment of information from multiple regions, such as neural invasion and cancer invasion depth, but also significantly improves the reliability and clinical applicability of the report, effectively reducing the burden on pathologists in reading slides and writing reports. It provides an efficient, accurate, and practical technical path for the large-scale and standardized implementation of AI-assisted pathology diagnosis.

[0178] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0179] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method, characterized in that, include: By using a three-dimensional position encoding and vector fusion mechanism, spatial layout information and contextual image semantic information from multiple full-slice images are fused to obtain a fused vector; Through a learnable lexical compression mechanism, the visual lexical units composed of the fused vectors are dynamically aggregated into a compressed representation of fixed length. The compressed representation is input into the multimodal large model to obtain a structured analysis report.

2. The method according to claim 1, characterized in that, The method further includes: The training dataset is used to drive the multimodal large model to be trained to perform classification and report generation tasks, and the joint loss is calculated based on the execution results of the classification and report generation tasks based on the joint loss function; wherein, the training dataset includes training data, and the training data is obtained in the same way as the compressed representation. The network parameters of the multimodal large model to be trained are optimized using the joint loss to obtain the multimodal large model.

3. The method according to claim 2, characterized in that, The joint loss function includes a first loss function, a second loss function, and a balance coefficient; The process involves using the training dataset to drive a large multimodal model to perform classification and report generation tasks, and calculating the joint loss based on the execution results of the classification and report generation tasks using a joint loss function. This includes: The training data is input into the classification branch of the multimodal large model to be trained to obtain the classification result, and the classification loss is calculated based on the first loss function; the classification loss is used to measure the difference between the classification result and the true category. The training data is input into the report generation branch of the multimodal large model to be trained to obtain the generation result, and the generation loss is calculated based on the second loss function; the generation loss is used to measure the difference between the generation result and the real result; The joint loss is obtained by weighting the classification loss and the generation loss using the balance coefficient.

4. The method according to claim 3, characterized in that, After optimizing the network parameters of the multimodal large model to be trained using the joint loss to obtain the multimodal large model, the method further includes: Remove the classification branch of the multimodal large model.

5. The method according to any one of claims 1-4, characterized in that, Before fusing spatial layout information and contextual semantic information from multiple full-slice images through a three-dimensional position encoding and vector fusion mechanism to obtain the fused vector, the method further includes: Multiple full-slice images of the same object are divided into image blocks to obtain multiple image blocks.

6. The method according to claim 5, characterized in that, The method involves fusing spatial layout information and contextual semantic information from multiple full-slice images using a three-dimensional position encoding and vector fusion mechanism to obtain a fused vector, including: Each image block is 3D position encoded to obtain a position vector representing the spatial position of the image block in multiple full-slice images; Semantic encoding is performed on each of the image blocks to obtain a semantic vector representing the semantic features of the image block; The position vector and the semantic vector are concatenated and fused to obtain the fused vector that integrates three-dimensional position information and contextual semantic information.

7. The method according to claim 6, characterized in that, The step of performing three-dimensional position encoding on each of the image blocks to obtain a position vector representing the spatial position of the image block in multiple whole-slice images includes: The three-dimensional position parameters of each image patch are determined; the three-dimensional position parameters include a first position parameter and a second position parameter characterizing the planar position of the image patch in the whole slice image to which it belongs, and a third position parameter characterizing the sequential position of the whole slice image to which the image patch belongs in multiple whole slice images; Cosine coding is performed on the three-dimensional position parameters of each image block to obtain coded components in three spatial dimensions; The encoded components of the three spatial dimensions are added one by one according to the index of the position vector to obtain the position vector of the image block in the spatial location of multiple full-slice images.

8. The method according to claim 7, characterized in that, The cosine coding of the three-dimensional position parameters of each image block yields three spatial dimension coding components, including: Determine a fixed dimension for the position vector; the fixed dimension represents the range of values ​​for the index of the position vector. For any one of the first position parameter, the second position parameter, and the third position parameter of each image block, perform the following encoding operation: For each index of the position vector, if the index is even, the position parameter is encoded using a sine function to obtain the encoded value of the index; If the index is odd, the position parameter is encoded using a cosine function to obtain the encoded value of the index; The encoding operation yields the encoded components of the three spatial dimensions of each image block.

9. The method according to any one of claims 1-4, characterized in that, The method of dynamically aggregating visual lexical units composed of the fused vectors into a fixed-length compressed representation through a learnable lexical compression mechanism includes: Determine a preset set of learnable compressible lexical units; The visual word set composed of the fused vectors is connected to the compressed word set to form a joint sequence; Using the joint sequence as input, multiple rounds of iterative computation are performed through a multi-head attention mechanism; If the iteration reaches a preset number of iterations or the update magnitude of the weight parameters of the compressed tokens is less than a preset threshold, the iteration stops, and a compressed representation of fixed length is obtained.

10. The method according to claim 9, characterized in that, The step of taking the joint sequence as input and performing multiple rounds of iterative computation through a multi-head attention mechanism includes: In any round of iterative computation, using the learnable compressible lexicon set as the query vector matrix and the joint sequence as the key vector matrix and value vector matrix, the weight parameters of the compressible lexicon set are dynamically adjusted in the following manner: The query vector matrix, the key vector matrix, and the value vector matrix are linearly projected and split into multiple sets of sub-vector matrices; each set of sub-vector matrices corresponds to an independent attention head in the multi-head attention mechanism. For each set of subvector matrices, the similarity score between the query vector in the subquery matrix and the key vector in the subkey matrix is ​​calculated, and the similarity score is normalized to obtain the attention weight of the visual word element to the compressed word element; the attention weight represents the degree of contribution of different visual words elements to the compressed word element. Based on the attention weights, the value vectors in the sub-value matrix are weighted and summed to obtain the information aggregation result of each independent attention head; The aggregation results of all independent attention heads are concatenated and fused through a linear transformation layer to generate the parameter update vector for this round of iteration; The weight parameters of the compressed lexical set are adjusted using the parameter update vector to obtain the adjusted compressed lexical set; the adjusted compressed lexical set is used for the next iteration.

11. A data processing apparatus, characterized in that, include: The 3D image encoding module is configured to fuse spatial layout information and contextual image semantic information from multiple full-slice images through a 3D position encoding and vector fusion mechanism to obtain a fused vector. The lexical compression module is configured to dynamically aggregate visual lexical units composed of the fused vectors into a fixed-length compressed representation through a learnable lexical compression mechanism. The prediction module is configured to input the compressed representation into a multimodal large model to obtain a structured analysis report.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.

13. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

Images