Diatom detection method and system based on multi-modal large model

Abstract

The invention discloses a diatom detection method and system based on a multi-modal large model, and relates to the technical field of forensic medicine examination and computer vision crossing, and the method comprises the steps: constructing an image-text pair training data, carrying out the fine tuning of the multi-modal large model based on a Transform architecture, keeping the universal feature extraction capability of a pre-training visual encoder, and carrying out the recognition of the multi-modal large model; a diatom morphology domain dictionary is injected into a text encoder to serve as a learnable embedded vector, and cross-modal alignment of visual features and text features is optimized by adopting a contrast learning loss function; and after knowledge distillation and model quantification, the model carries out space attention fusion on the acquired diatom microscopic image with the known amplification factor and the text embedding vector, target rectangular frame parameters are output through a self-attention weight screening mechanism of a cross-modal decoder, and the size of the diatom is detected and calculated according to the known amplification factor. According to the invention, an intelligent and automatic new solution is provided for forensic medicine diatom inspection.

Description

Technical Field

[0001] The present invention relates to the cross - technical field of forensic examination and computer vision. Specifically, it relates to a diatom detection method and system based on a multimodal large model. Background Art

[0002] In the forensic diagnosis of drowning, the diatom test, as the acquisition of key evidence, highly depends on manual identification and measurement under a microscope. Traditional methods require examiners to visually screen diatoms one by one and compare them with a microscopic micrometer, which has technical bottlenecks such as low detection efficiency and large subjective judgment deviations. Existing automated solutions based on image processing mostly use traditional edge detection algorithms or morphological operations, but face two major limitations in practical applications: First, diatom samples are often mixed with impurity particles, and traditional algorithms are difficult to stably distinguish non - target objects with similar shapes. Second, diatoms are tiny in size and diverse in shape, and conventional detection methods are prone to feature omission or misdetection. In recent years, although object detection technologies based on deep learning have made progress in fields such as industrial inspection and medical image analysis, their application in forensic diatom testing is still restricted by two core problems: First, the data annotation cost is too high. Second, the inconsistency of annotation standards, and there are significant differences in the determination of diatom boundaries by different annotators, resulting in label noise during model training and affecting the detection robustness. In addition, existing technologies have not effectively solved the problem of connecting the detection results with physical size measurement, and still require manual pixel - micron conversion, which restricts the degree of full - process automation. Summary of the Invention

[0003] Aiming at the problems of low efficiency and insufficient automation of traditional diatom testing methods, as well as high annotation costs and the disconnection between detection results and physical measurements of existing deep - learning solutions, the purpose of the present invention is to provide a diatom detection technology based on a multimodal large model, aiming to provide an intelligent and automated solution for forensic diatom testing.

[0004] To achieve the above - mentioned technical purpose, the present application provides a diatom detection method based on a multimodal large model, including the following steps: Based on a multimodal large model with a Transformer architecture, according to the constructed image - text pair training data, by retaining the general feature extraction ability of the pre - trained visual encoder, injecting a dictionary in the field of diatom morphology as a learnable embedding vector into the text encoder, and optimizing the cross - modal alignment of visual features and text features using a contrastive learning loss function, for fine - tuning; Based on the fine - tuned multimodal large model, after knowledge distillation and quantization of the model, the collected diatom microscopic images with known magnification are fused with the text embedding vector through spatial attention, and the target rectangle box parameters are output through the self - attention weight screening mechanism of the cross - modal decoder, and the size of the diatoms is detected and calculated according to the known magnification.

[0005] Preferably, during the process of constructing the image-text pair training data, the lableme annotation tool is used to annotate the diatom rectangular boxes in the scanning electron microscope images, generate a JSON file, convert the annotation data into the COCO format, associate the morphological description text of the diatoms, and construct an image-text pair dataset.

[0006] Preferably, when fine-tuning the multimodal large model based on the Transformer architecture, the morphological text of diatoms is injected into the text encoder, the cross-modal alignment is optimized through the contrastive learning loss, the prior knowledge constraints of the aspect ratio and size range of diatoms are established, and the response of non-target impurities is suppressed.

[0007] Preferably, during knowledge distillation, the cross-modal attention matrix of the teacher model is extracted and transferred to the student model through the KL divergence loss.

[0008] Preferably, when quantifying the model, TensorRT INT8 symmetric quantization is performed on the detection head network layer.

[0009] Preferably, when quantifying the model, the FP16 precision of the text encoder is retained to maintain semantic consistency.

[0010] Preferably, when detecting and calculating the size of diatoms, based on the target rectangular box parameters, the width and height parameters of the rectangular box are respectively multiplied by the magnification factor, and the larger value of the two is taken as the actual major axis, and the smaller value is taken as the actual minor axis.

[0011] The present invention discloses a diatom detection system based on a multimodal large model, including: A dataset construction module, which is used to use the lableme annotation tool to annotate the diatom rectangular boxes in the scanning electron microscope images, generate a JSON file, convert the annotation data into the COCO format, associate the morphological description text of the diatoms, and construct an image-text pair dataset; A model fine-tuning module, which is used to fine-tune the multimodal large model based on the Transformer architecture. Among them, for the multimodal large model based on the Transformer architecture, according to the image-text pair training data, by retaining the general feature extraction ability of the pre-trained visual encoder, the morphological dictionary of the diatom field is injected into the text encoder as a learnable embedding vector, and the contrastive learning loss function is used to optimize the cross-modal alignment of visual features and text features for fine-tuning; The detection module is used to perform spatial attention fusion on the collected diatom microscopic images with known magnification and text embedding vectors based on the fine-tuned multi-modal large model after knowledge distillation and quantization of the model, and output the target rectangle box parameters through the self-attention weight screening mechanism of the cross-modal decoder, and calculate the size of the diatoms based on the known magnification.

[0012] Preferably, the detection module is also used to extract the cross-modal attention matrix of the teacher model and transfer it to the student model through the KL divergence loss.

[0013] Preferably, the detection module is also used to perform TensorRT INT8 symmetric quantization on the detection head network layer and retain the FP16 precision of the text encoder to maintain semantic consistency.

[0014] The present invention discloses the following technical effects: The detection efficiency of the present invention is greatly improved compared with manual measurement; the model can be fine-tuned based on hundreds of labeled samples, and the labeling cost is greatly reduced compared with the traditional scheme; the impurity misdetection rate is significantly reduced compared with the traditional scheme by dynamically adjusting the visual feature weights through text prompts; the lightweight model accelerated by TensorRT greatly reduces the required computing resources; automatically calculates the actual size of diatoms; provides a new intelligent and automated solution for forensic diatom inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0016] Figure 1 is the flow schematic diagram of the present invention; Figure 2 is the schematic diagram of the Transformer architecture of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part rather than all of the embodiments of this application. Components of the embodiments of this application usually described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative efforts belong to the scope of protection of this application.

[0018] As Figure 1-2 shown, the present invention provides a diatom detection method based on a multimodal large model, including the following steps: Based on a multimodal large model of the Transformer architecture, according to the constructed image-text pair training data, by retaining the general feature extraction ability of the pre-trained visual encoder, injecting a diatom morphology field dictionary as a learnable embedding vector into the text encoder, and using a contrastive learning loss function to optimize the cross-modal alignment of visual features and text features, for fine-tuning; Based on the fine-tuned multimodal large model, after knowledge distillation and quantization of the model, the collected diatom microscopic images with known magnification are subjected to spatial attention fusion with the text embedding vector, and the target rectangle box parameters are output through the self-attention weight screening mechanism of the cross-modal decoder, and the size of the diatoms is detected and calculated according to the known magnification.

[0019] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, in the process of constructing the image-text pair training data, the lableme annotation tool is used to annotate the diatom rectangle boxes in the scanning electron microscope images, generate a JSON file, convert the annotation data into the COCO format, associate the morphological description text of the diatoms, and construct an image-text pair dataset.

[0020] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, when fine-tuning the multimodal large model based on the Transformer architecture, inject the diatom morphology text into the text encoder, optimize the cross-modal alignment through contrastive learning loss, establish prior knowledge constraints on the aspect ratio and size range of the diatoms, and suppress the response of non-target impurities.

[0021] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, when performing knowledge distillation, extract the cross-modal attention matrix of the teacher model and transfer it to the student model through the KL divergence loss.

[0022] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, when quantifying the model, TensorRT INT8 symmetric quantization is performed on the detection head network layer.

[0023] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, when quantifying the model, the FP16 precision of the text encoder is retained to maintain semantic consistency.

[0024] Further preferably, for a diatom detection method based on a multimodal large model provided by the present invention, when detecting and calculating the size of diatoms, based on the target rectangle frame parameters, the width and height parameters of the rectangle frame are respectively multiplied by the magnification factor, and the larger value of the two is taken as the actual major axis, and the smaller value is taken as the actual minor axis.

[0025] The present invention discloses a diatom detection system based on a multimodal large model, including: A dataset construction module, which is used to label the diatom rectangle frame in the scanning electron microscope image using the lableme annotation tool, generate a JSON file, convert the annotation data into the COCO format, associate the morphological description text of the diatoms, and construct an image-text pair dataset; A model fine-tuning module, which is used to fine-tune a multimodal large model based on the Transformer architecture. Among them, for the multimodal large model based on the Transformer architecture, according to the image-text pair training data, by retaining the general feature extraction ability of the pre-trained visual encoder, a diatom morphology domain dictionary is injected into the text encoder as a learnable embedding vector, and a contrastive learning loss function is used to optimize the cross-modal alignment of visual features and text features for fine-tuning; A detection module, which is used to, based on the fine-tuned multimodal large model, after knowledge distillation and model quantization, perform spatial attention fusion on the collected diatom microscopic images with known magnification factors and the text embedding vectors, and output the target rectangle frame parameters through the self-attention weight screening mechanism of the cross-modal decoder, and detect and calculate the size of the diatoms based on the known magnification factor.

[0026] Further preferably, the detection module of a diatom detection system based on a multimodal large model disclosed by the present invention is further used to extract the cross-modal attention matrix of the teacher model and transfer it to the student model through the KL divergence loss.

[0027] Further preferably, the detection module of a diatom detection system based on a multimodal large model disclosed by the present invention is further used to perform TensorRT INT8 symmetric quantization on the detection head network layer and retain the FP16 precision of the text encoder to maintain semantic consistency.

[0028] Example: The present invention provides a diatom frustule detection technology based on a multimodal large model, which specifically includes the following processes: S1: Convert the diatom image dataset generated by the annotation tool and the diatom morphological feature text into a format to generate multimodal training data containing image features and morphological text descriptions.

[0029] The format conversion in step S1 specifically includes: parsing the JSON file annotated by lableme to extract the rectangular box coordinates and converting them into the COCO annotation format; associating the morphological description texts of each diatom to construct image-text pair training data.

[0030] S2: Based on the multimodal dataset constructed in step S1, adopt a domain adaptation fine-tuning strategy to optimize the parameters of the multimodal large model based on the Transformer architecture, and construct a multimodal detection model sensitive to diatom features.

[0031] The domain adaptation fine-tuning strategy in step S2 includes: freezing the visual encoder network parameters of the multimodal large model based on the Transformer architecture; injecting a diatom morphology domain dictionary into the text encoder as a learnable embedding vector; using a contrastive learning loss function to optimize the cross-modal alignment of visual features and text features.

[0032] S3: Transfer the cross-modal feature alignment ability of the fine-tuned model to the lightweight student model through a knowledge distillation framework to form a detection architecture with real-time inference ability.

[0033] The knowledge distillation framework in step S3 includes: extracting the cross-modal attention matrix of the teacher model as a supervision signal; constructing a visual-text feature interaction constraint loss function for the student model; realizing transfer learning of attention weights through minimizing the KL divergence.

[0034] S4: Perform low-bit quantization compression on the detection model obtained in step S3 to generate a lightweight deployment version.

[0035] The specific implementation of the low-bit quantization in step S4 is: performing TensorRT INT8 symmetric quantization on the detection head network layer; retaining the FP16 floating-point calculation precision of the text encoder to maintain the semantic alignment stability.

[0036] S5: Obtain a diatom microscopic image with a known magnification through a scanning electron microscope.

[0037] S6: Input the microscopic image and the diatom morphological feature text into the detection model in step S4, and output the horizontal rectangular box parameters of pixel-level localization through a cross-modal decoder.

[0038] The cross-modal decoder workflow in step S6 includes: performing spatial attention fusion on the visual feature map and the text embedding vector; and outputting the target rectangle box parameters through the self-attention weight screening mechanism of the cross-modal decoder.

[0039] S7: Automatically calculate the actual major and minor diameters of the diatoms based on the rectangle box geometric parameters and the image magnification factor output in step S6.

[0040] The size calculation in step S7 includes: multiplying the width and height parameters of the rectangle box by the magnification factor respectively, and taking the larger value of the two as the actual major diameter and the smaller value as the actual minor diameter.

[0041] The present invention analyzes the JSON file labeled by lableme to extract the rectangle box coordinates and convert them into the COCO annotation format, associates the morphological description texts of each diatom, constructs the image-text pair training data; fine-tunes the multi-modal large model based on the Transformer architecture, retains the general feature extraction ability of the pre-trained visual encoder, injects the diatom morphological text into the text encoder, optimizes the cross-modal alignment through the contrastive learning loss, establishes the prior knowledge constraints on the aspect ratio and size range of the diatoms, and suppresses the response of non-target impurities; extracts the cross-modal attention matrix of the teacher model and transfers it to the student model through the KL divergence loss; performs INT8 symmetric quantization on the detection head, and the text encoder retains the FP16 precision to maintain semantic consistency; performs spatial attention fusion on the visual feature map and the text embedding vector to output the target rectangle box; combines the image magnification factor to output the actual major and minor diameters of the diatoms. The detection efficiency of the present invention is greatly improved compared with manual measurement; the model can be fine-tuned based on hundreds of labeled samples, and the labeling cost is greatly reduced compared with the traditional scheme; the visual feature weights are dynamically adjusted through text prompts, and the impurity misdetection rate is significantly reduced compared with the traditional scheme; the lightweight model accelerated by TensorRT greatly reduces the required computing resources; automatically calculates the actual size of the diatoms.

[0042] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0043] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality of" means two or more, unless otherwise specifically defined.

[0044] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A diatom detection method based on a multimodal large model, characterized in that: The following steps are involved: A large multimodal model based on the Transformer architecture is fine-tuned based on the constructed image-text pair training data, by retaining the general feature extraction capability of the pre-trained visual encoder, injecting the diatom morphology domain dictionary into the text encoder as a learnable embedding vector, and using the contrastive learning loss function to optimize the cross-modal alignment of visual features and text features; Based on the fine-tuned multimodal large model, after knowledge distillation and quantization model, the collected diatom microscopic images of known magnifications are spatially fused with the text embedding vector. The target rectangular frame parameters are output through the self-attention weight screening mechanism of the cross-modal decoder, and the size of the diatoms is detected and calculated based on the known magnification.

2. A diatom detection method based on a multimodal large model according to claim 1, characterized in that: In the process of constructing the image-text pair training data, the labelme annotation tool is used to annotate the diatom rectangular frame on the scanning electron microscope image, generate a JSON file, convert the annotation data into COCO format, associate the morphological description text of the diatom, and construct the image-text pair dataset.

3. A diatom detection method based on a multimodal large model according to claim 2, characterized in that: When fine-tuning the large multimodal model based on the Transformer architecture, diatom morphology text is injected into the text encoder, and cross-modal alignment is optimized through contrastive learning loss. Prior knowledge constraints on the aspect ratio and size range of diatoms are established to suppress non-target impurity responses.

4. The diatom detection method based on a multimodal large model according to claim 3 is characterized in that: When performing knowledge distillation, the cross-modal attention matrix of the teacher model is extracted and transferred to the student model through the KL divergence loss.

5. A diatom detection method based on a multimodal large model according to claim 4, characterized in that: When quantizing the model, TensorRT INT8 symmetric quantization is performed on the detection head network layer.

6. A diatom detection method based on a multimodal large model according to claim 5, characterized in that: When quantizing the model, the FP16 precision of the text encoder is retained to maintain semantic consistency.

7. A diatom detection method based on a multimodal large model according to claim 6, characterized in that: When detecting and calculating the size of diatoms, based on the target rectangular frame parameters, the width and height parameters of the rectangular frame are multiplied by the magnification respectively, and the larger value of the two is taken as the actual long diameter, and the smaller value is taken as the actual short diameter.

8. A diatom detection system based on a multimodal large model, characterized in that: include: The dataset construction module is used to annotate the diatom rectangles on the SEM images using the labelme annotation tool, generate a JSON file, convert the annotation data into COCO format, associate the morphological description text of the diatoms, and construct an image-text pair dataset; A model fine-tuning module, used for fine-tuning a large multimodal model based on a Transformer architecture, wherein the large multimodal model based on the Transformer architecture is fine-tuned based on the image-text pair training data, by retaining the general feature extraction capability of the pre-trained visual encoder, injecting a diatom morphology domain dictionary into the text encoder as a learnable embedding vector, and using a contrastive learning loss function to optimize the cross-modal alignment of visual features and text features; The detection module is used to perform spatial attention fusion on the collected diatom microscopic images of known magnifications with the text embedding vector based on the fine-tuned multimodal large model after knowledge distillation and quantization. The target rectangular frame parameters are output through the self-attention weight screening mechanism of the cross-modal decoder, and the size of the diatoms is detected and calculated based on the known magnification.

9. A diatom detection system based on a multimodal large model according to claim 8, characterized in that: The detection module is also used to extract the cross-modal attention matrix of the teacher model and transfer it to the student model through the KL divergence loss.

10. A diatom detection system based on a multimodal large model according to claim 9, characterized in that: The detection module is also used to perform TensorRT INT8 symmetric quantization on the detection head network layer, retaining the FP16 precision of the text encoder to maintain semantic consistency.

Images