Optical splitter inspection image multi-modal compliance discrimination method and device and electronic equipment

Abstract

The application discloses a splitter inspection image multi-modal compliance discrimination method and device and electronic equipment, comprising: constructing a picture-text data set for reprinting discrimination and bar code label compliance discrimination under a splitter inspection scene; generating negative samples with the same distribution according to splitter inspection rules, and adding the negative samples to the picture-text data set; using a picture-text semantic alignment network to perform picture-text data joint enhancement on the picture-text data set to obtain a training data set; using a multi-task optimization fine-tuning algorithm based on gradient consistency constraint to fine-tune a multi-modal large model according to the training data set to obtain a splitter compliance discrimination model; using a knowledge graph to perform graph retrieval reasoning on a real-time query request to obtain corresponding target associated articles, inputting the real-time query request and the target associated articles into the splitter compliance discrimination model to obtain a splitter compliance discrimination result. The application improves the accuracy and reliability of splitter inspection image compliance discrimination, and can be applied to the field of computer vision technology.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and electronic device for multimodal compliance discrimination of beam splitter inspection images. Background Technology

[0002] Currently, during the inspection of beam splitters, issues such as photocopying and misplacement of equipment barcode labels exist. Existing technologies rely on manual review or a combination of manual image uploads and AI-assisted judgment to determine beam splitter image compliance. Different AI models are trained for different tasks; for example, two different models are used to separately identify photocopying and barcode label compliance in beam splitter inspection scenarios. Specifically, inspection personnel take images of the beam splitter (including the barcode label) using a terminal (mobile phone / tablet) and upload them. The AI ​​model then automatically determines whether the barcode label is correctly affixed and whether the image is a photocopy, replacing traditional manual review and improving the efficiency of beam splitter image compliance determination.

[0003] The aforementioned existing technology has the following problems: 1) The AI ​​models used in existing solutions for judging the compliance of images of beam splitters are mostly based on single-modal CNN models, relying solely on image information without combining text descriptions (such as the reasons for judgment). They lack joint modeling of the semantic consistency between images and text descriptions, resulting in poor interpretability and low reliability. At the same time, different tasks require different AI models, making the training of a large number of AI models a complex process.

[0004] 2) In the existing scheme, negative samples such as reproduced images and bar labels without pasting are few in real-world scenarios, resulting in an imbalance with positive samples and poor model generalization ability.

[0005] 3) Existing data augmentation methods are mostly limited to image-level transformations (such as rotation and cropping) and do not consider multimodal semantic alignment. Traditional augmentation methods may destroy the semantic consistency between images and text.

[0006] 4) Existing solutions using general LoRA multi-task fine-tuning may encounter gradient conflict issues. If there is a conflict or coupling between the two tasks, the interaction between the two tasks needs to be considered.

[0007] 5) The existing large model does not have prior knowledge of the telecommunications domain. It simply conducts supervised training on the model in a rigid manner through fine-tuning, rather than making judgments based on understanding the standard rules of the telecommunications domain. It also lacks interpretability. At the same time, for retrieval based on single vector similarity, the retrieved relevant texts lack domain robustness.

[0008] The above problems urgently need to be addressed. Summary of the Invention

[0009] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0010] Therefore, one objective of this invention is to provide a multimodal compliance discrimination method for beam splitter inspection images, which improves the accuracy and reliability of compliance discrimination for beam splitter inspection images.

[0011] Another objective of this invention is to provide a multimodal compliance discrimination device for beam splitter inspection images.

[0012] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a method for multimodal compliance determination of beam splitter inspection images, including the following steps: Construct an image and text dataset for photocopying and barcode label compliance assessment in a beam splitter inspection scenario; Negative samples conforming to the same distribution are generated according to the spectrometer inspection rules, and the negative samples are added to the image and text dataset; The image-text dataset is jointly augmented using an image-text semantic alignment network to obtain a training dataset. Based on the training dataset, a multi-task optimization and fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the multimodal large model to obtain a spectrometer compliance discrimination model. The real-time query request is used to perform graph retrieval reasoning to obtain the corresponding target related text. The real-time query request and the target related text are then input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

[0013] Furthermore, in one embodiment of the present invention, the step of generating negative samples conforming to the same distribution according to the spectrometer inspection rules specifically includes: The beam splitter inspection rules are transformed into a structured rule set, which includes rule types, rule descriptions, and image generation operators. Multiple positive sample images are obtained from the image and text dataset, and corresponding negative sample images are generated based on the image generation operator and the positive samples. Based on the rule type, the rule description, and the negative sample image, the corresponding negative sample text is generated by rewriting the large model. The negative sample is generated based on the negative sample image and the negative sample text.

[0014] Furthermore, in one embodiment of the present invention, the step of using a graph-text semantic alignment network to perform joint graph-text data augmentation on the graph-text dataset to obtain a training dataset specifically includes: Obtain the first sample image and the corresponding first sample text from the image and text dataset; Image enhancement is performed on the first sample image to obtain a second sample image and corresponding image transformation parameters; The image transformation parameters are input into a pre-constructed image-text semantic alignment network to obtain the corresponding text transformation description; The first sample text is enhanced according to the text transformation description to obtain the second sample text; Image-text augmentation samples are generated based on the second sample image and the second sample text, and the image-text augmentation samples are added to the image-text dataset to obtain the training dataset.

[0015] Furthermore, in one embodiment of the present invention, the step of fine-tuning the multimodal large model using a multi-task optimization fine-tuning algorithm based on gradient consistency constraints according to the training dataset to obtain a spectrometer compliance discrimination model specifically includes: Read the target image and text samples and their corresponding sample labels sequentially from the training dataset; The target image and text samples are respectively input into the re-photographing discrimination task branch and the barcode label discrimination task branch of the multimodal large model to obtain the re-photographing discrimination result and the barcode label discrimination result; The re-photographing discrimination result, the barcode label discrimination result, and the sample label are used to determine the re-photographing discrimination loss value and the barcode label discrimination loss value. Calculate the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task based on the re-photographing discrimination loss value and the barcode label discrimination loss value, respectively; The gradients for the photo re-image discrimination task and the barcode label discrimination task are subjected to conflict detection and gradient merging to obtain the target task gradient; The parameters of the multimodal large model are fine-tuned based on the target task gradient to obtain the compliance discrimination model of the spectrometer.

[0016] Furthermore, in one embodiment of the present invention, the step of performing conflict detection and gradient merging on the gradient of the photocopying discrimination task and the gradient of the barcode label discrimination task to obtain the target task gradient specifically includes: Calculate the inner product of the gradient of the photocopying discrimination task and the gradient of the barcode label discrimination task; When the inner product is greater than or equal to 0, it is determined that there is no conflict between the re-photographing identification task and the barcode label identification task, and the gradient of the target task is determined according to the sum of the gradient of the re-photographing identification task and the gradient of the barcode label identification task. When the inner product is less than 0, it is determined that there is a conflict between the photocopying identification task and the barcode label identification task. The gradient of the photocopying identification task is then projected and corrected according to the gradient of the barcode label identification task. Finally, the target task gradient is determined based on the sum of the projected gradient of the photocopying identification task and the gradient of the barcode label identification task.

[0017] Furthermore, in one embodiment of the present invention, the step of using a knowledge graph to perform graph retrieval reasoning on a real-time query request to obtain the corresponding target related text specifically includes: Entity extraction is performed on the real-time query request to obtain an entity set; The entity set is injected into a preset domain knowledge graph for multi-hop queries to obtain a set of related clauses; Calculate the relevance score between each associated clause in the associated clause set and the real-time query request; The relevant clauses with a correlation score greater than a preset first threshold are selected as the target relevant clauses.

[0018] Furthermore, in one embodiment of the present invention, the step of inputting the real-time query request and the target associated clause into the splitter compliance discrimination model to obtain the splitter compliance discrimination result specifically includes: Get the preset prompt template; The target prompt statement is obtained by combining the real-time query request and the target related text according to the prompt template. The target prompt statement is input into the beam splitter compliance discrimination model to obtain the beam splitter compliance discrimination result.

[0019] On the other hand, embodiments of the present invention provide a multimodal compliance discrimination device for beam splitter inspection images, comprising: The dataset construction module is used to build image and text datasets for photocopying and barcode label compliance judgment in the scenario of spectrometer inspection; The negative sample generation module is used to generate negative samples that conform to the same distribution according to the spectrometer inspection rules, and add the negative samples to the image and text dataset; The dataset augmentation module is used to perform joint augmentation of the image and text dataset using an image-text semantic alignment network to obtain a training dataset. The model fine-tuning module is used to fine-tune the multimodal large model based on the training dataset using a multi-task optimization fine-tuning algorithm based on gradient consistency constraints, so as to obtain the compliance discrimination model of the spectrometer. The model discrimination module is used to perform graph retrieval reasoning on real-time query requests using knowledge graphs to obtain corresponding target related texts. The real-time query request and the target related texts are input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

[0020] On the other hand, embodiments of the present invention provide an electronic device, which includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the computer program is executed by the processor, it implements the multimodal compliance discrimination method for beam splitter inspection images as described above.

[0021] On the other hand, embodiments of the present invention also provide a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the multimodal compliance discrimination method for beam splitter inspection images as described above.

[0022] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the multimodal compliance discrimination method for beam splitter inspection images as described above.

[0023] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention constructs an image-text dataset for image re-photographing and barcode label compliance judgment in a beam splitter inspection scenario. Negative samples conforming to the same distribution are generated according to beam splitter inspection rules and added to the image-text dataset. An image-text semantic alignment network is used to jointly enhance the image-text data, resulting in a training dataset. Based on the training dataset, a multi-task optimization and fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the multimodal large model, resulting in a beam splitter compliance judgment model. A knowledge graph is used to perform graph retrieval reasoning on real-time query requests to obtain the corresponding target-related clauses. The real-time query request and the target-related clauses are input into the beam splitter compliance judgment model to obtain the beam splitter compliance judgment result. This invention generates negative samples according to beam splitter inspection rules, performs joint image-text data enhancement based on an image-text semantic alignment network, and fine-tunes the model using a multi-task optimization and fine-tuning algorithm based on gradient consistency constraints to obtain the beam splitter compliance judgment model. The knowledge graph guides the beam splitter compliance judgment model to perform beam splitter compliance judgment on real-time query requests, improving the accuracy and reliability of beam splitter inspection image compliance judgment. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating one step of the multimodal compliance determination method for beam splitter inspection images provided in an embodiment of the present invention. Figure 2 A flowchart of step S102 provided in an embodiment of the present invention; Figure 3 A flowchart of step S103 provided in an embodiment of the present invention; Figure 4 A flowchart of step S104 provided in an embodiment of the present invention; Figure 5 A flowchart of step S1045 provided in an embodiment of the present invention; Figure 6 A flowchart of step S105 provided in an embodiment of the present invention; Figure 7 Another flowchart of step S105 provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the process for jointly enhancing text and image data provided in an embodiment of the present invention; Figure 9 This is a schematic diagram illustrating the principle of the image-text semantic alignment network provided in an embodiment of the present invention. Figure 10 This is a schematic diagram of the knowledge graph-guided discrimination process provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of the multimodal compliance discrimination device for beam splitter inspection images provided in an embodiment of the present invention; Figure 12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the structure of the storage medium provided in an embodiment of the present invention. Detailed Implementation

[0026] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are only set for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] The following explains some of the terms used in the embodiments of the present invention.

[0029] Multimodal Large Language Model: An artificial intelligence model capable of processing both image and text information simultaneously.

[0030] Optical Splitter: A passive device used to distribute optical signals into multiple optical fibers.

[0031] Rephotographed image: refers to a secondary image obtained by taking a picture or taking a screenshot of the original picture. It is often used for fraudulent inspection records.

[0032] Negative Sample: In this scheme, it specifically refers to two types of samples that do not meet the inspection requirements, including "images with barcode labels not affixed / abnormally affixed on the beam splitter + corresponding text" and "images taken during beam splitter inspections + corresponding text".

[0033] Data augmentation is a technique that expands the size and diversity of a dataset and improves the generalization ability of a model by performing reasonable artificial transformations (such as rotation and noise addition) on the original data (such as images, text, and audio).

[0034] CNN (Convolutional Neural Network) is a deep learning model that extracts local features through convolutional layers and compresses data dimensionality through pooling layers, making it adept at processing grid-structured data such as images.

[0035] LoRA (Low-Rank Adaptation) is a method for efficiently fine-tuning pre-trained models using low-rank matrix factorization. It requires updating only a small number of parameters to adapt to a specific task, significantly reducing computational costs and storage requirements while maintaining performance comparable to full-model fine-tuning. It is widely used in natural language processing, computer vision, and other fields. DPA-NET (Dynamic Position-semantic Alignment Network) is a network structure proposed in this invention to ensure semantic consistency between text and images.

[0036] Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP). It aims to automatically identify entities with specific meanings, such as names of people, places, organizations, times, and numbers, from text and label them with their respective categories.

[0037] The multimodal compliance discrimination method for beam splitter inspection images provided in this application embodiment can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing a driver state monitoring method based on machine vision, but is not limited to the above forms.

[0038] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0039] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0040] like Figure 1 The diagram shown is a flowchart of one step of the multimodal compliance discrimination method for beam splitter inspection images provided by an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for multimodal compliance determination of beam splitter inspection images, specifically including the following steps: S101. Construct an image and text dataset for photocopying and barcode label compliance identification in the scenario of beam splitter inspection; S102. Generate negative samples that conform to the same distribution according to the beam splitter inspection rules, and add the negative samples to the image and text dataset. S103. Use the image-text semantic alignment network to perform joint image-text data augmentation on the image-text dataset to obtain the training dataset; S104. Based on the training dataset, a multi-task optimization and fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the multimodal large model to obtain the compliance discrimination model of the spectrometer. S105. Use the knowledge graph to perform graph retrieval reasoning on the real-time query request to obtain the corresponding target related text. Input the real-time query request and the target related text into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

[0041] It can be recognized that, in the embodiments of the present invention, negative samples are generated according to the beam splitter inspection rules, image and text data are jointly enhanced based on the image-text semantic alignment network, and a multi-task optimization fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the model to obtain the beam splitter compliance discrimination model. The knowledge graph is used to guide the beam splitter compliance discrimination model to perform beam splitter compliance discrimination on real-time query requests, thereby improving the accuracy and reliability of beam splitter inspection image compliance discrimination.

[0042] like Figure 2 The diagram shown is a flowchart of step S102 provided in an embodiment of the present invention. (Refer to...) Figure 2 As an optional implementation, negative samples conforming to the same distribution are generated according to the spectrometer inspection rules, specifically including: S1021. Transform the beam splitter inspection rules into a structured rule set, which includes rule type, rule description, and image generation operator; S1022. Obtain multiple positive sample images from the image and text dataset, and generate corresponding negative sample images based on the image generation operator and the positive samples; S1023. Based on the rule type, rule description, and negative sample image, rewrite and generate the corresponding negative sample text according to the large model; S1024. Generate negative samples based on negative sample images and negative sample text.

[0043] Specifically, we first construct an image and text dataset for judging the re-photographing and the compliance of barcode labels, collect positive samples (correct pasting + standard text) and scarce negative samples (real re-photographing / not pasted + text), and define the labeling fields "image ID - whether it is a re-photographing - whether it is pasted - reason for judgment".

[0044] Then, using an information extraction model, key rules are extracted from telecommunications operation and maintenance specification documents based on task type to form a negative sample generation strategy. The specific steps are as follows: 1) Information is extracted from telecommunications operation and maintenance specification documents based on task type using a large model and prompt words, resulting in the following formatted output: { "type": "remake", "description": "Images reproduced from the screen often exhibit noticeable moiré patterns." "feature": ["moiré pattern"] }" 2) Pre-defined mapping dictionary. By querying this dictionary, the extracted key features are mapped to the corresponding image generation operators, forming executable generation instructions. The final output is a structured, complete set of rules containing operator sequences. as follows: { "rule_id": "r1", "type": "remake", "description": "Images reproduced from the screen often exhibit noticeable moiré patterns." "feature": ["moiré pattern"], "operator": [" "] } Where operator is the corresponding operator.

[0045] 3) Targeted negative sample image generation: For positive samples ,according to Operator selection operator Instead of random selection, that is .

[0046] 4) Targeted negative sample text generation: Based on the type and description in the rule set, the large model is used to rewrite and generate... Corresponding text The final result is a high-quality image and text dataset with multiple negative samples.

[0047] Preferably, a business rule-driven negative sample generation strategy is adopted, and the specific steps are as follows: 1) Define the rule set: Using the BERT large model with strong generalization ability and prompt words, information is extracted from telecommunications operation and maintenance specification documents (such as the "Optical Splitter Installation and Acceptance Specification") according to the task type. The prompt words are as follows: Extract all descriptions related to "splitter photocopying" or "improper labeling" from the text, summarize the violation types and key features, and output them in the following format. { "type":"remake", "description": "Images reproduced from the screen often exhibit noticeable moiré patterns." "feature": ["moiré pattern"] }" 2) Pre-defined mapping dictionary. By querying this dictionary, the extracted key features are mapped to the corresponding image generation operators, forming executable generation instructions. This mapping dictionary is a predefined key-value pair structure, some examples of which are shown below: Key feature "moiré pattern" mapped to image generation operator Key feature: "Missing label mapping to image generation operator" .in The operator is used to simulate the moiré effect in reproduced images. It is achieved by overlaying and blending the input image with a sinusoidal raster layer of specific frequency and direction. Operator: Used to simulate missing labels. Internally, it first calls a pre-trained spectrometer label detection model to locate the target area where the label should be pasted, and then performs pixel erasure or background filling on that area.

[0048] The final output is a structured, complete set of rules containing a sequence of operators. .For example: { "rule_id":"r1", "type":"remake", "description": "Images reproduced from the screen often exhibit noticeable moiré patterns." "feature":["moiré pattern"], "operator":[" "] }, { "rule_id":"r2", "type":"Label missing", "description":"Label malfunctions often occur on the front of the equipment." "feature":["missing post"], "operator":[" "] } Where "type" is the type of negative sample, "description" is the rule description, "feature" is the key feature, and "operator" is the negative sample generation operator.

[0049] 3) Targeted negative sample image generation: For positive samples If negative samples of remakes are generated, according to Select the operator corresponding to the re-photographing. ,Right now , Apply transformation: ,in Based on r1, which represents synthesized moiré noise. If negative samples are missing from the generated labels... Then apply the transformation: ,in This is an r2-based region erasure or occlusion operation. Similarly, other operators are called sequentially from the operator library according to the operators specified in the `operator` field of the rules to generate negative sample images. .

[0050] 4) Targeted negative sample text generation: based on rules The type and description in the model are rewritten and filled with a preset text template using a large model.

[0051] Template example: {"result":"xxx","reason":"xxx"}.

[0052] For example, for the enhanced image Generate corresponding violation description text. For images with erased labels, the generated text is: {"result":"false","reason":"Label not pasted in the specified area of ​​the device".} Finally, a high-quality negative sample dataset was obtained. To address the scarcity of negative samples in the field of spectrometer inspection, its data distribution... Near-realistic scene distribution .

[0053] Furthermore, it is also possible to generate composite images of the same type of single violation image using multiple rules. Through the ordered combination of various generation operators, complex negative samples that more closely resemble real-world scenes can be generated. The generation formula is: For example, if negative samples of remakes are generated, then , This enables the generation of multiple rules in combination, among which For the moiré pattern generation operator, To generate operators for black borders in photo reproductions, the following transformations are applied: .

[0054] like Figure 3 The diagram shown is a flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 3 As an optional implementation, a graph-text semantic alignment network is used to jointly augment the graph-text dataset to obtain a training dataset, which specifically includes: S1031. Obtain the first sample image and the corresponding first sample text from the image and text dataset; S1032. Perform image enhancement on the first sample image to obtain the second sample image and the corresponding image transformation parameters; S1033. Input the image transformation parameters into a pre-constructed image-text semantic alignment network to obtain the corresponding text transformation description; S1034. Perform text enhancement on the first sample text according to the text transformation description to obtain the second sample text; S1035. Generate image-text augmented samples based on the second sample image and the second sample text, and add the image-text augmented samples to the image-text dataset to obtain the training dataset.

[0055] Specifically, such as Figure 8 The diagram shown is a schematic representation of the process for jointly enhancing text and image data according to an embodiment of the present invention. Figure 9 The diagram shown illustrates the principle of the image-text semantic alignment network provided in this embodiment of the invention. This embodiment enhances both the image and text, with the specific steps as follows: 1) Targeted Image Enhancement: The image is rotated at random angles to simulate shooting from different angles. Random contrast and brightness adjustments are applied to simulate shooting under different lighting conditions. Specifically, the preferred random contrast adjustment range is 0.6-1.4, and the preferred random brightness adjustment range is 0.5-1.5. Random noise blur is added to simulate shooting under different image quality and noise environments. Specifically, Gaussian blur and Gaussian noise are used, with the Gaussian blur using a Gaussian kernel size of 5 to enhance the image enhancement effect.

[0056] 2) Image-Text Semantic Alignment Module: When an image is rotated, the words describing the absolute position of the image change. To address the inconsistency between image and text semantics caused by geometric transformations, we designed a Dynamic Position-semantic Alignment Network (DPA-Net) based on an attention mechanism. This network automatically learns the correspondence between image transformations and changes in text positional vocabulary through a small neural network, significantly improving the accuracy and generalization ability of the alignment. The specific process is as follows: Input: Receive the image transformation parameter matrix T and the original text description. Let the image transformation parameters be:

[0057] in The angle is the rotation angle.

[0058] Location entity recognition: A lightweight NER module (named entity model, preferably BERT-based) is used to identify all words in the text that describe absolute location (such as "top left" or "right").

[0059] Coordinate Mapping and Transformation: Each identified location word is mapped to its normalized coordinates in the image (e.g., "top left" is mapped to (0,0), "top" to (0.5,0), and "bottom right" to (1,1)), and stored in the word mapping table L. An affine transformation matrix M_T is predicted using a learnable micro-transformation network (a fully connected layer) controlled by transformation parameters T, and the new coordinates of the location words in the transformed image are calculated.

[0060] Predict coordinate offsets using a lightweight MLP:

[0061] in: , The first layer weights and biases; For activation functions; , For output layer weights and biases; This represents the predicted coordinate offset.

[0062] Calculate the new coordinates after the transformation:

[0063] Attention reweighting: The similarity between the new coordinates and the coordinates of all standard position words is calculated, and the weight distribution is obtained through the Softmax attention mechanism. This mechanism can dynamically and accurately establish the correspondence between position words before and after the transformation (for example, after the image is rotated 90 degrees clockwise, the attention weight of "top left corner" will be highly concentrated on "bottom left corner").

[0064]

[0065] in: This scheme uses temperature as a parameter to control the sharpness of attention distribution. ; For the vocabulary list The Middle Standard coordinates of each positional word; Indicates that the new coordinates are from the first The probability described by each positional word.

[0066] 3) Aligned Text Generation: Based on attention weights, transformed positional words are dynamically generated or selected to replace corresponding words in the original text, ultimately outputting a text description that is strictly semantically aligned with the enhanced image. This method ensures that semantic consistency between the image and text is maintained even under complex combined transformations.

[0067]

[0068] in l newThese are the position words in the L vocabulary corresponding to the words with the highest weight.

[0069] 4) Specialized Text Augmentation: Customized synonym replacements are then applied to specialized terms in the text, effectively improving the model's sensitivity to such terms. For some descriptive text, the NLTK library in Python is used for synonym replacement. Specifically, a custom vocabulary is used to find specialized synonyms such as "splitter," "moiré pattern," and "splitter port," and these are replaced. The NLTK library's thesaurus is also used to find and replace synonyms. This invention utilizes a large model to rewrite text using synonyms, ultimately generating image-text datasets with various augmentation strategies. This solves the cross-modal semantic inconsistency problem that may occur in data augmentation, while simultaneously considering data diversity and semantic accuracy.

[0070] like Figure 4 The diagram shown is a flowchart of step S104 provided in an embodiment of the present invention. (Refer to...) Figure 4 As an optional implementation, the multimodal large model is further fine-tuned using a multi-task optimization fine-tuning algorithm based on gradient consistency constraints according to the training dataset to obtain a spectrometer compliance discrimination model, which specifically includes: S1041. Read the target image and text samples and their corresponding sample labels sequentially from the training dataset; S1042. Input the target image and text sample into the re-photographing discrimination task branch and the barcode label discrimination task branch of the multimodal large model respectively to obtain the re-photographing discrimination result and the barcode label discrimination result. S1043. Determine the re-photographing discrimination loss value and the barcode label discrimination loss value based on the re-photographing discrimination result, the barcode label discrimination result, and the sample label; S1044. Calculate the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task based on the re-photographing discrimination loss value and the barcode label discrimination loss value, respectively. S1045. Perform conflict detection and gradient merging on the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task to obtain the target task gradient; S1046. Fine-tune the parameters of the multimodal large model according to the target task gradient to obtain the compliance discrimination model of the spectrometer.

[0071] like Figure 5 The diagram shown is a flowchart of step S1045 provided in an embodiment of the present invention. (Refer to...) Figure 5 As an optional implementation, conflict detection and gradient merging are performed on the gradients of the photocopying and barcode labeling tasks to obtain the target task gradient, which specifically includes: S10451. Calculate the inner product of the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task; S10452. When the inner product is greater than or equal to 0, it is determined that there is no conflict between the re-photographing discrimination task and the barcode label discrimination task. The gradient of the target task is determined based on the sum of the gradients of the re-photographing discrimination task and the barcode label discrimination task. S10453. When the inner product is less than 0, it is determined that there is a conflict between the re-photographing discrimination task and the barcode label discrimination task. The gradient of the re-photographing discrimination task is projected and corrected according to the gradient of the barcode label discrimination task, and the gradient of the target task is determined according to the sum of the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task after projection correction.

[0072] Specifically, the enhanced dataset is used to fine-tune the compliance judgment of optical splitter inspection in telecommunications equipment rooms. The LLaMA-Factory framework is used, and LoRa fine-tuning is adopted. The fine-tuning strategy adopts multi-task optimization fine-tuning based on gradient consistency constraints. The specific steps of the algorithm are as follows: 1) Calculate the loss and gradient: For a batch Given samples, calculate the cross-entropy loss between the two sub-tasks:

[0073]

[0074] in For real labels, Predict probabilities for the model.

[0075] Calculate the gradients of the two subtasks:

[0076] 2) Gradient conflict detection and correction: First, calculate the inner product of the task gradients:

[0077] like If the two gradients are in the same direction or do not conflict, they can be directly merged.

[0078] like If this is the case, it indicates a conflict, and the gradient needs to be corrected by projection. This invention uses the following correction formula:

[0079] in, The projection intensity coefficient, This is a numerically stable term that can be dynamically adjusted based on the accuracy of the validation set. The validation set data shows that when... It is 0.8. 10 12 The timing is preferred.

[0080] 3) Gradient merging and parameter updates are as follows:

[0081] Finally, the corrected gradient is used. The AdamW optimizer is preferred, with a batch size of 16, to update the model parameters:

[0082] in, The learning rate is preferably 2×10. -4 .

[0083] 4) Repeat the above steps until the training convergence condition is met. The final result is a multi-task inspection compliance judgment model that possesses both stability and high accuracy.

[0084] The embodiments of the present invention differ from traditional loss weight allocation schemes by directly eliminating conflicts at the gradient level, ensuring that the optimization direction of each task is consistent, thereby improving the training stability and discrimination accuracy of multi-task models. It is especially suitable for complex environments such as inspection scenarios where negative samples are scarce and tasks are unbalanced.

[0085] like Figure 6 The diagram shown is a flowchart of step S105 provided in an embodiment of the present invention. (Refer to...) Figure 6 As an optional implementation, a knowledge graph is used to perform graph retrieval reasoning on real-time query requests to obtain the corresponding target related text, specifically including: S1051. Extract entities from real-time query requests to obtain an entity set; S1052. Inject the entity set into the preset domain knowledge graph and perform multi-hop queries to obtain the set of related clauses; S1053. Calculate the relevance score between each related clause in the related clause set and the real-time query request; S1054. Select the relevant clauses with a relevance score greater than the preset first threshold as the target relevant clauses.

[0086] like Figure 7 The diagram shown is another flowchart of step S105 provided in an embodiment of the present invention. (Refer to...) Figure 7 As an optional implementation, the real-time query request and the target related clause are input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result, which specifically includes: S1055. Obtain the preset prompt template; S1056. Combine the real-time query request and the target related clauses according to the prompt template to obtain the target prompt statement; S1057. Input the target prompt statement into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

[0087] Specifically, such as Figure 10 The diagram shown illustrates the knowledge graph-guided judgment process provided in this embodiment of the invention. During the reasoning process, this embodiment employs a knowledge graph-enhanced retrieval method. Given a query Q (e.g., "Please determine whether the barcode label on the spectrometer in the image is compliant"), relevant regulatory provisions are obtained through multi-hop retrieval. The specific steps are as follows: 1) Domain Entity Recognition: Using the NER model from Extract entity set (e.g., [splitter, label]).

[0088] 2) Graph query and reasoning: ... Injecting knowledge graph Perform multi-hop queries to find all related sets of specification clauses. .

[0089] 3) Relevance scoring and screening: for Each clause Calculate its relationship with the query The comprehensive relevance score is mainly evaluated from two dimensions: "semantic relevance" and "domain suitability." Knowledge with a confidence level ≥ 0.6 is selected for guidance. The results are as follows:

[0090] in For text vectors, This is a keyword database for the telecommunications field (such as {splitter, ODF, moiré pattern, label specification, ...}). Weighting coefficients ( Select , .

[0091] Filter out ( The final search results consist of entries with a threshold set to 0.6. .

[0092] 4) Prompt build: User query Combine the preset prompt templates to form the final input. Give a large model The model outputs the discrimination results and the reasons for the discrimination.

[0093] The specific implementation process of the present invention has been described above. It can be understood that, in this embodiment of the invention, negative samples are generated according to the beam splitter inspection rules, image and text data are jointly enhanced based on an image-text semantic alignment network, a multi-task optimization fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the model to obtain a beam splitter compliance discrimination model, and a knowledge graph is used to guide the beam splitter compliance discrimination model to perform beam splitter compliance discrimination on real-time query requests, thereby improving the accuracy and reliability of beam splitter inspection image compliance discrimination.

[0094] Furthermore, the embodiments of this invention effectively improve the model's recognition accuracy in scenarios where negative samples are scarce. The enhanced data maintains multimodal semantic consistency, avoids training noise, and outputs interpretable judgment reasons, facilitating review and decision-making by maintenance personnel. It guides the model to judge the compliance of optical splitters in telecommunications equipment rooms based on relevant telecommunications standards and specifications, increasing the credibility of the judgment. By dynamically adapting the weights to the differences in the difficulty of multiple tasks, it solves the problem of uneven task accuracy caused by fixed weights, improving the overall discrimination performance. At the same time, the overall architecture has good scalability, supports cross-task and cross-domain applications, and is suitable for multi-discrimination tasks that require interpretability when negative samples are missing.

[0095] Compared with the prior art, the embodiments of the present invention have the following advantages: 1) Break through the semantic consistency bottleneck of traditional multimodal enhancement, improve the alignment accuracy of image and text data, the model discrimination accuracy and interpretability, and provide a technical paradigm for fine-grained discrimination of large multimodal models in professional fields.

[0096] 2) Effectively alleviates the pain point of scarce negative samples in the inspection of optical splitters, solves the problem of "imbalance between barcode compliance and the accuracy of photocopying", reduces the cost of manual review, improves the efficiency and reliability of compliance judgment in the inspection of telecom optical splitters, and ensures the standardization of operation and maintenance processes.

[0097] 3) Integrate knowledge graph-guided, gradient consistency-consistency-based multi-task optimization and fine-tuning technologies into telecommunications operation and maintenance scenarios to provide reusable solutions for multimodal equipment inspection in the communications industry. These solutions can be extended to equipment compliance judgment in fields such as power and transportation, promoting the implementation and deepening of multimodal intelligent applications in professional fields.

[0098] like Figure 11 The diagram shown is a structural schematic of the multimodal compliance discrimination device for beam splitter inspection images provided in an embodiment of the present invention. (Refer to...) Figure 11 This invention provides a multimodal compliance discrimination device for beam splitter inspection images, comprising: The dataset construction module is used to build image and text datasets for photocopying and barcode label compliance judgment in the scenario of spectrometer inspection; The negative sample generation module is used to generate negative samples that conform to the same distribution according to the spectrometer inspection rules, and add the negative samples to the image and text dataset. The dataset augmentation module is used to perform joint augmentation of the image and text datasets using an image-text semantic alignment network to obtain a training dataset. The model fine-tuning module is used to fine-tune the multimodal large model based on the training dataset using a multi-task optimization fine-tuning algorithm based on gradient consistency constraints, so as to obtain the compliance discrimination model of the spectrometer. The model discrimination module is used to perform graph retrieval reasoning on real-time query requests using knowledge graphs to obtain the corresponding target related text. The real-time query request and the target related text are input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

[0099] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0100] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned multimodal compliance discrimination method for beam splitter inspection images. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0101] like Figure 12 The diagram shown is a hardware structure schematic of an electronic device provided in an embodiment of the present invention. (Refer to...) Figure 12 This invention provides an electronic device, comprising: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the multimodal compliance discrimination method for beam splitter inspection images according to the embodiments of this invention. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0102] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0103] like Figure 13 The diagram shown is a structural schematic of the storage medium provided in an embodiment of the present invention. (Refer to...) Figure 13 The present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs 1301, which can be executed by one or more processors to implement the above-mentioned multimodal compliance discrimination method for beam splitter inspection images.

[0104] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0105] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the above-described method for multimodal compliance determination of beam splitter inspection images.

[0106] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0107] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0109] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0110] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0111] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0114] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0115] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0118] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for multimodal compliance determination of beam splitter inspection images, characterized in that, Includes the following steps: Construct an image and text dataset for photocopying and barcode label compliance assessment in a beam splitter inspection scenario; Negative samples conforming to the same distribution are generated according to the spectrometer inspection rules, and the negative samples are added to the image and text dataset; The image-text dataset is jointly augmented using an image-text semantic alignment network to obtain a training dataset. Based on the training dataset, a multi-task optimization and fine-tuning algorithm based on gradient consistency constraints is used to fine-tune the multimodal large model to obtain a spectrometer compliance discrimination model. The real-time query request is used to perform graph retrieval reasoning to obtain the corresponding target related text. The real-time query request and the target related text are then input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

2. The method for multimodal compliance determination of beam splitter inspection images according to claim 1, characterized in that, The generation of negative samples conforming to the same distribution according to the spectrometer inspection rules specifically includes: The beam splitter inspection rules are transformed into a structured rule set, which includes rule types, rule descriptions, and image generation operators. Multiple positive sample images are obtained from the image and text dataset, and corresponding negative sample images are generated based on the image generation operator and the positive samples. Based on the rule type, the rule description, and the negative sample image, the corresponding negative sample text is generated by rewriting the large model. The negative sample is generated based on the negative sample image and the negative sample text.

3. The method for multimodal compliance determination of beam splitter inspection images according to claim 1, characterized in that, The step of using a graph-text semantic alignment network to perform joint graph-text data augmentation on the graph-text dataset to obtain a training dataset specifically includes: Obtain the first sample image and the corresponding first sample text from the image and text dataset; Image enhancement is performed on the first sample image to obtain a second sample image and corresponding image transformation parameters; The image transformation parameters are input into a pre-constructed image-text semantic alignment network to obtain the corresponding text transformation description; The first sample text is enhanced according to the text transformation description to obtain the second sample text; Image-text augmentation samples are generated based on the second sample image and the second sample text, and the image-text augmentation samples are added to the image-text dataset to obtain the training dataset.

4. The method for multimodal compliance determination of beam splitter inspection images according to claim 1, characterized in that, The step of fine-tuning the multimodal large model using a gradient consistency constraint-based multi-task optimization fine-tuning algorithm based on the training dataset to obtain a spectrometer compliance discrimination model specifically includes: Read the target image and text samples and their corresponding sample labels sequentially from the training dataset; The target image and text samples are respectively input into the re-photographing discrimination task branch and the barcode label discrimination task branch of the multimodal large model to obtain the re-photographing discrimination result and the barcode label discrimination result; The re-photographing discrimination result, the barcode label discrimination result, and the sample label are used to determine the re-photographing discrimination loss value and the barcode label discrimination loss value. Calculate the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task based on the re-photographing discrimination loss value and the barcode label discrimination loss value, respectively; Conflict detection and gradient merging are performed on the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task to obtain the target task gradient; The parameters of the multimodal large model are fine-tuned based on the target task gradient to obtain the compliance discrimination model of the spectrometer.

5. The method for multimodal compliance determination of beam splitter inspection images according to claim 4, characterized in that, The process of performing conflict detection and gradient merging on the gradient of the image reproduction discrimination task and the gradient of the barcode label discrimination task to obtain the target task gradient specifically includes: Calculate the inner product of the gradient of the re-photographing discrimination task and the gradient of the barcode label discrimination task; When the inner product is greater than or equal to 0, it is determined that there is no conflict between the re-photographing identification task and the barcode label identification task, and the gradient of the target task is determined according to the sum of the gradient of the re-photographing identification task and the gradient of the barcode label identification task. When the inner product is less than 0, it is determined that there is a conflict between the photocopying identification task and the barcode label identification task. The gradient of the photocopying identification task is then projected and corrected according to the gradient of the barcode label identification task. Finally, the target task gradient is determined based on the sum of the projected gradient of the photocopying identification task and the gradient of the barcode label identification task.

6. The method for multimodal compliance determination of beam splitter inspection images according to claim 1, characterized in that, The method of using knowledge graphs to perform graph retrieval reasoning on real-time query requests to obtain corresponding target related texts specifically includes: Entity extraction is performed on the real-time query request to obtain an entity set; The entity set is injected into a preset domain knowledge graph for multi-hop queries to obtain a set of related clauses; Calculate the relevance score between each associated clause in the associated clause set and the real-time query request; The relevant clauses with a correlation score greater than a preset first threshold are selected as the target relevant clauses.

7. A method for multimodal compliance determination of beam splitter inspection images according to any one of claims 1 to 6, characterized in that, The step of inputting the real-time query request and the target associated clause into the splitter compliance discrimination model to obtain the splitter compliance discrimination result specifically includes: Get the preset prompt template; The target prompt statement is obtained by combining the real-time query request and the target related text according to the prompt template. The target prompt statement is input into the beam splitter compliance discrimination model to obtain the beam splitter compliance discrimination result.

8. A multimodal compliance discrimination device for beam splitter inspection images, characterized in that, include: The dataset construction module is used to build image and text datasets for photocopying and barcode label compliance judgment in the scenario of spectrometer inspection; The negative sample generation module is used to generate negative samples that conform to the same distribution according to the spectrometer inspection rules, and add the negative samples to the image and text dataset; The dataset augmentation module is used to perform joint augmentation of the image and text dataset using an image-text semantic alignment network to obtain a training dataset. The model fine-tuning module is used to fine-tune the multimodal large model based on the training dataset using a multi-task optimization fine-tuning algorithm based on gradient consistency constraints, so as to obtain the spectrometer compliance discrimination model. The model discrimination module is used to perform graph retrieval reasoning on real-time query requests using knowledge graphs to obtain corresponding target related texts. The real-time query request and the target related texts are input into the splitter compliance discrimination model to obtain the splitter compliance discrimination result.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, a computer program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the computer program is executed by the processor, it implements the multimodal compliance discrimination method for beam splitter inspection images as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal compliance discrimination method for beam splitter inspection images as described in any one of claims 1 to 7.

Images