Certificate quality detection method and device, storage medium and electronic equipment

Abstract

The invention provides a certificate quality detection method and apparatus, a storage medium and an electronic device. The method comprises the steps of obtaining a to-be-detected image of a to-be-detected certificate, work record information of certificate production equipment, certificate information and corresponding certificate detection text prompt information; detecting a task based on the information; obtaining knowledge graph feature information corresponding to the detection task, image feature information of the to-be-detected image and knowledge graph feature information, and carrying out feature alignment on the information to obtain feature-aligned embedded features; based on the detection task, performing quality detection on the embedded features to obtain detection output data; performing multi-expert analysis on the detection output data by using a multi-expert-multi-modal neural network model to determine optimal output data; and determining a quality detection result according to the optimal output data by using the trained large language model. Therefore, the accurate detection result is obtained, the reason about the abnormity of the certificate is determined, the certificate making process is adjusted in time, and the certificate making efficiency and the certificate making quality are improved.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method, device, storage medium and electronic device for detecting the quality of a certificate. Background Art

[0002] With the development of social informatization, the use of various certificates is becoming more and more widespread. In the process of using certificates, the image on the surface of the certificate not only contains the user's personal information, but also contains anti-counterfeiting information for the certificate. This requires high-quality certificate production and testing in the production stage of the certificate to prevent the issued certificate surface image from containing defects or image anomalies. For this reason, certificate quality detection technology is born. At present, the certificate detection technology used in the industry mainly relies on the use of trained abnormal image detection neural network models for anomaly detection. Training such abnormal image detection neural networks often requires obtaining a large amount of sample data with standard anomalies to support the training of neural network models. Moreover, this anomaly detection method is also mostly based on predicting the anomaly map and anomaly score of the image to be detected for anomaly detection. The training of the model requires a lot of time and labor costs, and the detection process can only detect potential abnormal defects. It is difficult to give a clear abnormal judgment, and it is impossible to describe the detected abnormalities in detail, such as the abnormal position, degree of abnormality, and shape. For this reason, how to provide a certificate quality detection method that does not need to rely on massive sample data for model training and has a clear description of the abnormal detection results has become a technical problem that needs to be solved in the industry. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, device, storage medium and electronic device for detecting the quality of a certificate to at least or partially solve the above-mentioned problems.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting the quality of a certificate, comprising:

[0005] Obtain the image to be tested of the certificate to be tested, the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information;

[0006] Determine the inspection task for the document to be inspected based on the work record information of the document production equipment, the document information and the corresponding document inspection text prompt information;

[0007] Acquire knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate exception information, exception solution and time information; extract image feature information of the image to be detected; acquire text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information;

[0008] Performing feature alignment on the knowledge graph feature information, the image feature information, and the text feature information to obtain feature-aligned embedded features;

[0009] Based on the detection task, perform quality detection on the embedded features to obtain detection output data;

[0010] Using a multi-expert-multimodal neural network model, performing multi-expert analysis on the detection output data to determine optimal output data;

[0011] The trained large language model is used to determine the quality inspection result for the document to be inspected based on the optimal output data.

[0012] Optionally, in an embodiment of the present application, obtaining the image to be detected of the document to be detected includes: obtaining 3D images and RGB images of the document to be detected taken under illumination of multiple different light sources, and determining the 3D images and the RGB images as the image to be detected.

[0013] Optionally, in an embodiment of the present application, the step of aligning the knowledge graph feature information, the image feature information, and the text feature information to obtain an aligned embedded feature includes:

[0014] Using a preset interactive alignment module to receive the knowledge graph feature information, the image feature information and the text feature information for feature alignment;

[0015] Sequentially pass through the text-to-image cross attention module, the multi-layer perceptron module, and the image-to-text cross attention module to obtain the embedding feature of the feature alignment, the embedding feature includes the feature information of the output text and image alignment

[0016] Optionally, in an embodiment of the present application, before extracting features from the 3D image and the RGB image using a 3D image encoder, the method further includes: acquiring a frequency domain image of the 3D image;

[0017] Correspondingly, the extracting the image feature information of the image to be detected includes: using a 3D image encoder to perform feature extraction on the RGB image, the frequency domain image and the 3D image to obtain the image feature information of the image to be detected;

[0018] The 3D encoder is provided with at least two 3D analysis layers for performing hierarchical feature extraction on the image features.

[0019] Optionally, in an embodiment of the present application, the step of obtaining text feature information of the certificate production equipment work record information, the certificate information, and the corresponding certificate detection text prompt information includes:

[0020] Using the configured large language model, feature extraction is performed on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information to obtain the text feature information.

[0021] Optionally, in an embodiment of the present application, before using the multi-expert judgment neural network model to perform multi-expert analysis on the detection output data to determine the optimal output data, the method further includes:

[0022] Pre-constructing a basic multi-expert judgment neural network model, wherein the multi-expert detection neural network model includes a visual encoder, a visual projection layer, a word feature layer, multiple stacked language models and a multi-expert module;

[0023] Using the labeled sample graphic data, the parameters of the constructed multi-expert judgment neural network model are adjusted to generate the multi-expert-multimodal neural network model with sparse paths.

[0024] Optionally, in an embodiment of the present application, before using the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data, the method further includes:

[0025] The constructed large language model is trained by using a federated learning method to obtain the trained large language model.

[0026] Optionally, in one embodiment of the present application, a detection interactive interface is configured to support users in uploading images to be detected, inputting text prompts of items to be detected, and recording production process information.

[0027] In a second aspect, the embodiment of the present application further provides a device for detecting the quality of a certificate, comprising:

[0028] An acquisition unit, used to acquire an image of the certificate to be detected, work record information of the certificate production equipment, certificate information and corresponding certificate detection text prompt information;

[0029] A deployment unit, configured to determine a detection task for the certificate to be detected based on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information;

[0030] An extraction unit is used to obtain the knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate anomaly information, anomaly solution and time information; extract the image feature information of the image to be detected; obtain the text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information;

[0031] An alignment unit, used to perform feature alignment on the knowledge graph feature information, the image feature information and the text feature information to obtain feature-aligned embedded features;

[0032] A multi-agent detection unit, based on the detection task, performs quality detection on the embedded features and obtains detection output data;

[0033] A judgment unit, used to use a multi-expert-multimodal neural network model to perform multi-expert analysis on the detection output data to determine the best output data;

[0034] The output unit is used to use the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data.

[0035] In a third aspect, an embodiment of the present application further provides a computer storage medium having computer executable instructions stored thereon, and when the computer executable instructions are executed, any one of the certificate quality detection methods described in the first aspect of the embodiment of the present application is performed.

[0036] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute any one of the document quality detection methods described in the first aspect of the embodiment of the present application.

[0038] The present application provides a method, device, storage medium and electronic device for detecting the quality of a certificate, which obtain an image to be detected, work record information of a certificate production device, certificate information and corresponding certificate detection text prompt information of the certificate to be detected; determine the detection task for the certificate to be detected based on the work record information of the certificate production device, the certificate information and the corresponding certificate detection text prompt information; obtain knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate abnormality information, abnormal solution and time information; extract image feature information of the image to be detected; obtain text feature information of the work record information of the certificate production device, the certificate information and the corresponding certificate detection text prompt information; perform feature alignment on the knowledge graph feature information, the image feature information and the text feature information to obtain feature aligned embedded features; perform quality detection on the embedded features based on the detection task to obtain detection output data; use a multi-expert-multimodal neural network model to perform multi-expert analysis on the detection output data to determine the best output data; use a trained large language model to determine the quality detection result for the certificate to be detected based on the best output data. In this way, the embodiment of the present application can clearly obtain accurate test results, determine the cause of the abnormality of the certificate, make timely adjustments to the certificate production process, and improve the efficiency and quality of certificate production. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A schematic diagram of the workflow of a method for detecting the quality of a certificate provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a processing flow for obtaining image feature information provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of the structure of a device for detecting the quality of a certificate provided in an embodiment of the present application;

[0043] Figure 4 A schematic structural diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0045] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0046] Embodiment 1

[0047] The present application embodiment provides a method for detecting the quality of a certificate, such as Figure 1 As shown, Figure 1 A workflow diagram of a method for detecting the quality of a certificate provided in an embodiment of the present application includes:

[0048] Step S101, obtain the image to be detected, the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information of the certificate to be detected. In the embodiment of the present application, the work record information of the certificate production equipment can be obtained by extracting from the work log data of the equipment, including production process, production batch, equipment number, production material information, etc. The certificate information includes the type of certificate, the name of the user to whom the certificate belongs, the date of issuance and other certificate description information. The certificate detection text prompt information refers to the relevant information indicating what kind of detection is performed on the certificate to be detected, such as: whether the front text is missing, two-color word ghosting, front printing offset, lamination defects, black edges, portrait defects, diagonal lines, printing horizontal lines, impurities, secondary image defects, printing original image defects, endorsement text defects; whether the passport's pagination, sewing, hot stamping, laser punching are qualified, etc.; whether the punching is qualified, whether the chip embedding and chip welding are qualified. At this stage, the embodiment of the present application limits the above data to the object to be tested, so as to facilitate the subsequent acquisition of a relatively complete full-quantity test of the product and production process of the certificate to be tested, improve the information richness of the obtained test results, and provide more convenience for the production process of certificate making.

[0049] Optionally, in one embodiment of the present application, obtaining the image to be detected of the certificate to be detected includes: obtaining a 3D image and an RGB image of the certificate to be detected taken under illumination of a plurality of different light sources, and determining the 3D image and the RGB image as the image to be detected. Among them, the different light rays can be infrared light, ultraviolet light, visible light or X-ray, so as to meet the targeted detection of different types of images or anti-counterfeiting watermarks of the certificate. In this stage, the embodiment of the present application can also support all-round detection of various certificates by obtaining the 3D image and RGB image of the certificate to be detected, such as by obtaining photo images of the stages of certificate printing, punching, chip embedding, and certificate personalization. For example, an electronic card-type certificate includes a transparent layer, a front pattern printing layer, a leveling layer, a bearing layer, a leveling layer, a back pattern printing layer, and an erasable layer. In the process of certificate production, the certificate to be detected after each step of processing can be photographed. The production process of an electronic passport includes: cover, hot stamping, second cover page, third cover page, data page, inner page, sewing thread, and electronic component layer. When the electronic passport is subjected to the quality inspection described in the embodiment of the present application, each processing part of its production process is photographed. In this way, the whole process quality management of the certificate to be inspected is realized, the manufacturing quality of the certificate is improved, and the user's satisfaction with the certificate production process is improved.

[0050] Optionally, in an optional implementation method of an embodiment of the present application, obtaining the image to be detected of the document to be detected may also include: obtaining production video data of the document to be detected, extracting key frame images in the production video data as the image to be detected, so as to simplify the implementation process of obtaining the image to be detected.

[0051] Step S102, based on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information, determine the detection task for the certificate to be detected. In the embodiment of the present application, the detection task is used to characterize which parts of the image to be detected are to be quality-checked. At this stage of the embodiment of the present application, the monitoring task is determined through the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information, which can improve the accuracy of the determined detection task, reduce the difficulty of deploying the detection task, and facilitate the operation and maintenance personnel to accurately control the quality inspection and the whole process of the certificate.

[0052] Step S103, obtain the knowledge graph feature information corresponding to the detection task, the knowledge graph feature information is composed of equipment information, certificate information, certificate exception information, exception solution and time information; extract the image feature information of the image to be detected; obtain the text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information.

[0053] At this stage, the embodiment of the present application obtains the knowledge graph feature information of the corresponding detection task, so as to map and locate the cause of the abnormal quality of the certificate in a more comprehensive and accurate manner according to the knowledge graph feature information, and quickly determine whether the abnormal quality is caused by an abnormal problem in the certificate production equipment or other reasons based on the positioning information, thereby providing a more convenient operation and maintenance management experience for operation and maintenance personnel.

[0054] Specifically, in an optional implementation of an embodiment of the present application, the knowledge graph feature information corresponding to the detection task is obtained, including: pre-constructing a full knowledge graph about the certificate to be inspected, the full knowledge graph is composed of equipment information, certificate information, certificate anomaly information, anomaly solutions and production step time information; based on the detection task, intercepting the corresponding knowledge graph from the full knowledge graph, and extracting features from the intercepted knowledge graph to obtain the knowledge graph feature information. In this way, the convenience and accuracy of obtaining knowledge graph feature information can be reduced in the embodiment of the present application. In specific implementation, the process can be: through a configured knowledge graph encoder, in combination with the detection task, feature extraction is performed on the intercepted knowledge graph to obtain the knowledge graph feature information. In this way, the knowledge graph encoder can obtain the most relevant graph nodes from the input knowledge graph based on the detection task, use the relationship graph attention network to extract the features of the graph nodes, and generate knowledge graph feature information. This method is simple and easy to implement, and the accuracy of the acquired knowledge graph feature information is good. Furthermore, the method also includes inputting the extracted features into a configured knowledge graph extraction module to convert the knowledge graph features into text feature data that can be understood by the large language model to facilitate subsequent processing.

[0055] Specifically, in an optional implementation of an embodiment of the present application, based on the detection task, a corresponding knowledge graph is intercepted from the full knowledge graph, and features are extracted from the intercepted knowledge graph to obtain feature information of the knowledge graph, including: using a pre-trained knowledge graph integration neural network model to intercept a sub-knowledge graph corresponding to the detection task from the full knowledge graph, and integrating the intercepted sub-knowledge graph sections to support the acquisition of feature information of the knowledge graph. This embodiment of the present application is exemplarily described here:

[0056] Use the pre-trained knowledge graph integrated neural network model to intercept the sub-knowledge graph corresponding to the detection task from the full knowledge graph, including: intercepting the sub-graph set D retrieved according to the search term information x contained in the question prompt information corresponding to the retrieval task. This process can be expressed as: H(x,D)=∑ d∈D w d f(x,d)

[0057] Where: wd is the document weight of the text prompt information, and f(x,d) is the preset feature extraction function.

[0058] Use the multi-head attention mechanism to fuse the information of the sub-graphs in the retrieved sub-graph set D:

[0059] MultiHead(Q,K,V)=Concat(h1,...,h h )W O

[0060] Among them, each attention head is calculated as follows:

[0061] The knowledge graph obtained after fusion output, that is, the intercepted knowledge graph, can be expressed as:

[0062] P(y|x,D)=Softmax(W o ·H(x,D)+b o )

[0063] Furthermore, in an optional implementation of the embodiment of the present application, the method further includes: when pre-training the knowledge graph to integrate the neural network model, a combined loss function is used: L total =α1L CE +α2L KL +α3L reg , promote the training convergence process of the model; among them, L CE is the cross entropy loss, L KL is the KL divergence loss, L reg is a regularization term. It is used to improve the convergence efficiency of model training and the data processing accuracy of the model.

[0064] Optionally, in one embodiment of the present application, before using the 3D image encoder to perform feature extraction on the 3D image and the RGB image, the method further includes: obtaining a frequency domain image of the 3D image. Correspondingly, extracting image feature information of the image to be detected includes: using the 3D image encoder to perform feature extraction on the RGB image, the frequency domain image and the 3D image to obtain image feature information of the image to be detected; wherein the 3D encoder is provided with at least two 3D parsing layers for performing hierarchical feature extraction on the image features. Exemplarily, as Figure 2 As shown, Figure 2A processing flow diagram for obtaining image feature information provided in an embodiment of the present application, wherein the 3D encoder is provided with a first 3D analysis layer, a first normalization layer, a multi-head self-attention module, a second 3D analysis layer, a second normalization layer and a feedforward layer, and a frequency domain analysis module; the layer structure parameters of the first 3D analysis layer and the second 3D analysis layer are the same, and each 3D analysis layer is composed of a normalization layer, a linear projection layer, a 3D convolution layer, an activation function layer and a linear projection layer; the frequency domain analysis module is composed of a frequency domain extraction module, a normalization layer, a linear projection layer, an activation function layer, and a linear projection layer. The embodiment of the present application uses at least two 3D analysis layers here to perform more hierarchical feature extraction on image features, and each module extracts features at different levels; it can reduce the interference of noise and irrelevant information in the data; and it can fit more complex functional relationships. By obtaining the image feature information of the image to be detected in this way, the accuracy of the acquired image feature information can be guaranteed. In addition, in order to better analyze the image changes in different time periods and different stereoscopic angles at this stage of the embodiment of the present application, it is limited to obtain the frequency domain image information of the 3D image of the certificate to be detected first. In order to better analyze the 3D image features, 3D stereo relationships and production time relationships. This stage introduces two 3D analysis layers. Specifically, Figure 2 As shown, the specific process of the 3D encoder for hierarchical extraction of image features can be: setting the 3D image encoder to consist of a first 3D parsing unit, a first normalization layer, a multi-head self-attention module, a second 3D parsing layer, a second normalization layer and a feedforward layer. Each 3D parsing layer is specifically composed of a normalization layer, a linear down-projection layer, a 3D convolution layer, a function activation layer and a linear up-projection layer, and is connected using a cross-layer connection. The function of the 3D convolution layer is to extract 3D image features. The purpose of the down-projection layer is to reduce the original high-dimensional features to a more compact low-dimensional feature in order to control the number of newly introduced parameters. The up-projection layer is used to restore the feature dimension. In the embodiment of the present application, when acquiring frequency domain image information, a discrete Fourier transform can be used to process the 3D image to generate corresponding frequency domain image information, and a corresponding frequency domain analysis module can be configured to perform feature extraction on the frequency domain image information, wherein the frequency domain extraction module is composed of a normalization layer, a linear down-projection layer, an activation function layer, and a linear up-projection layer, and is used to process the input frequency domain image information in sequence to obtain generated frequency domain image feature information, and finally merge the frequency domain image feature information with the 3D image feature information to generate image feature information that can reflect the three-dimensional properties and production time information of the document to be detected, as a processing process participating in step S104.

[0065] Optionally, in one embodiment of the present application, obtaining text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information includes: using a configured large language model to extract features of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information to obtain the text feature information. Specifically, at this stage of the embodiment of the present application, the configured large language model can be configured with a large language model base not limited to llama3, Qwen, BaiChuan, InternLM, etc. as the base large model, which reduces the difficulty of configuring the large language model while meeting the usage scenario requirements of this stage.

[0066] Step S104, feature align the knowledge graph feature information, the image feature information and the text feature information to obtain aligned embedded features. In this stage of the embodiment of the present application, the feature information obtained is feature aligned to obtain embedded feature information, and the embedded feature information is used as a basis for supporting subsequent processing. In the image reconstruction task, the multi-scale and multi-dimensional features are aligned to correctly combine the information of each dimension. The detail expression and enhanced feature representation of the image to be detected are improved, thereby improving the clarity and quality of the image, facilitating subsequent processing, reducing the difficulty of comprehensive detection of these different types, and improving the accuracy and comprehensiveness of the document quality detection described in the embodiment of the present application.

[0067] Specifically, in an optional implementation method of an embodiment of the present application, the knowledge graph feature information, the image feature information and the text feature information are feature aligned to obtain an embedded feature of the feature alignment, including: using a preset interactive alignment module to receive the knowledge graph feature information, the image feature information and the text feature information through a linear projection layer for feature alignment; sequentially passing through a self-attention module, a text-to-image cross-attention module, a multi-layer perceptron module, and an image-to-text cross-attention module to obtain the embedded feature of the feature alignment, wherein the embedded feature includes outputting text and image aligned feature information. In this stage of the present application, in order to align the acquired text feature information g and the extracted image feature information f l Align,∈×,f l is the l-th layer output of the 3D image encoder, H l , W l , S l Respectively represent the height, width, and slice of the image feature, C l represents the intermediate channel dimension of each l-th layer output. First, the text feature information g is projected into each f through a linear projection layer. lThe self-attention module, the cross-attention module from text to image, the multi-layer perceptron module, and the cross-attention module from image to text are used to linearly project the text feature information g and the image feature f. l After self-attention and cross-attention calculations, we can get image embedding aligned with the context * . The image decoder consists of a transposed convolution layer and a convolution layer, which uses cross-layer connections internally. The embodiment of the present application uses feature alignment to enable the model to learn the correspondence and semantic association between different modalities such as text, image, and knowledge graph, enhance the correlation between modalities, improve the accuracy and quality of the results, and at the same time improve the generalization ability of the model to achieve cross-modal knowledge transfer.

[0068] Step S105: Based on the detection task, perform quality detection on the embedded feature to obtain detection output data. The embodiment of the present application defines that by performing quality detection on the embedded feature information, the detection difficulty is low and the information richness of the detection result is high.

[0069] Specifically, in an optional implementation of the embodiment of the present application, based on the detection task, the embedded feature is quality-checked to obtain the detection output data, including: based on the detection task, determining the multi-agent detection neural network model to be used; based on the detection task, using the multi-agent detection neural network model to perform targeted detection on the embedded feature to obtain the detection output data. Among them, the multi-agent detection neural network model is a neural network model that includes multiple needle-line detections for different detection items. For example, when it is determined according to the text prompt information of the detection task whether there is a front text missing, font ghosting, print offset, coating defect, etc. on the certificate table to be inspected, a multi-agent detection neural network including four neural network models, namely, a text missing detection neural network model, a font ghosting detection neural network model, a print offset detection neural network model, and a coating full inspection detection model, can be used to perform targeted detection on different parts of the embedded data, and the detection results are accurate and reliable, and the implementation difficulty is also low. Specifically, each neural network detection model can analyze the equipment operation record information and corresponding tile information related to the detection task assigned to it, and based on the tile information and equipment operation record information, query the knowledge nodes and relationship elements associated with this information from the corresponding knowledge graph feature information, so as to facilitate subsequent comprehensive and accurate data analysis.

[0070] Preferably, in a specific implementation of an embodiment of the present application, the detection task-based, multi-agent detection neural network model uses at least: a device operation status analysis neural network, an image analysis neural network, and a knowledge graph analysis neural network; wherein the device operation status analysis neural network is used to analyze the device operation status and obtain operation status analysis results, including information such as device operation logs or work records. The image analysis neural network is used to analyze the images of the captured equipment and documents through multimodal visual analysis to determine the image analysis results. Knowledge graph analysis neural network: According to the knowledge graph combined with the image and equipment operation status information of the input model, the corresponding equipment or document-related knowledge nodes and relationship elements are queried from the intercepted knowledge graph to facilitate subsequent analysis. Correspondingly, based on the detection task, the embedded features are detected using a multi-agent detection neural network model, and the embedded features are detected using a multi-agent detection neural network model to obtain detection output data, including: generating detection results according to the device operation status determined by the device operation status analysis neural network analysis, image analysis and spectrum analysis results, wherein the detection results at least include: image defect information, device abnormality information, and information that the device needs to be adjusted to solve the image defects, such as stopping or starting a certain functional component of the device, adjusting the temperature, speed, pressure value or running time of a certain device. For example, when the multi-agent detection neural network model performs relevant detection of the laminating image information, it is found that the laminating result is damaged or incomplete for many times, and relevant information for adjusting the temperature and speed parameters of the laminating machine can be generated to improve the laminating quality. When printing ghosting detection, it is found that the certificate product printed by a certain printer has multiple ghostings, and a printing parameter adjustment strategy for the printer can be generated, such as calibrating the position of the printing sensor. Of course, the embodiments of the present application are only exemplary here, and do not mean that the present application is limited to this.

[0071] Furthermore, in an optional implementation of the embodiment of the present application, the method further includes: storing the detection history information of the multi-agent detection neural network, and adjusting the output result of the multi-agent detection neural network based on the detection history information of the multi-agent detection neural network performing the detection task, so as to improve the accuracy of the detection result of the multi-agent detection neural network model. Wherein, the historical information includes short-term historical information and long-term historical information, which can be stored in the form of feature vectors, SQL databases or structured lists to achieve targeted storage and management, and facilitate system query and access.

[0072] The examples of the present application are listed here as examples

[0073] Step S106, using a multi-expert-multimodal neural network model, perform multi-expert analysis on the detection output data to determine the optimal output data. The embodiment of the present application performs multi-expert analysis on the output by setting a multi-expert-multimodal neural network model. Specifically, the top-k experts analyze the input data and summarize the output by weighted summation according to the weight of the router. When the corresponding top-k expert unit is activated, the remaining expert units remain silent, thereby ensuring the accuracy of the determined optimal output data.

[0074] Optionally, in one embodiment of the present application, before using a multi-expert judgment neural network model to perform multi-expert analysis on the detection output data and determine the optimal output data, the method further includes: pre-constructing a basic multi-expert judgment neural network model, wherein the multi-expert detection neural network model includes a 3D visual encoder, a linear projection layer, a word feature layer, a plurality of stacked language models and a multi-expert module; each expert module contains a plurality of feedforward neural networks. The pre-constructed basic multi-expert judgment neural network model has N experts when initialized, and it uses a linear layer to distribute the input information to different experts for analysis and processing. Specifically, the basic multi-expert judgment model calculates the routing values ​​of N experts for each input data x: r=softmax(x·W r );W r Represents the weight matrix of the router, each element r in r i represents the probability of activating the i-th expert. The Top-k strategy is used to determine the activated experts with the highest k routing values. The output data y is the weighted sum of the expertise of the activated experts according to the routing value. y = ∑ ＝1 -()·()。 Use the labeled sample graphic data to adjust the parameters of the constructed multi-expert judgment neural network model to generate the multi-expert-multimodal neural network model with sparse paths. The embodiment of the present application generates a sparse model (sparse model) so that the constructed multi-expert-multimodal neural network model does not need to use every part of the model to process each input, but uses conditional computational learning to route the input to different "experts" in the huge network. In this way, the model can be enlarged while keeping the computational cost unchanged. This is not only an effective and environmentally friendly way to scale the model, but also can improve the performance of model judgment and analysis.

[0075] Step S107: Use the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data. Specifically, the quality inspection result may include abnormal detection image results, abnormal detection text description information, and inspection reports and solutions containing other information. This provides the operation and maintenance personnel with inspection results containing rich information to meet the high inspection requirements of the operation and maintenance personnel.

[0076] Optionally, in one embodiment of the present application, before using the trained large language model to determine the quality inspection result of the document to be inspected based on the optimal output data, the method also includes: using federated learning to train the pre-built large language model to obtain the trained large language model.

[0077] This application implements a specific implementation method as exemplified herein, and explains how to train a pre-built large language model through the federated learning method:

[0078] During model training, since the training sample data used may contain a lot of real user's private information, in order to protect the security of user's private information, the embodiment of the present application is defined here, using the method of federated learning to train the pre-built large language model, which can be: at the user's location, a lightweight local model is used to extract the training sample data knowledge set from private data, and an adaptive multi-task multi-modal hybrid expert neural network model is used to integrate the knowledge into the training data set to support the training of the basic model. The basic model can learn user data and enhance its own data processing capabilities by using the training data set of private user data for model training. This method not only protects privacy, but also enhances the ability of large language models to handle complex anomaly detection tasks involving multi-modality.

[0079] Specifically, the above method can be: configure the client to integrate the certificate knowledge from the user data in a federated learning manner to expand and increase the output processing capacity of the language model. For N clients, each client represents a certificate production center, including private equipment and certificate data D n Assume that the private dataset D n Contains all abnormal patterns {M1, ..., M M}, such as device printer exception, database connection exception, document lamination exception, and can execute all tasks {T1, ..., T T}, such as ID chip detection, device operation status detection, chip packaging detection, etc. Each client trains a model f n =[ENC n ();DEC n ()]Use data D n, where ENC n () is a set of multimodal encoders, and DECn() is a set of multitask decoders / predictors. Therefore, f n The model parameters θ n Can be divided into encoder and for the decoder These two parameters will be further uploaded to the server.

[0080] Setting up the server: Deploy a basic model for device and document anomaly detection on the server, denoted as F. The equipment maintenance and certificate knowledge represented by F are injected into F, and new knowledge is absorbed from F and then updated These updated encoders and aggregated decoders will be distributed to the corresponding clients for learning in the next round of communication. In order to facilitate the update of client parameters, a small amount of public basic data is placed on the server at this stage, denoted as D p .

[0081] Extract knowledge from the client: Configure the client neural network model, including M modality-specific encoders (there are 3 corresponding modalities in the embodiment of this application, namely text data, knowledge graph data and image data) and T task-specific decoders. Use the following loss function to train each client neural network model:

[0082]

[0083]

[0084] in, is a dataset for a specific task, is a set of data, features and corresponding ground truth values ​​in the dataset, and l t is a task-specific loss function, such as cross entropy. ENC n,m The parameters are The modal encoder of Mm. DEC n,t The parameters are The decoder of the tth task. ENC n The number of mid-modal encoders is determined by the input data, while the number of tasks determines the number of task-oriented decoders. After local training, the encoder and decoder parameters are Upload to the server.

[0085] Aggregate knowledge to the server: Assume that the detection capability of F is better than the uploaded decoder Useful knowledge is mainly contained in the encoder On the server side, the goal of this stage of the embodiment of the present application is to use public data Dp to store device operation and certificate production knowledge Injected into the large language model F. Before injection, first aggregate the knowledge information uploaded from each client in a traditional federated learning manner, such as FedAvg or FedProx, namely:

[0086]

[0087] fFL can be replaced by any federated learning method, such as personalized FL method, differential privacy-based FL method, or adaptive FL method.

[0088] In the embodiment of the present application, because the large language model F cannot directly use these different modality-specific encoders, it is difficult to effectively inject knowledge θe. The embodiment of the present application defines here that the feature alignment strategy is trained by connecting the modality embedding with the task prompt to the large language model F. Subsequently, the original multimodal multitask hybrid expert is embedded into the base model. The hybrid expert model allows the base model F to adaptively select a specific expert system given different task and mode combinations.

[0089] The process of knowledge injection is as follows:

[0090] Step 1: Feature alignment. For each input data from the tth task First, use the aggregated encoder θ e Get its feature representation, that is Where g(·) is a linear mapping function. Use the encoder of F to embed the task prompt p t , that is, P t =EMB f (p t ), where EMB f () is the text embedding layer of F, and then the data features and the task prompt word feature P t The concatenation of is taken as the input of F encoder, denoted as

[0091] Step 2: A multimodal multi-task mixture of experts module that allows the model to dynamically distinguish tasks.

[0092] The existing solution is to directly use the aligned features However, this end-to-end fine-tuning method not only has weak discriminability for different tasks, but also ignores the generalization ability of the model to unseen tasks. t and task T t Related modal description M tAs input, we calculate the relevance of each expert for a given task and modality, where M t is related to task T t The concatenation of all relevant modal descriptions. t and M t First, it is encoded by the embedding layer of the base model F; and then processed to output the weights for expert selection as follows:

[0093] α t =softmax(MLP(Pooling(β t ))),

[0094]

[0095] In the above formula, α t ∈R P ,P is the number of experts. q , W k , W v represents the attention matrix, d k is the dimension size.

[0096] This module effectively integrates the injected knowledge managed by two independent routers, resulting in a more streamlined and context-aware weight calculation. Output α t Represents an attention-weighted selection of experts optimized for modality and specific tasks. This approach provides the flexibility required to handle complex scenarios by selecting appropriate experts based on the context.

[0097] Step 3: Efficient fine-tuning of hybrid expert parameters based on LoRA. Finally, in the embodiment of the present application, based on LoRA and the learned hybrid expert, the weights use the formula in step 2 to generate the representations of each layer of the forward pass in F, as shown below:

[0098]

[0099] Where W f represents the frozen parameter of F, B p A p Represents the low-order adaptive module as the p-th expert system. is the aligned feature, and the final output of F and the standard answer will be used Fine-tune the model.

[0100] During training, the modality-specific encoder θ e Gradually align with the multimodal large model F that contains rich knowledge obtained through pre-training. The prior knowledge in F is also extracted during the knowledge injection process in the form of adjusted parameters restored in the encoder. This method updates the encoder θ e and the aggregated decoder θd It is passed back to the local end and knowledge-driven iterative training is performed until convergence, thereby updating the local model and facilitating the next round of knowledge injection.

[0101] The embodiment of the present application ensures the privacy of the acquired training sample data through the above-mentioned example method, while improving the data processing capability of the trained large language model.

[0102] Optionally, in one embodiment of the present application, a detection interactive interface is configured to support users to upload images to be detected, input text prompts of items to be detected, and record information of the production process, so as to facilitate human-computer interaction and provide richer detection scenario support for operation and maintenance personnel.

[0103] The present application provides a method for detecting the quality of a certificate, which comprises obtaining an image to be detected, work record information of a certificate production device, certificate information and corresponding certificate detection text prompt information of the certificate to be detected; determining a detection task for the certificate to be detected based on the work record information of the certificate production device, the certificate information and the corresponding certificate detection text prompt information; obtaining knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of device information, certificate information, certificate anomaly information, anomaly solution and time information; extracting image feature information of the image to be detected; obtaining text feature information of the work record information of the certificate production device, the certificate information and the corresponding certificate detection text prompt information; aligning the knowledge graph feature information, the image feature information and the text feature information to obtain feature-aligned embedded features; performing quality detection on the embedded features based on the detection task to obtain detection output data; performing multi-expert analysis on the detection output data using a multi-expert-multimodal neural network model to determine optimal output data; and determining a quality detection result for the certificate to be detected based on the optimal output data using a trained large language model. In this way, the embodiment of the present application can clearly obtain accurate test results, determine the cause of the abnormality of the certificate, make timely adjustments to the certificate production process, and improve the efficiency and quality of certificate production.

[0104] Embodiment 2

[0105] Based on the method for detecting the quality of a certificate provided in the first embodiment of the present application, the embodiment of the present application further provides a device for detecting the quality of a certificate, such as Figure 2 As shown, Figure 2 A schematic diagram of the structure of a device 20 for detecting the quality of a certificate provided in an embodiment of the present application, wherein the device 20 for detecting the quality of a certificate comprises:

[0106] The acquisition unit 201 is used to acquire the image to be detected of the certificate to be detected, the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information;

[0107] A deployment unit 202 is used to determine a detection task for the certificate to be detected based on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information;

[0108] The extraction unit 203 is used to obtain the knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate exception information, exception solution and time information; extract the image feature information of the image to be detected; obtain the text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information;

[0109] An alignment unit 204 is used to perform feature alignment on the knowledge graph feature information, the image feature information and the text feature information to obtain feature-aligned embedded features;

[0110] A multi-agent detection unit 205 performs quality detection on the embedded features based on the detection task to obtain detection output data;

[0111] The analysis unit 206 is used to use a multi-expert-multimodal neural network model to perform multi-expert analysis on the detection output data to determine the best output data;

[0112] The output unit 207 is used to use the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data.

[0113] Optionally, in an embodiment of the present application, the acquisition unit 201 is further used to: acquire 3D images and RGB images of the document to be detected taken under illumination of multiple different light sources, and determine the 3D images and the RGB images as the image to be detected.

[0114] Optionally, in one embodiment of the present application, the alignment unit 204 is further used to: use a preset interactive alignment module to receive the knowledge graph feature information, the image feature information and the text feature information for feature alignment; sequentially pass through a text-to-image cross attention module, a multi-layer perceptron module, and an image-to-text cross attention module to obtain an embedded feature of the feature alignment, wherein the embedded feature includes outputting feature information of text and image alignment.

[0115] Optionally, in one embodiment of the present application, the device 20 also includes a conversion unit (not shown in the accompanying drawings), which, before using the 3D image encoder to extract features from the 3D image and the RGB image: obtains a frequency domain image of the 3D image; the extraction unit 203 is also used to: use the 3D image encoder to extract features from the RGB image, the frequency domain image and the 3D image to obtain image feature information of the image to be detected; wherein the 3D encoder is composed of a first 3D parsing layer, a multi-head self-attention module, a second 3D parsing layer, a batch normalization layer and a feedforward layer; the layer structure parameters of the first 3D parsing layer and the second 3D parsing layer are the same, and each 3D parsing layer is composed of a normalization layer, a linear projection layer, a 3D convolution layer, an activation layer and a linear up-projection layer.

[0116] Optionally, in one embodiment of the present application, the extraction unit 203 is also used to: use a configured large language model to perform feature extraction on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information to obtain the text feature information.

[0117] Optionally, in one embodiment of the present application, the device 20 is also used to construct a unit (not shown in the drawings), which is used to pre-construct a basic multi-expert judgment neural network model before using the multi-expert judgment neural network model to perform multi-expert analysis on the detection output data to determine the optimal output data, and the multi-expert detection neural network model includes a visual encoder, a visual projection layer, a word feature layer, multiple stacked language models and a multi-expert module; using the labeled sample graphic data, the parameters of the constructed multi-expert judgment neural network model are adjusted to generate the multi-expert-multimodal neural network model with a sparse path.

[0118] Optionally, in one embodiment of the present application, the device 20 also includes a training unit (not shown in the drawings), which is used to use the trained large language model to train the constructed large language model using a federated learning method before determining the quality inspection result of the document to be inspected based on the optimal output data to obtain the trained large language model.

[0119] Optionally, in one embodiment of the present application, the device 20 also includes an interactive unit (not shown in the drawings), which has a user-configured detection interactive interface that supports users uploading images to be detected, entering text prompts for items to be detected, and recording production process information.

[0120] The present application provides a device for detecting the quality of a certificate, wherein an acquisition unit is provided to acquire an image to be detected, work record information of a certificate production device, certificate information, and corresponding certificate detection text prompt information of the certificate to be detected; a deployment unit is provided to determine a detection task for the certificate to be detected based on the work record information of the certificate production device, the certificate information, and the corresponding certificate detection text prompt information; an extraction unit is provided to acquire knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of device information, certificate information, certificate abnormality information, abnormality solution, and time information; image feature information of the image to be detected is extracted; the work record information of the certificate production device, the certificate information, and the corresponding certificate detection text prompt information are acquired. The text feature information of this prompt information; an alignment unit is set to perform feature alignment on the knowledge graph feature information, the image feature information and the text feature information to obtain the embedded features of feature alignment; a multi-agent detection unit is set to perform quality detection on the embedded features based on the detection task to obtain detection output data; a research and judgment unit is set to use a multi-expert-multimodal neural network model to perform multi-expert analysis on the detection output data to determine the best output data; an output unit is set to use a trained large language model to determine the quality detection results for the certificate to be detected based on the best output data, including marking abnormal results on the image, generating a detection report, including abnormal description, cause analysis and providing solutions, etc. In this way, the embodiment of the present application can clearly obtain accurate detection results, determine the reasons for the abnormality of the certificate, make timely adjustments to the certificate making process, and improve the efficiency and quality of certificate making.

[0121] Embodiment 3

[0122] An embodiment of the present application also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements any one of the document quality detection methods described in the first embodiment of the present application.

[0123] Embodiment 4:

[0124] The present application also provides an electronic device, such as Figure 3 As shown, Figure 3 A schematic diagram of the structure of an electronic device 30 is provided for an embodiment of the present application. The electronic device 30 includes:

[0125] One or more processors 301, communication interface 302, memory 303 and communication bus 304, the processor 301, memory 303 and communication interface 302 communicate with each other via the communication bus 304;

[0126] Memory 303, used to store one or more programs;

[0127] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement any one of the methods for detecting the quality of a certificate as described in the first embodiment of the present application.

[0128] So far, the application has described specific embodiments of the subject matter. In some cases, the actions recorded in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0129] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system layer on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0130] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0131] The system layers, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with corresponding functions having corresponding prefix names.

[0132] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0133] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, system layers or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0135] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific transactions or implement specific abstract data types. The present application may also be practiced in distributed computing environments where transactions are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0136] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system layer embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0137] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for detecting the quality of a certificate, characterized in that: include: Obtain the image to be tested of the certificate to be tested, the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information; Determine the inspection task for the document to be inspected based on the work record information of the document production equipment, the document information and the corresponding document inspection text prompt information; Acquire knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate exception information, exception solution and time information; extract image feature information of the image to be detected; acquire text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information; Performing feature alignment on the knowledge graph feature information, the image feature information, and the text feature information to obtain feature-aligned embedded features; Based on the detection task, perform quality detection on the embedded features to obtain detection output data; Using a multi-expert-multimodal neural network model, performing multi-expert analysis on the detection output data to determine optimal output data; The trained large language model is used to determine the quality inspection result for the document to be inspected based on the optimal output data.

2. The method for detecting the quality of a certificate according to claim 1, characterized in that: The obtaining of the image to be detected of the document to be detected includes: obtaining a 3D image and an RGB image of the document to be detected photographed under illumination of a plurality of different light sources, and determining the 3D image and the RGB image as the image to be detected.

3. The method for detecting the quality of a certificate according to claim 2, characterized in that: The step of aligning the knowledge graph feature information, the image feature information, and the text feature information to obtain aligned embedded features includes: Using a preset interactive alignment module to receive the knowledge graph feature information, the image feature information and the text feature information for feature alignment; [HZTC-HXJKJD241006][HS2411224CCN] sequentially passes through a text-to-image cross-attention module, a multi-layer perceptron module, and an image-to-text cross-attention module to obtain the embedded features of the feature alignment, wherein the embedded features include feature information of the output text and image alignment.

4. The method for detecting the quality of a certificate according to claim 2, characterized in that: Before extracting features from the 3D image and the RGB image using the 3D image encoder, the method further includes: acquiring a frequency domain image of the 3D image; Correspondingly, the extracting the image feature information of the image to be detected includes: using a 3D image encoder to perform feature extraction on the RGB image, the frequency domain image and the 3D image to obtain the image feature information of the image to be detected; The 3D encoder is provided with at least two 3D analysis layers for performing hierarchical feature extraction on the image features.

5. The method for detecting the quality of a certificate according to claim 1, characterized in that: The obtaining of the working record information of the certificate production equipment, the certificate information and the text feature information of the corresponding certificate detection text prompt information includes: Using the configured large language model, feature extraction is performed on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information to obtain the text feature information.

6. The method for detecting the quality of a certificate according to claim 1, characterized in that: Before using the multi-expert judgment neural network model to perform multi-expert analysis on the detection output data to determine the optimal output data, the method further includes: Pre-constructing a basic multi-expert judgment neural network model, the multi-expert detection neural network model includes a 3D visual encoder, a linear projection layer, a word feature layer, multiple stacked language models and multi-expert modules, each expert module contains multiple feedforward neural networks; Using the labeled sample graphic data, the parameters of the constructed multi-expert judgment neural network model are adjusted to generate the multi-expert-multimodal neural network model with sparse paths.

7. The method for detecting the quality of a certificate according to claim 1, characterized in that: Before using the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data, the method further includes: Use federated learning to train the pre-built large language model to obtain the trained large language model 8. The method for detecting the quality of a certificate according to claim 1, characterized in that: Configure the detection interactive interface to support users to upload images to be detected, enter text prompts for items to be detected, and record information about the production process.

9. A device for detecting the quality of a certificate, characterized in that: include: An acquisition unit, used to acquire an image of the certificate to be detected, work record information of the certificate production equipment, certificate information and corresponding certificate detection text prompt information; A deployment unit, configured to determine a detection task for the certificate to be detected based on the work record information of the certificate production equipment, the certificate information and the corresponding certificate detection text prompt information; An extraction unit is used to obtain the knowledge graph feature information corresponding to the detection task, wherein the knowledge graph feature information is composed of equipment information, certificate information, certificate anomaly information, anomaly solution and time information; extract the image feature information of the image to be detected; obtain the text feature information of the certificate production equipment work record information, the certificate information and the corresponding certificate detection text prompt information; An alignment unit, used to perform feature alignment on the knowledge graph feature information, the image feature information and the text feature information to obtain feature-aligned embedded features; A multi-agent detection module performs quality detection on the embedded features based on the detection task and obtains detection output data; A judgment unit, used to use a multi-expert-multimodal neural network model to perform multi-expert analysis on the detection output data to determine the best output data; The output unit is used to use the trained large language model to determine the quality inspection result for the document to be inspected based on the optimal output data.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the method for detecting the quality of a certificate as described in any one of claims 1 to 8 is executed.

Images