Complex scene speculation decoding method and system based on calligraphy and student large model

Abstract

The invention discloses a complex scene speculation decoding method and system based on a calligraphy large model, and belongs to the field of real scene exception understanding. Acquiring multi-scene industrial anomaly detection image data and a corresponding text problem task; obtaining text embedding and a corresponding last layer hiding state based on a text problem task, and obtaining a compressed visual token and a global visual feature vector based on an image sample; generating draft probability distribution and a to-be-verified token sequence through a draft model according to the compressed visual token, the global visual feature vector, text embedding and a corresponding last layer hiding state, calculating to obtain target probability distribution by a target model based on the to-be-verified token sequence, and comparing the draft probability distribution with the target probability distribution to judge whether a corresponding candidate token is reserved or not; according to the method, speculative decoding is introduced in the field of industrial detection, so that efficient multi-modal anomaly understanding and positioning are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of complex defect scene understanding, and in particular to a complex scene decoding method and system based on a scholar large model. BACKGROUND

[0002] At present, industrial anomaly detection has become a key requirement in modern production processes, triggering widespread attention in the field of complex scene understanding. Industrial anomaly detection uses automation technology to promote real-time monitoring and identification of product anomalies, so that timely intervention can be made when defects occur to prevent errors from further spreading to maintain product quality. With the rise of multi-modal large language models, recent research has attempted to apply multi-modal large language models to industrial anomaly detection tasks, not only achieving rapid discrimination of current scene defects, but also detailing the characteristics and categories of defects, giving the location of the defect area, analyzing the specific causes of defects and the possible impact, i.e. extending the simple industrial anomaly detection task to the understanding task in complex scenes. This type of method using multi-modal large language models can directly address the root cause of the problem, making it more comprehensive in solving problems, and can achieve better benefits when applied in real-world environments.

[0003] Among them, the scholar large model is a typical multi-modal large language model, and its structure mainly includes a visual encoder, a text embedding layer, a decoder, and a classification head. The visual encoder is responsible for feature extraction of input image or video data, converting it into a high-dimensional semantic representation; the text embedding layer is responsible for converting discrete text symbols (Token) into continuous vector representation, the decoder is responsible for extracting the hidden state sequence after deep processing; the classification head projects the hidden state to the dimension of the entire vocabulary, outputting each word as the probability of the next word, achieving autoregressive generation.

[0004] Although the method combined with multi-modal large language models can solve the complex scene understanding task more deeply, it often faces a more difficult problem: there is a significant efficiency bottleneck in the reasoning process. This type of model usually generates text token by token in an autoregressive manner, and each step needs to perform complete forward propagation, making the reasoning process computationally expensive and slow in response, making it difficult to meet the real-time needs of actual deployment and interactive applications. At the same time, the model uses a visual encoder to process input images into a number of visual tokens, which are then input together with text tokens into a large language model for joint modeling. As the parameter size increases, the number of visual tokens continues to increase, and the computational and memory overhead of the model also significantly increases, further limiting the application of multi-modal large language models in real-world scenarios.

[0005] To address the aforementioned issues, speculative decoding has been proposed as a lossless acceleration strategy. Its core idea is to reduce redundant computations in the autoregressive inference process of large language models while maintaining the quality and distribution consistency of model generation. This method introduces a lightweight draft model and a speculative generation and verification mechanism in the multimodal decoding process. It rapidly predicts multiple candidate tokens, and then the original large target model performs parallel verification of the prediction results, thereby significantly improving inference efficiency without changing the output. In recent years, this strategy has demonstrated stable acceleration effects in text generation tasks, enabling large language models to achieve several times faster decoding while maintaining generation consistency. It not only ensures cross-modal semantic alignment and output fidelity but also effectively alleviates problems such as low inference efficiency.

[0006] However, in the field of industrial anomaly detection, multimodal large language model methods still have great potential for inference efficiency when dealing with complex scene understanding tasks due to the lack of effective lossless acceleration techniques. Summary of the Invention

[0007] To overcome the problem of low inference efficiency of existing multimodal large language models when handling complex scene understanding tasks on current datasets, this invention provides a method and system for speculative decoding of complex scenes based on the Shusheng large model, so as to achieve efficient complex scene understanding.

[0008] In a first aspect, this invention proposes a speculative decoding method for complex scenes based on a large-scale scholar model, wherein the large-scale scholar model includes at least a visual encoder, a text embedding layer, a multi-layer decoder, and a classification head; the method includes: S1, acquire multi-scenario industrial anomaly detection image data and corresponding text question tasks for image samples; S2, the text question task input into the Shusheng model is converted into a text embedding. The multi-layer decoder receives the text embedding and outputs the corresponding last hidden state. S3, input the image sample into the visual encoder of the Shusheng large model to obtain the image token, and input the image token into the visual adapter of the draft model to obtain the compressed visual token and the global visual feature vector. S4, the compressed visual token, text embedding, and corresponding final hidden state are uniformly input into the single-layer decoder of the draft model. After the initial final hidden state is enhanced by introducing a global visual feature vector, the enhanced final hidden state is input into the classification head of the draft model to obtain the draft probability distribution. Further, the token sequence to be verified is obtained through autoregression. The draft probability distribution contains the probability of each candidate token. The token sequence to be verified contains a batch of candidate tokens. S5, input the token sequence to be verified into the classification head of the Shusheng big model, and output the target probability distribution; the target probability distribution contains the probability of each candidate token in the token sequence to be verified; S6 compares the target probability distribution with the draft probability distribution token by token to determine whether the corresponding candidate token should be retained, and finally outputs all retained tokens.

[0009] Furthermore, the draft model consists of a lightweight visual adapter, a single-layer decoder, and a classification head.

[0010] Furthermore, the visual adapter is a Transformer encoder with a batch of learnable query vectors, comprising a linear projection layer and an attention layer.

[0011] Furthermore, S3 specifically refers to: S301, Input the image sample into the visual encoder of the Shusheng large model to obtain the image token sequence; S302, input the image token sequence into the linear projection layer of the visual adapter, and use the key weight matrix and value weight matrix to transform the image token sequence into matrix K and matrix V respectively; S303, the first n-1 query vectors of the visual adapter are used as compressed query vectors, the nth query vector is used as the global query vector, and all query vectors are used to form a matrix Q; S304. Input matrices Q, K, and V into the attention layer of the visual adapter, and use the cross-attention mechanism to calculate n-1 compressed visual tokens and 1 global visual feature vector.

[0012] Furthermore, S6 specifically includes: S601, compare the probability values ​​of each candidate token in the draft probability distribution. and the probability value in the target probability distribution ,like If the candidate token is found to be valid, the corresponding candidate token is retained and the calculation ends; otherwise, the corresponding candidate token and all subsequent candidate tokens in the token sequence to be verified are deleted, and the next step of the calculation is performed. S602, the classification head of the scholar model resamples all the retained candidate tokens to obtain a new token, which is then combined with all the retained candidate tokens to form a retained token sequence; S603, repeat S4-S6 for iteration until the semantic integrity of the token sequence is preserved; S604 outputs the final sequence of retained tokens.

[0013] Furthermore, in S4, the global visual feature vector is used to enhance the last hidden state of the input single-layer decoder; at the i-th text position, the enhanced last hidden state... The formula for calculation is: ; in It is the last hidden state of the i-th text position. It is a projection matrix. It is a global visual feature vector.

[0014] Furthermore, the first generation probability is the probability that the draft model generates the corresponding candidate token in the current context; the second generation probability is the probability that the scholar model generates the corresponding candidate token in the current context.

[0015] Furthermore, the text-based problem task includes three sub-tasks: defect identification, defect localization, and defect classification. The defect identification task requires the multimodal large language model to determine whether a defect exists in the current scene. The defect localization task requires the multimodal large language model to provide the pixel-level coordinates of the defect area. The defect classification task requires the multimodal large language model to analyze the defect features and infer the classification result after confirming the existence of a defect and locating the defect area.

[0016] Furthermore, the subtask consists of a question and options, and the options include distractors.

[0017] Secondly, this invention proposes a speculative decoding system for complex scenes based on the scholar's big model, which is used to implement the aforementioned speculative decoding method for complex scenes based on the scholar's big model.

[0018] Compared with the prior art, the present invention has the following characteristics: This invention is a method and system for speculative decoding of complex scenarios based on a large language model. In its implementation, this invention uses speculative decoding and a multimodal large language model.

[0019] (1) By using speculative decoding, the present invention not only extracts rich compressed visual tokens and global visual feature vectors from image tokens, but also generates candidate tokens in sequence and iterates them into the final generated sequence through a verification mechanism, which significantly improves the model inference efficiency, thereby achieving efficient defect area localization, description and analysis.

[0020] (2) By using a multimodal large language model, this invention leverages the powerful visual understanding and text reasoning capabilities of the Shusheng large model to extract rich defect features and location information from complex scene representations, unifying different downstream abnormal tasks into a model-oriented question-and-answer format, thereby achieving comprehensive reasoning on different types of data and effectively testing the model's performance in each type of scene and corresponding task.

[0021] (3) By combining speculative decoding and multimodal large language model, this invention can fully extract spatial, feature and semantic information in complex scenarios and conduct inference tests on various complex scenario datasets, thereby achieving efficient defect area localization, description and analysis. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall framework of the present invention.

[0023] Figure 2 This is a detailed flowchart of the present invention. Detailed Implementation

[0024] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 and Figure 2 As shown, the present invention proposes a speculative decoding method for complex scenes based on a scholar-like large model, which mainly includes the following steps: Step 1 involves acquiring multi-scene industrial anomaly detection image data from a public dataset, along with corresponding text-based question tasks for each image sample. These text-based question tasks primarily encompass three sub-tasks: defect identification, defect localization, and defect classification. The defect identification task requires a multimodal language model to determine whether a defect exists in the current image scene, i.e., unnatural features or semantically confused elements. The defect localization task requires the multimodal language model to provide pixel-level coordinates of each defect region in the complex image scene during defect analysis, i.e., the coordinates of the top-left and bottom-right corners of the defect region's bounding box. The defect classification task requires the multimodal language model, combined with text prompts, to analyze the features of all defects in the image and infer the classification result, assuming the presence and location of defects in the current image sample. Each sub-task consists of a question and options, and each question contains challenging distractors. Step 2: Use a multimodal large language model as the target model (in this specific embodiment, the target model is the Shusheng multimodal large language model). Input the text question into the text embedding layer of the target model. After the text embedding layer outputs the text embedding, the decoder of the target model further obtains the corresponding final hidden state based on the text embedding.

[0026] Step 3: The draft model contains a lightweight visual adapter and a lightweight version of the target model. The visual encoder of the target model is used to process the image samples into image tokens. The image tokens are then input into the lightweight visual adapter of the draft model, and the adapter extracts the compressed visual token and global visual feature vector from the image tokens.

[0027] The specific steps include: 3.1) The draft model consists of a lightweight visual adapter, a single-layer decoder, and a classification head, where the lightweight visual adapter is a Transformer encoder with n learnable query vectors.

[0028] 3.2) Input the image sample into the visual encoder of the target model. The visual encoder processes the image sample into an image token.

[0029] 3.3) After the image token is input into the visual adapter, the linear projection layer inside the adapter passes the image token through the weight matrix respectively. and weight matrix The corresponding matrices K and V are calculated.

[0030] 3.4) The first n-1 query vectors in the lightweight visual adapter are used as compressed query vectors for subsequent generation of compressed visual tokens. The nth query vector is used as the global query vector for subsequent extraction of global visual features to generate a global visual feature vector. All query vectors form a matrix Q.

[0031] 3.5) Input matrices Q, K and V into the attention layer inside the visual adapter, and use the cross-attention mechanism to calculate n-1 compressed visual tokens and 1 global visual feature vector.

[0032] Step 4: Input the compressed visual token, global visual feature vector, text embedding and corresponding final hidden state into the decoder of the draft model to generate the final hidden state. Input the classification head of the draft model to obtain the probability of each candidate token. Then, through autoregression, obtain the token sequence to be verified composed of all candidate tokens.

[0033] The specific steps include: 4.1) Taking the text embedding and the corresponding last hidden state from step 2, the n-1 compressed visual tokens from step 3 and one global visual feature vector as input, the last hidden state is first generated in the decoder of the draft model based on the text embedding and the corresponding last hidden state from step 2 and the n-1 compressed visual tokens from step 3. Then, the global visual feature vector is introduced to enhance the last hidden state to obtain the enhanced last hidden state.

[0034] The global visual feature vector is used to enhance the initial hidden state, ensuring that the generated result remains consistent with the semantics of the image.

[0035] At the i-th text position, the enhanced last hidden state The formula for calculation is: in It is the original last hidden state of the i-th text position. It is a learnable projection matrix. It is a global visual feature vector.

[0036] 4.2) Input the enhanced last hidden state into the classification head of the draft model to obtain the probability of each candidate token. Autoregression is then used to obtain a sequence of tokens to be verified, consisting of candidate tokens.

[0037] Step 5: Input the sequence of tokens to be verified into the classification head of the target model, and calculate the probability of each candidate token in the sequence. .

[0038] Step 6: Compare the results of each candidate token in the sequence of tokens to be verified. and To determine whether the candidate token should be retained, if not, the token sequence to be verified is iteratively updated until the semantics of the retained token sequence are complete, and the retained token sequence is output.

[0039] The specific steps include: 6.1) Compare the probabilities of each candidate token. With probability ,like This indicates that the candidate token meets the requirements, and the candidate token is retained; if This indicates that the candidate token does not meet the requirements. The candidate token and all candidate tokens in the sequence following this candidate token are discarded. The target model classification head resamples and generates a new token based on all the retained candidate tokens, which together with all the retained candidate tokens form the retained token sequence.

[0040] 6.2) Repeat steps five and six to iterate until the semantic integrity of the token sequence is preserved.

[0041] 6.3) Output the final sequence of retained tokens, which is the final reasoning process and conclusion generated by the multimodal large language model based on the input image and text question task.

[0042] To verify the effectiveness of the method of the present invention, the following experiment was further designed. The specific steps in the embodiments will not be repeated.

[0043] This invention selects ten publicly available datasets for complex scenarios: MVTecAD, MVTec-LOCO, MSD, MPDD, MIAD, VisA, GoodsAD, DS, NanoTwice, and one self-built dataset. All samples in these datasets are used to test a unified text-based problem task encompassing defect localization, defect discrimination, and defect classification. InternVL3.5-4B is chosen as the multimodal large language model. To objectively evaluate the performance of this invention, two evaluation metrics, Speedup ratio and Average Acceptance Length, are used to assess the lossless acceleration effect on the selected test sets. Experimental results show that, compared to the method without this invention, the speedup ratio reaches 1.36, and the Average Acceptance Length reaches 0.85. This demonstrates that when applying a multimodal large language model to anomaly understanding tasks in complex scenarios, this invention combines speculative decoding to extract semantically rich compressed visual tokens and global visual feature vectors from image samples, and then uses a draft model to generate an iterative sequence of tokens to be verified. This effectively improves the model's inference efficiency, enabling efficient inference and analysis.

[0044] Based on the same inventive concept, this invention also provides a speculative decoding system for complex scenes based on a scholar-like large model, comprising: The data acquisition module is used to acquire multi-scenario industrial anomaly detection image data and corresponding text question tasks for image samples. The text processing module is used to convert the text problem task input into the Shusheng model's text embedding layer into text embeddings. The decoder receives the text embeddings and outputs the corresponding hidden state of the last layer. The image processing module is used to input image samples into the visual encoder of the large model to obtain image tokens. The image tokens are further input into the visual adapter to obtain compressed visual tokens and global visual feature vectors. The draft model processing module is used to input the compressed visual token, text embedding, and corresponding final hidden state into the single-layer decoder of the draft model. After introducing a global visual feature vector to enhance the generated initial final hidden state, the enhanced final hidden state is input into the classification head of the draft model to obtain the draft probability distribution. Further, an autoregression is used to obtain the token sequence to be verified. The draft probability distribution contains the probability of each candidate token. The token sequence to be verified contains a batch of candidate tokens. The target model processing module is used to input the token sequence to be verified into the classification head of the Shusheng model and output the target probability distribution; the target probability distribution contains the probability of each candidate token in the token sequence to be verified. The probability comparison module compares the draft probability distribution with the target probability distribution token by token to determine whether the corresponding candidate token should be retained, and finally outputs all retained tokens.

[0045] For the system embodiments of the present invention, since they basically correspond to the foregoing method embodiments, the relevant content can be referred to the description of the method embodiments, and will not be repeated here. The specific implementation methods of the remaining modules will also not be repeated here. The above system embodiments are merely illustrative implementations. The modules described as independent units can be physically independent components, or they can be integrated into one unit according to actual needs; the modules described as functional units can be logical units, and are not necessarily limited to independent physical entities. Each module can be centrally located in the same device, or it can be distributed across multiple devices or network nodes for collaborative implementation. Depending on the actual application requirements, some or all of the modules can be selected to achieve the purpose of the present invention. Those skilled in the art can understand and implement the above system embodiments of the present invention without any creative effort.

[0046] The system embodiments of the present invention can be applied to any device or equipment with data processing capabilities. The device can be a computer terminal, server, embedded device, or other programmable electronic device. The system embodiments can be implemented in software, hardware, or a combination of both. When implemented in software, this system, as a logical device, can have its processor execute computer program instructions stored in non-volatile storage media and load the instructions into memory for execution, thereby achieving the functions described in this invention.

[0047] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and other modifications are possible. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should be considered within the scope of protection of this invention.

Claims

1. A complex scene speculation decoding method based on a scholar large model, characterized in that, The bookman large model comprises at least a visual encoder, a text embedding layer, a multi-layer decoder and a classification head; the method comprises: S1, obtaining multi-scene industrial anomaly detection image data and text problem tasks corresponding to image samples; S2, inputting the text problem tasks into the text embedding layer of the bookman large model to convert the text problem tasks into text embeddings, and inputting the text embeddings into the multi-layer decoder to output corresponding last-layer hidden states; S3, inputting the image samples into the visual encoder of the bookman large model to obtain image tokens, and further inputting the image tokens into the visual adapter of the draft model to obtain compressed visual tokens and a global visual feature vector; S4, inputting the compressed visual tokens, the text embeddings and the corresponding last-layer hidden states into the single-layer decoder of the draft model, introducing the global visual feature vector to enhance the initial last-layer hidden state generated, inputting the enhanced last-layer hidden state into the classification head of the draft model to obtain a draft probability distribution, and further obtaining a to-be-verified token sequence through self-recurrence; the draft probability distribution comprises probabilities of each candidate token; the to-be-verified token sequence comprises a plurality of candidate tokens; S5, inputting the to-be-verified token sequence into the classification head of the bookman large model to output a target probability distribution; the target probability distribution comprises probabilities of each candidate token in the to-be-verified token sequence; S6, comparing the draft probability distribution and the target probability distribution token by token to determine whether the corresponding candidate token is retained, and finally outputting all retained tokens.

2. The complex scene speculative decoding method based on shu sheng big model according to claim 1, characterized in that, The draft model comprises a lightweight visual adapter, a single-layer decoder and a classification head.

3. The complex scene speculative decoding method based on shu sheng big model according to claim 1, characterized in that, The visual adapter is a Transformer encoder with a plurality of learnable query vectors, and comprises a linear projection layer and an attention layer.

4. The complex scene speculative decoding method based on Shuxing model according to claim 3, characterized in that, S3 specifically comprises: S301, inputting the image samples into the visual encoder of the bookman large model to obtain an image token sequence; S302, inputting the image token sequence into the linear projection layer of the visual adapter to convert the image token sequence into a matrix K and a matrix V using a key weight matrix and a value weight matrix respectively; S303, taking the first n-1 query vectors of the visual adapter as compressed query vectors, taking the nth query vector as a global query vector, and taking all the query vectors to form a matrix Q; S304, inputting the matrix Q, the matrix K and the matrix V into the attention layer of the visual adapter to calculate n-1 compressed visual tokens and one global visual feature vector by using a cross-attention mechanism.

5. The complex scene speculation decoding method based on Shuxing model according to claim 1, characterized in that, S6 specifically comprises: S601, compare the probability value of each candidate token in the target probability distribution and the probability value in the draft probability distribution If , the corresponding candidate token is retained, and the calculation is ended; otherwise, the corresponding candidate token and all candidate tokens after it in the token sequence to be verified are deleted, and the next calculation is entered. S602, the classification head of the bookman large model resamples a new token according to all the retained candidate tokens to form a retained token sequence with all the retained candidate tokens; S603, repeating S4-S6 to iterate until the semantics of the retained token sequence is complete; S604, outputting the final retained token sequence.

6. The complex scene speculation decoding method based on Shuxing model according to claim 1, characterized in that, In S4, the global visual feature vector is used to enhance the initial last-layer hidden state; at the i-th text position, the enhanced last-layer hidden state is calculated as follows: ； wherein is the last layer hidden state of the i-th text position, is a projection matrix, is a global visual feature vector.

7. The complex scene speculation decoding method based on ShuSheng big model according to claim 1, characterized in that, The probability is the probability of generating the corresponding candidate token by the model under the current context.

8. The complex scene speculation decoding method based on Shuxing model according to claim 1, characterized in that, The text problem task includes three subtasks: a defect judgment task, a defect positioning task, and a defect classification task. The defect judgment task requires the multi-modal large language model to determine whether a defect exists in the current scene. The defect positioning task requires the multi-modal large language model to provide pixel-level coordinates of the defect region. The defect classification task requires the multi-modal large language model to analyze defect features and infer classification results on the premise of confirming the existence of defects and locating the defect region.

9. The complex scene speculation decoding method based on Shuxing model according to claim 8, characterized in that, The subtasks are composed of questions and options, and the options contain interference items.

10. A system for implementing the method of claim 1, wherein the system is a complex scene speculation decoding system based on a scholar large model. It includes: a data acquisition module for acquiring multi-scene industrial anomaly detection image data and text problem tasks corresponding to image samples; a text processing module for converting text problem tasks into text embeddings by inputting them into the text embedding layer of the book big model, and receiving the text embeddings by the decoder and outputting the corresponding last layer hidden state; an image processing module for inputting image samples into the visual encoder of the book big model to obtain image tokens, and further inputting the image tokens into the visual adapter to obtain compressed visual tokens and global visual feature vectors; a draft model processing module for inputting compressed visual tokens, text embeddings, and corresponding last layer hidden states into the single-layer decoder of the draft model, introducing global visual feature vectors to enhance the generated initial last layer hidden state, and inputting the enhanced last layer hidden state into the classification head of the draft model to obtain a draft probability distribution, and further obtaining a to-be-verified token sequence through autoregression; the draft probability distribution includes the probability of each candidate token; the to-be-verified token sequence includes a batch of candidate tokens; a target model processing module for inputting the to-be-verified token sequence into the classification head of the book big model to output a target probability distribution; the target probability distribution includes the probability of each candidate token in the to-be-verified token sequence; a probability comparison module for comparing the draft probability distribution and the target probability distribution token by token to determine whether the corresponding candidate token is retained, and finally outputting all retained tokens.

Images