Cylinder cover surface defect detection method and system based on multi-modal large model

Abstract

The invention belongs to the field of engine cylinder cover defect detection, and discloses a cylinder cover surface defect detection method and system based on a multi-modal large model, and the method comprises the steps: collecting a cylinder picture, and carrying out the preprocessing of the cylinder picture; according to the preprocessed air cylinder picture, a data set used for multi-mode large model fine adjustment is constructed; using the constructed data set to finely adjust the multi-modal large model; and detecting the surface defects of the cylinder cover by using the finely-adjusted multi-mode large model. According to the invention, high-precision detection of types of surface defects of the cylinder cover is realized.

Description

Technical Field

[0001] The present invention belongs to the field of engine cylinder head defect detection, and in particular relates to a cylinder head surface defect detection method and system based on a multi-modal large model. Background Art

[0002] Cylinder heads are important parts for automobile manufacturing. During the production process of cylinder heads, due to external conditions, various defects and marks are often generated on the surface of the cylinder heads. These defects not only affect the appearance of the cylinder heads, but also affect the safety of the cars manufactured with them. Therefore, it is of great significance to strictly control the quality of cylinder head products. Currently, many factory production lines use manual inspection to detect defects. This inspection method is not only slow and inefficient, but also the accuracy of the inspection is closely related to the subjectivity of the inspection workers. Therefore, a cylinder head surface defect detection algorithm is needed.

[0003] After searching, the 2023 Chinese invention patent (CN202311543130.5) established an isolated feature tree by analyzing the grayscale image of the cylinder head to evaluate the probability of defects in the connected domain to judge the cylinder quality. In the same year, the Chinese invention patent (CN202311543130.5) proposed a surface defect detection method for the cylinder head of an engine block based on ultra-high resolution, which uses the yolov7 model to detect cylinder head defects. Prior to this, there was also a Chinese invention patent (CN202211509429.4) that proposed a method of detecting defects by building a three-dimensional model of the cylinder head and comparing it with a standard cylinder head model.

[0004] In summary, the current technology for cylinder head surface defect detection has the following problems: (1) it can only determine whether the cylinder head has defects as a whole, but cannot accurately detect the location and type of defects; (2) the accuracy of defect detection is not high enough. The present invention preprocesses the cylinder head defect data, proposes a data set method for fine-tuning a multimodal large model, uses multimodal data to fine-tune the multimodal large model to enhance its defect detection scene detection capability, and realizes high-precision detection of the location and type of cylinder head surface defects. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a cylinder head surface defect detection method and system based on a multi-modal large model, which realizes high-precision detection of cylinder head surface defect types.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A cylinder head surface defect detection method based on a multi-modal large model, the method comprising:

[0008] Collect cylinder pictures and pre-process the cylinder pictures;

[0009] Based on the preprocessed cylinder images, a dataset for fine-tuning a multimodal large model is constructed;

[0010] Fine-tune a large multimodal model using the constructed dataset;

[0011] The cylinder head surface defects are detected using the fine-tuned multimodal large model.

[0012] Preferably, collecting the cylinder image and preprocessing the cylinder image includes:

[0013] Divide the cylinder image into blocks, and the size of the block image is M*M;

[0014] Get the height and width of the cylinder image. If they are not multiples of M, fill the height and width of the cylinder image to an integer multiple of M.

[0015] Design a window function h(m,n,a,b), and use it to slide on the filled image. Each time the window is slid, it is determined whether the window contains defects. If it does, the window image is cropped. If not, it slides to the next window. The image w(x0,y0) after cropping is: w i =h i (m,n,a,b)*w(x0+m,y0+n), where x0 and y0 represent the horizontal and vertical coordinates of the lower left corner of the image. Among them, m and n represent the width and height of the window, and a and b represent the relative horizontal and vertical coordinates of the lower left corner of the window;

[0016] Recalculate the label coordinates of the cropped image; the horizontal coordinate formula is: x′=xz*M, and the vertical coordinate calculation formula is: y′=yj*M, where z represents the number of horizontal columns, j represents the number of vertical rows, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate;

[0017] Save the cropped image labels with the corresponding label coordinates recalculated.

[0018] Preferably, constructing a data set for fine-tuning a multimodal large model based on the preprocessed cylinder images includes:

[0019] Get the original label json file, read the defect type, the coordinates of the upper left corner and lower right corner of the defect coordinate box, and the path of the corresponding image in the original label json file;

[0020] Design a fine-tuning data question-answering format for each image, write the read image coordinates into the question, and embed the fine-tuning instruction text. The specific content of the text is divided into marker words according to the content of the specified task;

[0021] Write the defect type contained in the read image, the coordinates of the upper left corner and lower right corner of the defect box into the answer content, and use the marker words to segment the answer content. If there are multiple defects in an image, write <ref>As the starting marker for the next defect and defect coordinates;

[0022] The new question and answer content generated for each image is written into a json file as a dataset for fine-tuning the multimodal large model.

[0023] Preferably, fine-tuning the multimodal large model using the constructed dataset includes:

[0024] Get the constructed dataset;

[0025] Use visual encoder to extract features from input images in the dataset;

[0026] Use the visual language adapter to compress the image features output by the visual encoder to a fixed length;

[0027] The visual feature sequence output by the visual language adapter and the instruction text are fed into the large language model to obtain the best recognition result.

[0028] Preferably, using a visual encoder to extract features from an input image in a data set includes:

[0029] The visual encoder first cuts the input image into several non-overlapping images, and maps the cut images to a vector of fixed dimension through linear projection. At the same time, it adds position encoding to each vector to preserve the spatial structure information of the image. After normalizing the vector, it sends it to the multi-head self-attention mechanism to learn the relationship between any two of the input vectors. The Attention operation formula is:

[0030]

[0031] Among them, Q, K, V are the vector matrices obtained by mapping the input information through the linear layer, and d is the dimension of the K matrix; the weight of each V is updated through the attention mechanism to obtain a new output; the multi-head attention mechanism performs:

[0032] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0033]

[0034] The multi-head attention mechanism performs head operations on Q and K; performs scaled dot product operations on Q and K in each head, and multiplies the structure with V; concatenates the results calculated in each head; each head focuses on different aspects of the input, executes the attention mechanism in parallel, and finally merges the results of all heads to obtain a representation;

[0035] At the same time, a residual connection is added between the input and the multi-head self-attention mechanism:

[0036] Output=o+F(o)

[0037] Feature o after passing through network F is added to feature o before entering the network;

[0038] After that, the model is fed through a feedforward neural network to learn the mapping relationship.

[0039] Preferably, using the visual language adapter to compress the image features output by the visual encoder to a fixed length comprises:

[0040] The vision-language adapter consists of a single-layer criss-cross attention module with random initialization; this module uses a set of trainable vectors as the query vector q and adds a positional encoding to q:

[0041]

[0042] The position encoding uses the cosine function. pos represents the position of the image after cutting in the original large image, i represents the position of the vector obtained after linear mapping of the image, and d model is the dimension of the vector; through position encoding, the position information of the cut image is added;

[0043] Then, the image features K, V output by the visual encoder are used with a set of learnable vectors q for Attention operation; finally, the visual feature sequence is compressed to a fixed length of 256.

[0044] Preferably, sending the visual feature sequence output by the visual language adapter and the instruction text into the large language model comprises:

[0045] The text instructions are embedded into vectors through the embedding layer and sent to the large language model together with the previous image vector, and then enter the encoder; in the encoder, the input features are normalized through the RMSNorm normalization layer, and the relationship between the input features is captured through the multi-head attention mechanism to update the weight output, and then the output is passed through the feedforward neural network and enters the next encoder. The result is obtained through N encoders and output.

[0046] The present invention also provides a cylinder head surface defect detection system based on a multi-modal large model, the system is used to implement any one of the methods described above, the system comprises: a pre-processing module, a data set building module, a model fine-tuning module and a defect detection module;

[0047] The preprocessing module is used to collect cylinder images and preprocess the cylinder images;

[0048] The data construction module is used to construct a data set for fine-tuning a multi-modal large model based on the preprocessed cylinder images;

[0049] The model fine-tuning module is used to fine-tune the multimodal large model using the constructed dataset;

[0050] The defect detection module is used to detect cylinder head surface defects using a fine-tuned multi-modal large model.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention provides a cylinder head surface defect detection method and system based on a multimodal large model. The method specifically includes (1) an image preprocessing method (2) a method for constructing a data set for fine-tuning the multimodal large model (3) fine-tuning the multimodal large model for the cylinder head surface defect detection method. The invention realizes high-precision detection of cylinder head surface defect types. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 It is a schematic flow chart of a cylinder head surface defect detection method based on a multi-modal large model according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of a large image before cutting of an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of a cropped picture according to an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of constructing a data set for fine-tuning a multimodal large model according to an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the network structure of a multi-modal large model according to an embodiment of the present invention;

[0059] Figure 6 is the inference result image of the embodiment of the present invention;

[0060] Figure 7 This is a dirty-clear-not-defect detection effect diagram of an embodiment of the present invention;

[0061] Figure 8 This is a diagram showing the collision-clear-defect detection effect of an embodiment of the present invention;

[0062] Fig. 9 This is a diagram showing the collision-clear-acceptance detection effect of an embodiment of the present invention;

[0063] Fig.10 This is a sand block-clear-defect detection effect diagram of an embodiment of the present invention;

[0064] Fig.11 This is a diagram showing the effect of bulge-clear-defect detection according to an embodiment of the present invention;

[0065] Fig.12 This is a diagram showing the detection effect of protrusion-clear-acceptance according to an embodiment of the present invention;

[0066] Fig.13 This is a cold shut-clear-defect detection effect diagram of an embodiment of the present invention;

[0067] Fig.14 This is a flash-clear-not-defect detection effect diagram of an embodiment of the present invention;

[0068] Fig.15 This is a diagram showing the effect of pen printing - clear - not considered a defect detection in an embodiment of the present invention;

[0069] Fig.16 This is a diagram showing the effect of aluminum chips - clear - not considered a defect detection in an embodiment of the present invention;

[0070] Fig.17 This is a repair-clear-no-defect detection effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] Embodiment 1

[0074] Since the size of the collected cylinder images is large and the defect size is small, there is a large size difference between the two. A cylinder large model defect detection algorithm based on block training is proposed. The technical solution of the present invention is as follows Figure 1 shown.

[0075] The present invention provides a cylinder head surface defect detection method based on a multi-modal large model, the method comprising:

[0076] Collect cylinder pictures and pre-process the cylinder pictures;

[0077] Based on the preprocessed cylinder images, a dataset for fine-tuning a multimodal large model is constructed;

[0078] Fine-tune a large multimodal model using the constructed dataset;

[0079] The cylinder head surface defects are detected using the fine-tuned multimodal large model.

[0080] In this embodiment, data preprocessing is performed using a sliding window function to divide the image into blocks, and the size of the image after block division is M*M. The steps are as follows:

[0081] Step S1-1: Read the height and width of the image. If they are not multiples of M, fill the height and width of the input image to an integer multiple of M.

[0082] Step S1-2: Design the window function h(m,n,a,b).

[0083]

[0084] Among them, m and n represent the width and height of the window, and a and b represent the relative horizontal and vertical coordinates of the lower left corner of the window.

[0085] Use the window to slide on the filled image. Each time the window is slid, it is determined whether the window contains defects. If it does, the window image is cropped. If not, it slides to the next window. Image w(x0,y0) where x0 and y0 represent the horizontal and vertical coordinates of the lower left corner of the image. The cropped image is:

[0086] w i =h i (m,n,a,b)*w(x0+m,y0+n)

[0087] Step S1-3: Recalculate the label coordinates corresponding to the cropped image.

[0088] The horizontal axis formula is:

[0089] *′=*-z

[0090] The vertical axis calculation formula is:

[0091] y′=yj*M

[0092] Where z represents the number of horizontal columns, j represents the number of vertical rows, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate. Figure 2 As shown, the cropped image is as follows Figure 3 shown.

[0093] Step S1-4: Save the cropped image label for subsequent use.

[0094] You can see Figure 2 The picture is larger, and the details of picture 3 are clearer after cropping, which makes it easier for the model to focus on the defect features.

[0095] In this embodiment, a data set is constructed: a data set consisting of cropped images and corresponding label json files is constructed into a data set for fine-tuning a multimodal large model. The steps are as follows:

[0096] Step S2-1: Figure 4 As shown, first read the original label json file, read the defect type, the coordinates of the upper left corner and lower right corner of the defect coordinate box, and the path of the corresponding image.

[0097] Step S2-2: Design a fine-tuning data question-answering format for each image, write the read image coordinates into the question, and embed the fine-tuning instruction text. The specific content of the text is based on the content of the specified task. For example, when used for defect detection, the text content can be "circle the defect and indicate the type". In order to distinguish between the image and the instruction text, use marker words to segment them. For example, the image path can be written in In the middle, add the command text at the end.

[0098] Step S2-3: Write the defect type contained in the read image and the coordinates of the upper left corner and lower right corner of the defect box into the answer content. In order to distinguish the two, you can use marker words to segment them. For example, you can write the defect type in <ref> and< / ref> Between, followed by the marker word <box> and< / box> , write the coordinates of the upper left corner and lower right corner of the defect box between them. If there are multiple defects in an image, write <ref>As the starting marker for the next defect and defect coordinates, the format is the same as above, and the same applies to the remaining defects.

[0099] Step S2-4: Write the new question and answer content generated for each of the above pictures into a json file as a dataset for fine-tuning the multimodal large model.

[0100] In this embodiment, the constructed dataset is used to fine-tune the multimodal large model.

[0101] The multimodal large model network structure is as follows Figure 5 The steps are as follows:

[0102] Step S3-1: Read the data set constructed in step 2.

[0103] Step S3-2: Use the visual encoder to extract features from the input image.

[0104] The visual encoder first cuts the input image into several non-overlapping small images, and maps these small images to a vector of fixed dimension through linear projection. At the same time, position encoding is added to each vector to preserve the spatial structure information of the image. After the vector is normalized, it is sent to the multi-head self-attention mechanism to learn the relationship between any two of the input vectors. The Attention operation formula is:

[0105]

[0106] Q, K, V are vector matrices obtained by mapping the input information through the linear layer, and d is the dimension of the K matrix. The weight of each V is updated through the attention mechanism to obtain a new output. The multi-head attention mechanism is based on this:

[0107] MultiHead(Q,K,V)=Concat(head1,...,head h )

[0108]

[0109] The multi-head attention mechanism performs separate head operations on Q and K. A scaled dot product operation is performed on Q and K in each head, and the structure is multiplied by V. The results calculated in each head are concatenated. Each head focuses on different aspects of the input, and the attention mechanism is executed in parallel, and finally the results of all heads are merged to obtain a richer and more detailed representation.

[0110] At the same time, a residual connection is added between the input and the multi-head self-attention mechanism:

[0111] Output=o+F(o)

[0112] After passing through the network F, feature o is added to feature o before entering the network to ensure that the input features of the subsequent network contain the original feature information and that each layer receives the original information.

[0113] After that, the model is fed through a feedforward neural network to learn more complex mapping relationships.

[0114] Step S3-3: Use the visual language adapter to compress the image features output by the visual encoder to a fixed length.

[0115] The vision-language adapter consists of a randomly initialized criss-cross attention module. This module uses a set of trainable vectors as the query vector q and adds a positional encoding to q:

[0116]

[0117] The position encoding uses the cosine function. pos represents the position of the image after cutting in the original large image, i represents the position of the vector obtained after linear mapping of the image, and d model is the dimension of the vector. Through position encoding, the position information of the small image is added.

[0118] Then, the image features K and V output by the visual encoder are used with a set of learnable vectors q for Attention operation. Finally, the visual feature sequence is compressed to a fixed length of 256. The results of this part of the operation retain the more important features of the image features for subsequent model input.

[0119] Step S3-4: Send the visual feature sequence and instruction text output by the visual language adapter to the large language model. The text instruction is embedded into a vector through the embedding layer, and sent to the large language model together with the image vector output by the visual language adapter, and enters the encoder. In the encoder, the input features are normalized through the RMSNorm normalization layer, and the relationship between the input features is captured through the multi-head attention mechanism to update the weight output, and then the feedforward neural network output enters the next encoder, and the result is obtained through N encoders for output. The inference result is compared with the label to update the model parameters.

[0120] Step S3-5: Repeat the above steps to obtain the best recognition result.

[0121] In this embodiment, the recognition effect of the multimodal large model is tested. The steps are as follows:

[0122] Step S4-1: Read the width and height of the input image, fill the width and height of the input image to an integer multiple of M, crop it into M*M small images, and number each small image.

[0123] Step S4-2: Send the small image to the large model for inference, parse the string output by the large model, circle the defect rectangle in the small image, and indicate the defect type.

[0124] Step S4-3: Splice the small images into a large image according to the position labels, crop the input image according to its width and height, subtract the padding when cropping the small images, and output the inference result image, such as Figure 6 shown.

[0125] There are various defects in Figure 6, which are detected when the small image is input into the model.

[0126] Taking the cold shut defect as an example, it is compared with the YOLOV10 model. Under the same data set, its detection effect is shown in Table 1.

[0127] Table 1

[0128]

[0129] Taking mAP50 as the evaluation indicator, we can see that the test results of the multimodal large model have increased by 91% based on Yolov10, and the detection accuracy has been significantly improved.

[0130] This method detects 11 types of cylinder head surface defects by using a fine-tuned visual language large model. The detection results are shown in the figure below.

[0131] Figure 7 This is the dirty-clear-not-defect inspection effect diagram. Figure 8 This is the bump-clear-defect detection effect diagram.

[0132] Fig. 9 This is a diagram of the bump-clear-acceptance test results.

[0133] Fig.10 This is the sand block-clear-defect detection effect diagram. Fig.11 This is the effect diagram of convex nodule-clear-defect detection. Fig.12 This is the result diagram of the bulge-clear-acceptance test.

[0134] Fig.13 This is the cold shut-clear-defect detection effect diagram. Fig.14 This is the inspection effect diagram of flash-clear-not-defect. Fig.15 This is a pen-printed, clear, and non-defective inspection effect diagram.

[0135] Fig.16 This is the aluminum chips - clear - not considered a defect detection effect diagram. Fig.17 This is a repair-clear-not-defect inspection rendering.

[0136] Embodiment 2

[0137] The present invention also provides a cylinder head surface defect detection system based on a multi-modal large model, the system is used to implement any one of the methods described above, the system comprises: a pre-processing module, a data set building module, a model fine-tuning module and a defect detection module;

[0138] The preprocessing module is used to collect cylinder images and preprocess the cylinder images;

[0139] The data construction module is used to construct a dataset for fine-tuning a multimodal large model based on the preprocessed cylinder images;

[0140] The model fine-tuning module is used to fine-tune the multimodal large model using the constructed dataset;

[0141] The defect detection module is used to detect cylinder head surface defects using a fine-tuned multimodal large model.

[0142] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.< / ref> < / ref>

Claims

1. A cylinder head surface defect detection method based on a multi-modal large model, characterized in that: The method comprises: Collect cylinder pictures and pre-process the cylinder pictures; Based on the preprocessed cylinder images, a dataset for fine-tuning a multimodal large model is constructed; Fine-tune a large multimodal model using the constructed dataset; The cylinder head surface defects are detected using the fine-tuned multimodal large model.

2. The method according to claim 1, characterized in that Collecting the cylinder image and preprocessing the cylinder image includes: Divide the cylinder image into blocks, and the size of the block image is M*M; Get the height and width of the cylinder image. If they are not multiples of M, fill the height and width of the cylinder image to an integer multiple of M. Design a window function h(m,n,a,b), and use it to slide on the filled image. Each time the window is slid, it is determined whether the window contains defects. If it does, the window image is cropped. If not, it slides to the next window. The image w(x0,y0) after cropping is: w i =h i (m,n,a,b)*w(x0+m,y0+n), where x0 and y0 represent the horizontal and vertical coordinates of the lower left corner of the image. Among them, m and n represent the width and height of the window, and a and b represent the relative horizontal and vertical coordinates of the lower left corner of the window; Recalculate the label coordinates of the cropped image; the horizontal coordinate formula is: x′=xz*M, and the vertical coordinate calculation formula is: y′=yj*M, where z represents the number of horizontal columns, j represents the number of vertical rows, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate; Save the cropped image labels with the corresponding label coordinates recalculated.

3. The method according to claim 2, characterized in that Based on the preprocessed cylinder images, the dataset for fine-tuning the multimodal large model includes: Get the original label json file, read the defect type, the coordinates of the upper left corner and lower right corner of the defect coordinate box, and the path of the corresponding image in the original label json file; Design a fine-tuning data question-answering format for each image, write the read image coordinates into the question, and embed the fine-tuning instruction text. The specific content of the text is divided into marker words according to the content of the specified task; Write the defect type contained in the read image, the coordinates of the upper left corner and lower right corner of the defect box into the answer content, and use the marker words to segment the answer content. If there are multiple defects in an image, write <ref> As the starting marker for the next defect and defect coordinates;< / ref> The new question and answer content generated for each image is written into a json file as a dataset for fine-tuning the multimodal large model.

4. The method according to claim 3, characterized in that Fine-tuning a large multimodal model using the constructed dataset includes: Get the constructed dataset; Use visual encoder to extract features from input images in the dataset; Use the visual language adapter to compress the image features output by the visual encoder to a fixed length; The visual feature sequence output by the visual language adapter and the instruction text are fed into the large language model to obtain the best recognition result.

5. The method according to claim 4, characterized in that Using the visual encoder to extract features from the input images in the dataset includes: The visual encoder first cuts the input image into several non-overlapping images, and maps the cut images to a vector of fixed dimension through linear projection. At the same time, it adds position encoding to each vector to preserve the spatial structure information of the image. After normalizing the vector, it sends it to the multi-head self-attention mechanism to learn the relationship between any two of the input vectors. The Attention operation formula is: Among them, Q, K, V are the vector matrices obtained by mapping the input information through the linear layer, and d is the dimension of the K matrix; the weight of each V is updated through the attention mechanism to obtain a new output; the multi-head attention mechanism performs: MultiHead(Q,K,V)=Concat(head1,……,head h ) head i =Attention(QW i Q ,KW i K ,VW i V ) The multi-head attention mechanism performs head operations on Q and K; performs scaled dot product operations on Q and K in each head, and multiplies the structure with V; concatenates the results calculated in each head; each head focuses on different aspects of the input, executes the attention mechanism in parallel, and finally merges the results of all heads to obtain a representation; At the same time, a residual connection is added between the input and the multi-head self-attention mechanism: Output=o+F(o) Feature o after passing through network F is added to feature o before entering the network; After that, the model is fed through a feedforward neural network to learn the mapping relationship.

6. The method according to claim 5, characterized in that Using the visual language adapter to compress the image features output by the visual encoder to a fixed length includes: The vision-language adapter consists of a single-layer criss-cross attention module with random initialization; this module uses a set of trainable vectors as the query vector q and adds a positional encoding to q: The position encoding uses the cosine function. pos represents the position of the image after cutting in the original large image, i represents the position of the vector obtained after the linear mapping of the image, and d model is the dimension of the vector; through position encoding, the position information of the cut image is added; Then, the image features K, V output by the visual encoder are used with a set of learnable vectors q for Attention operation; finally, the visual feature sequence is compressed to a fixed length of 256.

7. The method according to claim 6, characterized in that Feeding the visual feature sequence and instruction text output by the visual language adapter into the large language model includes: The text instructions are embedded into vectors through the embedding layer and sent to the large language model together with the previous image vector, and then enter the encoder; in the encoder, the input features are normalized through the RMSNorm normalization layer, and the relationship between the input features is captured through the multi-head attention mechanism to update the weight output, and then the output is passed through the feedforward neural network and enters the next encoder. The result is obtained through N encoders and output.

8. A cylinder head surface defect detection system based on a multi-modal large model, the system being used to implement the method of any one of claims 1 to 7, characterized in that: The system includes: a preprocessing module, a data set building module, a model fine-tuning module and a defect detection module; The preprocessing module is used to collect cylinder images and preprocess the cylinder images; The data construction module is used to construct a data set for fine-tuning a multi-modal large model based on the preprocessed cylinder images; The model fine-tuning module is used to fine-tune the multimodal large model using the constructed dataset; The defect detection module is used to detect cylinder head surface defects using a fine-tuned multi-modal large model.

Images