Cylinder head surface defect detection method and system based on multi-modal large model

Abstract

The application belongs to the field of engine cylinder cover defect detection, and discloses a cylinder cover surface defect detection method and system based on a multi-modal large model, which comprises the following steps: collecting cylinder pictures, and preprocessing the cylinder pictures; constructing a data set for fine-tuning of the multi-modal large model according to the preprocessed cylinder pictures; fine-tuning the multi-modal large model using the constructed data set; and detecting cylinder cover surface defects using the fine-tuned multi-modal large model. The application realizes high-precision detection of the types of cylinder cover surface defects.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of engine cylinder cover defect detection, and particularly relates to a cylinder cover surface defect detection method and system based on a multi-modal large model. BACKGROUND

[0002] The cylinder cover is an important part of manufacturing automobiles. During the production of the cylinder cover, various defects and marks are often generated on the surface of the cylinder cover due to external condition limitations. These defects not only affect the appearance of the cylinder cover, but also affect the safety of the automobile manufactured therewith, and therefore it is of great significance to strictly control the quality of the cylinder cover product. At present, the detection of defects on the production line of many factories is carried out by manual detection. This detection method is not only slow and inefficient, but also the accuracy of detection is greatly related to the subjectivity of the detection worker, and therefore a cylinder cover surface defect detection algorithm is needed.

[0003] Through retrieval, in 2023, a Chinese invention patent (CN202311543130.5) establishes an isolated feature tree to evaluate the defect probability of a connected domain by analyzing a cylinder cover grayscale image to determine the quality of the cylinder. In the same year, a Chinese invention patent (CN202311543130.5) proposes a method for detecting surface defects of an ultra-high resolution engine cylinder block and cylinder cover, which uses a yolov7 model to detect defects in the cylinder cover. Prior to this, a Chinese invention patent (CN202211509429.4) proposes a method for detecting defects by constructing a three-dimensional model of the cylinder cover and comparing it with a standard cylinder cover model.

[0004] In summary, the current technology for detecting defects on the surface of the cylinder cover has the following problems: (1) it can only determine whether the cylinder cover has defects as a whole, but cannot accurately detect the location and type of the defects; and (2) the accuracy of the detected defects is not high enough. The present application preprocesses the cylinder cover defect data, proposes a method for constructing a data set for fine-tuning a multi-modal large model, and uses a multi-modal data to fine-tune a multi-modal large model to enhance its scene detection capability in the defect detection field, thereby achieving high-precision detection of the location and type of defects on the surface of the cylinder cover. SUMMARY

[0005] To solve the problems existing in the prior art, the present application provides a cylinder cover surface defect detection method and system based on a multi-modal large model, which achieves high-precision detection of the type of defects on the surface of the cylinder cover.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0007] The cylinder cover surface defect detection method based on a multi-modal large model comprises the following steps:

[0008] Collecting a cylinder picture and preprocessing the cylinder picture;

[0009] According to the pretreated cylinder picture, a data set for fine-tuning of a multi-modal large model is constructed;

[0010] The multi-modal large model is fine-tuned using the constructed data set;

[0011] The fine-tuned multi-modal large model is used to detect defects on the cylinder head surface.

[0012] Preferably, the cylinder picture is collected, and the pretreatment of the cylinder picture comprises:

[0013] The cylinder picture is processed in blocks, and the size of the picture after blocking is M*M;

[0014] The height and width of the cylinder picture are obtained, and if they are not multiples of M, the height and width of the cylinder picture are padded to be integer multiples of M;

[0015] A window function h(m, n, a, b) is designed, and the window function h(m, n, a, b) is used to slide on the padded picture. Each time the window slides, it determines whether the window contains defects. If it contains defects, the window picture is cropped. If it does not exist, it slides to the next window. The cropped picture of the picture w(x0, y0) is: i i h(m, n, a, b) * w(x0+m, y0+n), where x0 and y0 represent the horizontal and vertical coordinates of the lower left corner of the picture, where m and n represent the width and height of the window, and a and b represent the relative horizontal and vertical coordinates of the lower left corner of the window;

[0016] The corresponding label coordinates of the cropped picture are recalculated. The horizontal coordinate formula is x' = x-z*M, and the vertical coordinate calculation formula is y' = y-j*M, where z represents the horizontal column number, j represents the vertical row number, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate;

[0017] The label of the cropped picture with recalculated corresponding label coordinates is saved.

[0018] Preferably, according to the pretreated cylinder picture, a data set for fine-tuning of a multi-modal large model is constructed, which comprises:

[0019] An original label json file is obtained, and the defect type, defect coordinate frame upper left and lower right corner coordinates, and corresponding picture path in the original label json file are read;

[0020] A fine-tuning data question and answer format is designed for each picture, the read picture coordinates are written into the question, and fine-tuning instruction words are embedded, and the specific content of the words is according to the content of the specified task, and the word content is segmented by using a flag word;

[0021] ​The defect type, the upper left corner and the lower right corner of the defect frame contained in the read picture are written into the answer content, the answer content is segmented by using a marker word, and if there are multiple defects in one picture, the defect types are written in the answer content <ref>as the next defect and the starting token of the defect coordinates;

[0022] Write the new question and answer content generated by each picture into a json file as a dataset for fine-tuning the multimodal large model.

[0023] Preferably, fine-tuning the multimodal large model using the constructed dataset comprises:

[0024] Obtain the constructed dataset;

[0025] Feature extraction of the input pictures in the dataset using a visual encoder;

[0026] Compress the picture features output by the visual encoder to a fixed length using a visual language adapter;

[0027] Send the visual feature sequence output by the visual language adapter and the instruction text into the large language model to obtain the best recognition result.

[0028] Preferably, feature extraction of the input pictures in the dataset using a visual encoder comprises:

[0029] The visual encoder first cuts the input picture into several non-overlapping pictures, and maps the cut pictures to a fixed dimension vector through linear projection; At the same time, add position encoding to each vector to preserve the spatial structure information of the image; After normalization, the vector is sent into the multi-head self-attention mechanism to learn the relationship between any two input vectors; The attention operation formula is:

[0030]

[0031] Where Q, K, and V are vector matrices obtained by mapping the input information through the linear layer, and d is the dimension of the K matrix; Update the weight of each V through the attention mechanism to obtain the new output; Multi-head attention mechanism:

[0032] MultiHead(Q,K,V)=Concat(head1,……,head h )

[0033]

[0034] The multi-head attention mechanism divides Q and K into heads; Perform scaled dot product operation on Q and K in each head, multiply the structure with V; Concatenate the results calculated in each head; Each head focuses on a different aspect of the input and performs attention mechanism in parallel, and finally combines the results of all heads to obtain the representation;

[0035] At the same time, add a residual connection between the input and the multi-head self-attention mechanism:

[0036] Output=o+F(o)

[0037] The feature o after passing through the network F is added to the feature o before entering the network;

[0038] After that, the model learns the mapping relationship through the feedforward neural network.

[0039] Preferably, the picture feature output by the visual language adapter is compressed to a fixed length using a visual language adapter, which includes:

[0040] The visual language adapter includes a single-layer cross-attention module initialized randomly; the module uses a set of trainable vectors as query vectors q, and adds position encoding to q:

[0041]

[0042] The position encoding uses the cos function, where pos represents the position of the cut picture in the original large picture, i represents the position of the vector after linear mapping of the picture, d model is the dimension of the vector; through the position encoding, the position information of the cut picture is added;

[0043] Then the image feature K, V output by the visual encoder is used for Attention operation with a set of learnable vectors q; finally, the visual feature sequence is compressed to a fixed length of 256.

[0044] Preferably, the visual feature sequence output by the visual language adapter is sent into a large language model together with the instruction text, which includes:

[0045] The text instruction is embedded into a vector through an embedding layer, and is sent into a large language model together with the picture vector, and enters an encoder; in the encoder, the input feature is normalized through an RMSNorm normalization layer, the relationship between the input features is captured through a multi-head attention mechanism to update the weight output, and then output through a feedforward neural network, and enter the next encoder, and get the result through N encoders for output.

[0046] The application also provides a cylinder cover surface defect detection system based on a multi-modal large model, which is used to implement any one of the methods, and the system includes a preprocessing module, a data set construction module, a model fine-tuning module and a defect detection module.

[0047] The preprocessing module is used to collect cylinder pictures and pre-process the cylinder pictures;

[0048] The data construction module is used to construct a data set for fine-tuning of the multi-modal large model according to the pre-processed cylinder pictures;

[0049] The model fine-tuning module is configured to fine-tune the multi-modal large model using the constructed data set;

[0050] The defect detection module is configured to detect the cylinder cover surface defects by using the fine-tuned multi-modal large model.

[0051] Compared with the prior art, the present application has the following advantages:

[0052] The present application provides a cylinder cover surface defect detection method and system based on a multi-modal large model. The method specifically includes (1) an image preprocessing method (2) a method for constructing a data set for fine-tuning of a multi-modal large model (3) a method for fine-tuning a multi-modal large model for cylinder cover surface defect detection. The present application realizes high-precision detection of the types of cylinder cover surface defects. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments. Obviously, the drawings described in the following only some embodiments of the present application, and for those skilled in the art, without creative labor, can also obtain other drawings according to these drawings.

[0054] Figure 1 The flowchart of the cylinder cover surface defect detection method based on the multi-modal large model of the embodiment of the present application;

[0055] Figure 2 The schematic diagram of the large picture before cropping of the embodiment of the present application;

[0056] Figure 3 The schematic diagram of the picture after cropping of the embodiment of the present application;

[0057] Figure 4 The schematic diagram of the data set construction for fine-tuning of the multi-modal large model of the embodiment of the present application;

[0058] Figure 5 The network structure schematic diagram of the multi-modal large model of the embodiment of the present application;

[0059] Figure 6 The inference result image of the embodiment of the present application;

[0060] Figure 7 The dirty-clear-not defect detection effect diagram of the embodiment of the present application;

[0061] Figure 8 The knock-clear-defect detection effect diagram of the embodiment of the present application;

[0062] Figure 9 The knock-clear-allow detection effect diagram of the embodiment of the present application;

[0063] Figure 10 A sand block-clear defect detection effect diagram for an embodiment of the present application;

[0064] Figure 11 A nodule-clear defect detection effect diagram for an embodiment of the present application;

[0065] Figure 12 A nodule-clear defect detection effect diagram for an embodiment of the present application;

[0066] Figure 13 A cold separation-clear defect detection effect diagram for an embodiment of the present application;

[0067] Figure 14 A flash-clear defect detection effect diagram for an embodiment of the present application;

[0068] Figure 15 A pen imprint-clear defect detection effect diagram for an embodiment of the present application;

[0069] Figure 16 An aluminum scrap-clear defect detection effect diagram for an embodiment of the present application;

[0070] Figure 17 A repair-clear defect detection effect diagram for an embodiment of the present application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0072] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0073] Embodiment one

[0074] Since the collected cylinder picture has a large size and the defect size is small, there is a large size difference between the two. A cylinder large model defect piece detection algorithm based on block training is proposed. The technical solution of the present application is as shown in Figure 1

[0075] The present application provides a cylinder cover surface defect detection method based on a multi-modal large model, which comprises the following steps:

[0076] Collecting a cylinder picture, and preprocessing the cylinder picture;

[0077] ​According to the pretreated cylinder picture, a data set for fine-tuning of a multi-modal large model is constructed;

[0078] The multi-modal large model is fine-tuned using the constructed data set;

[0079] The fine-tuned multi-modal large model is used to detect defects on the cylinder head surface.

[0080] In the embodiment, data preprocessing is performed. A sliding window function is used to process the picture in blocks. The size of the picture after blocking is M*M. The steps are as follows:

[0081] Step S1-1: Read the height and width of the picture. If they are not multiples of M, pad the input picture height and width to be an integer multiple of M.

[0082] Step S1-2: Design a window function h(m, n, a, b).

[0083]

[0084] Where m and n represent the width and height of the window, and a and b represent the relative horizontal coordinate and relative vertical coordinate of the lower left corner of the window.

[0085] The window is used to slide on the padded picture. Each time the window is slid, it determines whether the window contains defects. If it does, the window picture is cropped. If it does not, it is slid to the next window. The picture w(x0, y0) is shown in the following table, where x0 and y0 represent the horizontal and vertical coordinates of the lower left corner of the picture. The cropped picture is:

[0086] w i =h i (m, n, a, b) * w(x0 + m, y0 + n)

[0087] Step S1-3: Recalculate the label coordinates corresponding to the cropped picture.

[0088] The horizontal coordinate formula is:

[0089] x′=x-z

[0090] The vertical coordinate calculation formula is:

[0091] y′=y-j*M

[0092] Where z represents the number of horizontal columns, j represents the number of vertical rows, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate. The large picture before cropping is shown in the following table, and the cropped picture is shown in the following table. Figure 2 Figure 3

[0093] Step S1-4: Save the label of the cropped picture for subsequent use.

[0094] As can be seen​​ Figure 2 The cropped picture 3 is larger and has clearer details, which makes it easier for the model to focus on the defect features.

[0095] In this embodiment, a data set is constructed: the data set composed of the cropped pictures and the corresponding label json files is constructed into a data set for fine-tuning of a multi-modal large model. The steps are as follows:

[0096] Step S2-1: As shown in Figure 4 , first read the original label json file, read the defect category, defect coordinate box upper left and lower right corner coordinates, and the path of the corresponding picture.

[0097] Step S2-2: Design a fine-tuning data question and answer format for each picture, write the read picture coordinates into the question, and embed the fine-tuning instruction words at the same time. The specific content of the words is according to the content of the specified task. For example, when used for defect detection, the word content can be "circle the defect and mark the category". In order to distinguish the picture and the instruction words, a marker word is used to separate them. For example, the picture path can be written in the middle, and the instruction words are added after it.

[0098] Step S2-3: Write the defect category, defect box upper left and lower right corner coordinates contained in the read picture into the answer content. In order to distinguish between the two, a marker word can be used to separate them. For example, the defect category can be written between <ref>and< / ref> , and then the marker word <box>and< / box> is followed by the defect box upper left and lower right corner coordinates. If there are multiple defects in a picture, write <ref>As the next defect and the starting mark of the defect coordinates, the format is the same as above, and the remaining defects are continued in this way.

[0099] Step S2-4: Write the new question and answer content generated by each picture into a json file as a data set for fine-tuning of the multi-modal large model.

[0100] In this embodiment, the multi-modal large model is fine-tuned using the constructed data set.

[0101] The network structure of the multi-modal large model is as shown in Figure 5 The steps are as follows:

[0102] Step S3-1: Read the data set constructed in step 2.

[0103] Step S3-2: Feature extraction of the input picture using a visual encoder.

[0104] The visual encoder first cuts the input picture into several non-overlapping small pictures, and maps these small pictures to a fixed dimension vector through linear projection. At the same time, position encoding is added to each vector to preserve the spatial structure information of the image. After normalization, the vector is input into the multi-head self-attention mechanism to learn the relationship between any two input vectors. The Attention operation formula is:

[0105]

[0106] Q, K, and V are vector matrices obtained by mapping the input information through the linear layer, and d is the dimension of the K matrix. The weight of each V is updated through the attention mechanism to obtain a new output. The multi-head attention mechanism is based on this:

[0107] MultiHead(Q,K,V)=Concat(head1,……,head h )

[0108]

[0109] The multi-head attention mechanism divides Q and K into heads. The scaled dot product operation is performed on Q and K in each head, and the structure is multiplied by V. The results calculated in each head are spliced together. Each head focuses on different aspects of the input and performs attention mechanism in parallel, and finally the results of all heads are combined to obtain a more rich and detailed representation.

[0110] At the same time, a residual connection is added between the input and the multi-head self-attention mechanism:

[0111] Output=o+F(o)

[0112] The feature o is added after passing through the network F to ensure that the input features of the subsequent network contain the original feature information, so that each layer receives the original information.

[0113] Then, the model learns a more complex mapping relationship through the feedforward neural network.

[0114] Step S3-3: compress the picture features output by the visual encoder into a fixed length using the visual language adapter.

[0115] The visual language adapter includes a cross-attention module initialized randomly. The module uses a set of trainable vectors as query vectors q, and adds position encoding to q:

[0116]

[0117] The position encoding uses the cos function, where pos represents the position of the small picture in the original large picture, i represents the position of the vector after linear mapping of the picture, d model is the dimension of the vector. Through the position encoding, the position information of the small picture is added.

[0118] Then, the image features K, V output by the visual encoder are used for Attention operation with a set of learnable vectors q. Finally, the visual feature sequence is compressed to a fixed length of 256. The operation result of this part retains the more important features in the image features for subsequent model input.

[0119] Step S3-4: input the visual feature sequence output by the visual language adapter and the instruction text into the large language model. The text instruction is embedded into a vector through the embedding layer, and the picture vector output by the visual language adapter is sent into the large language model together, and enters the encoder. In the encoder, the input features are normalized through the RMSNorm normalization layer, the relationships between the input features are captured through the multi-head attention mechanism to update the weight output, then output through the feedforward neural network, and enter the next encoder. After N encoders, the result is output. Compare the obtained inference result with the label to update the model parameters.

[0120] Step S3-5: repeat the above steps to obtain the best recognition result.

[0121] In this embodiment, the multi-modal large model recognition effect is tested. The steps are as follows:

[0122] Step S4-1: read the width and height of the input image, pad the input image width and height to an integer multiple of M, crop it into M*M small pictures, and number each small picture.

[0123] Step S4-2: The small picture is input into the large model inference, the large model output string is parsed, and the defect rectangular frame is circled in the small picture, and the defect type is marked.

[0124] Step S4-3: The small picture is spliced into a large picture according to the position label, and is cropped according to the width and height of the input image, and the part filled when the small picture is cropped is subtracted, and the inference result image is output, as shown in Figure 6

[0125] There are various defects in picture 6, which are detected when the small picture is input into the model.

[0126] Taking the cold separation defect as an example, compared with the YOLOV10 model, the detection effect under the same data set is as shown in Table 1.

[0127] Table 1

[0128]

[0129] Taking mAP50 as the evaluation index, it can be seen that the test result of the multi-modal large model improves by 91% based on Yolov10, and the detection accuracy is obviously improved.

[0130] The method detects 11 kinds of cylinder cover surface defects by using the fine-tuned visual language large model, and the detection effect is as shown in the following figure.

[0131] Figure 7 The detection effect diagram of dirty-clear-not defect is Figure 8 The detection effect diagram of knock-clear-defect is

[0132] Figure 9 The detection effect diagram of knock-clear-acceptable is

[0133] Figure 10 The detection effect diagram of sand-clear-defect is Figure 11 The detection effect diagram of nodule-clear-defect is Figure 12 The detection effect diagram of nodule-clear-acceptable is

[0134] Figure 13 The detection effect diagram of cold separation-clear-defect is Figure 14 The detection effect diagram of flash-clear-not defect is Figure 15 The detection effect diagram of pen imprint-clear-not defect is

[0135] Figure 16 The detection effect diagram of aluminum scrap-clear-not defect is Figure 17 The detection effect diagram of repair-clear-not defect is

[0136] Example Two

[0137] ​The application also provides a cylinder cover surface defect detection system based on a multi-modal large model, which is used to implement any one of the methods, and comprises a preprocessing module, a data set construction module, a model fine-tuning module and a defect detection module.

[0138] The preprocessing module is used to collect cylinder pictures and pre-process the cylinder pictures.

[0139] The data construction module is used to construct a data set for fine-tuning of the multi-modal large model according to the pre-processed cylinder pictures.

[0140] The model fine-tuning module is used to fine-tune the multi-modal large model using the constructed data set.

[0141] The defect detection module is used to detect cylinder cover surface defects by using the fine-tuned multi-modal large model.

[0142] The above-described embodiments only describe the preferred modes of the application and do not limit the scope of the application. Without departing from the design spirit of the application, various modifications and improvements to the technical solutions of the application made by those skilled in the art shall fall within the protection scope defined by the claims of the application.< / ref> ​< / ref>

Claims

1. A cylinder head surface defect detection method based on a multi-modal large model, characterized by, The method comprises: Collecting a cylinder picture, and preprocessing the cylinder picture; According to the preprocessed cylinder picture, a data set for fine-tuning of a multi-modal large model is constructed; The multi-modal large model is fine-tuned using the constructed data set; The cylinder cover surface defect is detected by using the fine-tuned multi-modal large model; Collecting a cylinder picture, and preprocessing the cylinder picture comprises: The cylinder picture is processed in blocks, and the size of the picture after blocking is M*M; If neither the height nor the width of the cylinder picture is a multiple of M, the height and width of the cylinder picture are padded to an integer multiple of M; Designing a window function Using the window function Sliding on the filled picture, judging whether the window contains defects every time the window slides, if it contains, cutting the window picture, if it does not exist, sliding to the next window, the picture w(x0, y0) is cut as follows: Wherein, x0, y0 represent the horizontal coordinate and vertical coordinate of the lower left corner of the picture, Wherein, m, n represent the width and height of the window, a, b represent the relative horizontal coordinate and relative vertical coordinate of the lower left corner of the window. Recalculating the label coordinates corresponding to the cropped picture; wherein the horizontal coordinate formula is: and the vertical coordinate calculation formula is: wherein z represents the horizontal column number, j represents the vertical row number, x represents the original label horizontal coordinate, and y represents the original label vertical coordinate. The label of the cropped picture after recalculating the corresponding label coordinates is saved; According to the preprocessed cylinder picture, a data set for fine-tuning of a multi-modal large model is constructed, which comprises: An original label json file is obtained, and the defect type, defect coordinate frame upper left corner and right lower corner coordinates, and the path of the corresponding picture in the original label json file are read; A fine-tuning data question and answer format is designed for each picture, the read picture coordinates are written into the question, and a fine-tuning instruction word is embedded, the specific content of the word is according to the content of the specified task, and the word content is segmented by using a marker word; The defect type, the upper left corner and the lower right corner of the defect frame contained in the read picture are written into the answer content, the answer content is segmented by using a marker word, and if there are multiple defects in one picture, the defect types are written in the answer content <ref>The next defect and the starting marker word of the defect coordinates are taken as the next defect and the starting marker word of the defect coordinates.< / ref> The new question and answer content generated for each picture is written into a json file as a data set for fine-tuning of a multi-modal large model.

2. The method of claim 1, wherein, The multi-modal large model is fine-tuned using the constructed data set, which comprises: The constructed data set is obtained; The input picture in the data set is feature extracted using a visual encoder; The picture features output by the visual language adapter are compressed to a fixed length; The visual feature sequence output by the visual language adapter and the instruction text are sent into a large language model to obtain the best recognition result.

3. The method of claim 2, wherein, The input picture in the data set is feature extracted using a visual encoder, which comprises: The visual encoder first cuts the input picture into several non-overlapping pictures, maps the cut pictures to a fixed dimension vector through linear projection, adds position encoding to each vector to retain the spatial structure information of the image, normalizes the vector and sends it into a multi-head self-attention mechanism to learn the relationship between any two input vectors; the attention operation formula is: ； Wherein, Q, K and V are vector matrices obtained by mapping the input information through a linear layer, and d is the dimension of the K matrix; the weight of each V is updated through the attention mechanism to obtain a new output; the multi-head attention mechanism is: ； ； The Q and K are divided into heads in the multi-head attention mechanism; the Q and K in each head are subjected to scaling dot product operation, and the structure is multiplied by V; the results calculated in each head are spliced; each head focuses on different aspects of the input, and the attention mechanism is executed in parallel, and finally the results of all heads are combined to obtain a representation; Residual connection is added between the input and the multi-head self-attention mechanism: ； The feature o after passing through the network F is added to the feature o before entering the network; Then, a feedforward neural network is used to make the model learn the mapping relationship.

4. The method of claim 3, wherein, The picture features output by the visual encoder are compressed to a fixed length using a visual language adapter, which comprises: The visual language adapter includes a single-layer cross-attention module initialized randomly; the module uses a set of trainable vectors as a query vector q, and adds position encoding to q: ； Position encoding uses the cos function, pos represents the position of the cut picture in the original large picture, i represents the position of the vector obtained after linear mapping of the picture, d model is the dimension of the vector; through position encoding, the position information of the cut picture is added; Then use the image features K, V output by the visual encoder to perform Attention operation with a set of learnable vectors q; finally, the visual feature sequence is compressed to a fixed length of 256.

5. The method of claim 4, wherein, The visual language adapter outputs the visual feature sequence and the instruction text into the large language model, including: After embedding the text instruction into a vector through the embedding layer, it is sent into the large language model together with the picture vector, enters the encoder, is normalized by the RMSNorm normalization layer in the encoder to realize the input feature, is updated by the multi-head attention mechanism to capture the relationship between the input features and output the weight, then is output by the feedforward neural network, enters the next encoder, and finally the result is output after passing through N encoders.

6. A cylinder head surface defect detection system based on a multi-modal large model, the system being configured to implement the method of any one of claims 1-5, characterized in that, The system comprises a preprocessing module, a dataset construction module, a model fine-tuning module and a defect detection module; The preprocessing module is used for collecting cylinder pictures and preprocessing the cylinder pictures; The dataset construction module is used for constructing a dataset for fine-tuning of a multi-modal large model according to the preprocessed cylinder pictures; The model fine-tuning module is used for fine-tuning the multi-modal large model using the constructed dataset; The defect detection module is used for detecting the surface defects of the cylinder cover by using the fine-tuned multi-modal large model.

Images