Fine adjustment and reasoning method for drawing identification of large model

Abstract

The invention provides a fine tuning and reasoning method for a drawing identification large model, and the method comprises the steps: obtaining original sample data; performing augmentation processing according to the original sample data and the annotation information to generate training sample data; in the current training round, obtaining a to-be-processed image sequence, and obtaining a to-be-processed text sequence; the feature processing module performs adjustment and feature processing to obtain a feature processing result; the multi-modal identification module carries out adjustment and carries out multi-modal identification to obtain an identification result; and determining a loss result according to the recognition result, the true value and a preset loss function, performing iterative fine tuning on the initial drawing recognition model according to the loss result to obtain a target drawing recognition model, and performing reasoning by using the target drawing recognition model. According to the method, the target drawing recognition model can accurately position and understand information required by a user.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for fine-tuning and reasoning a large model for drawing recognition. Background Art

[0002] With the rapid development of mechanical design and manufacturing, engineering drawings play an increasingly important role in technical communication and product manufacturing. They not only ensure the accurate communication of design concepts and precise manufacturing of products, but also promote effective communication and collaboration among team members.

[0003] Drawing parameter recognition methods in the prior art can parse engineering drawings using rule-based engines, deep learning-based image recognition methods, and optical character recognition (OCR) technology.

[0004] However, when faced with complex drawings, these methods can usually only process drawing information of a single mode. When processing drawings involving multiple complex information such as geometric shapes, specifications, dimensions, tolerances, materials, technical requirements, etc., there are dual challenges of accuracy and efficiency. Summary of the invention

[0005] The purpose of this application is to provide a method for fine-tuning and reasoning a large model of drawing recognition in view of the deficiencies in the above-mentioned prior art, so as to solve the problems of poor accuracy and low efficiency of drawing recognition in the prior art.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows: In a first aspect, an embodiment of the present application provides a method for fine-tuning and reasoning a large model for drawing recognition, the method comprising: Acquire original sample data, annotation information, and a task statement, wherein the original sample data includes: a plurality of original engineering drawing images, and the task statement is used to indicate a task type of at least one task to be identified in the original sample data; According to the original sample data and the annotation information, the original sample data is augmented to generate the training sample data; In the current training round, the training sample data is input into the image processing module in the initial drawing recognition model to obtain an image sequence to be processed, and the task statement is input into the text processing module in the initial drawing recognition model to obtain a text sequence to be processed, wherein the initial drawing recognition model includes: an image processing module, a text processing module, a feature processing module and a multimodal recognition module; The feature processing module adjusts the parameters of the feature processing module based on a preset first enhancement factor and a loss result of a previous training round, and performs feature processing on the image sequence to be processed and the text sequence to be processed to obtain a feature processing result; The multimodal recognition module adjusts the parameters of the multimodal recognition module based on a preset second enhancement factor, a preset third enhancement factor, and a loss result of a previous training round, and the multimodal recognition module performs multimodal recognition on the feature processing result to obtain a recognition result; According to the recognition result, the true value and the preset loss function, a loss result is determined, and the initial drawing recognition model is iteratively fine-tuned according to the loss result to obtain a target drawing recognition model, and the target drawing recognition model is used for reasoning.

[0007] In a second aspect, another embodiment of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method described in the first aspect are executed.

[0008] The beneficial effects of the present application are as follows: by acquiring original sample data, annotation information and task statements, and augmenting the original sample data according to the original sample data and the annotation information to generate training sample data, the diversity of the training sample data can be improved, and in the current training round, the training sample data is input into the image processing module in the initial drawing recognition model to obtain the image sequence to be processed, and the task statement is input into the text processing module in the initial drawing recognition model to obtain the text sequence to be processed, the feature processing module adjusts the parameters of the feature processing module based on the preset first enhancement factor and the loss result of the previous training round, and the feature processing module performs Perform feature processing to obtain a feature processing result, and the multimodal recognition module adjusts the parameters of the multimodal recognition module based on a preset second enhancement factor, a preset third enhancement factor, and a loss result of a previous training round, and the multimodal recognition module performs multimodal recognition on the feature processing result to obtain a recognition result, so that the loss result can be determined according to the recognition result, the true value, and a preset loss function, and the initial drawing recognition model is iteratively fine-tuned according to the loss result to obtain a target drawing recognition model, and the target drawing recognition model is used for reasoning, which can improve the generalization ability and robustness of the target drawing recognition model, so that the target drawing recognition model can accurately locate and understand the information required by the user in complex engineering drawing images.

[0009] At the same time, by introducing the adapter, cross-task or cross-domain transfer learning can be achieved while keeping most of the structures of the feature processing module and the multimodal recognition module unchanged, so that the recognition effect of the target drawing recognition model is better. It also reduces the complexity and computational complexity of the feature processing module and the multimodal recognition module without sacrificing too much performance, thereby improving training efficiency.

[0010] In addition, by controlling the enhancement amplitude of the middle layer of the feature processing module and the multimodal recognition module through the enhancement factor, the output of the middle layer of the feature processing module and the multimodal recognition module can be adjusted more flexibly, thereby generating more diverse feature representations, which helps the feature processing module and the multimodal recognition module to learn a wider feature space during the training process, thereby improving the generalization ability of the model, and preventing overfitting during the training process, thereby improving the robustness of the feature processing module and the multimodal recognition module. It can also enable the feature processing module and the multimodal recognition module to better process sample data carrying noise, thereby improving the anti-interference ability of the feature processing module and the multimodal recognition module. Moreover, by introducing the enhancement factor, the training effect of the feature processing module and the multimodal recognition module can be optimized by adjusting the value of the enhancement factor, which helps to steadily improve the performance of the feature processing module and the multimodal recognition module. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 A flow chart of a method for fine-tuning and reasoning a large model for drawing recognition provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of the initial drawing recognition model provided in an embodiment of the present application; Figure 3 A schematic diagram of a process for generating training sample data in the fine-tuning and reasoning method of the large model for drawing recognition provided in an embodiment of the present application; Figure 4 A schematic diagram of a process for obtaining target sample data in the fine-tuning and reasoning method of the drawing recognition large model provided in an embodiment of the present application; Figure 5 A schematic diagram of a process for obtaining training sample data in the fine-tuning and reasoning method of the large model for drawing recognition provided in an embodiment of the present application; Figure 6A schematic diagram of the structure of a feature processing module in an initial drawing recognition model provided in an embodiment of the present application; Figure 7 A schematic diagram of a flow chart for adjusting the parameters of a feature processing module in the fine-tuning and reasoning method of a large model for drawing recognition provided in an embodiment of the present application; Figure 8 A schematic diagram of a flow chart for determining first parameter adjustment information of each first fully connected layer in the fine-tuning and reasoning method of the drawing recognition large model provided in an embodiment of the present application; Fig. 9 A schematic diagram of determining first parameter adjustment information of a first fully connected layer in the fine-tuning and reasoning method of a large model for drawing recognition provided in an embodiment of the present application; Fig.10 A schematic diagram of the structure of a multimodal recognition module in an initial drawing recognition model provided in an embodiment of the present application; Fig.11 A schematic diagram of a flow chart for adjusting the parameters of a multimodal recognition module in the fine-tuning and reasoning method of a large model for drawing recognition provided in an embodiment of the present application; Fig.12 A schematic diagram of a flow chart for determining the second parameter adjustment information of each attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in an embodiment of the present application; Fig.13 A schematic diagram of determining query parameter adjustment information of a self-attention layer in the fine-tuning and reasoning method of a large model for drawing recognition provided in an embodiment of the present application; Fig.14 A schematic diagram of determining key parameter adjustment information of a self-attention layer in a fine-tuning and reasoning method of a large model for drawing recognition provided in an embodiment of the present application; Fig.15 A schematic diagram of determining the value parameter adjustment information of the self-attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0014] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0015] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0016] Drawing parameter recognition methods in the prior art can parse engineering drawings based on rules, deep learning-based image recognition methods, and the use of OCR technology.

[0017] Specifically, rule-based drawing parsing methods identify drawing parameters through predefined rules, but when dealing with complex or varied drawings, it is difficult to adapt to different types of parameter recognition and has poor scalability. Deep learning-based image recognition methods use deep learning models such as convolutional neural networks (CNNs) to identify geometric shapes, but are still limited to processing image data and are difficult to combine with text or symbol information. Although the method that combines OCR technology with drawing content parsing can extract text information, it lacks accuracy when parsing symbol annotations, tolerance information, and processing technology, and requires frequent manual intervention.

[0018] In summary, when faced with complex drawings, the drawing parameter recognition methods in the prior art can usually only process drawing information of a single mode. When processing drawings involving multiple complex information such as geometric shapes, specifications, dimensions, tolerances, materials, technical requirements, etc., there are dual challenges of accuracy and efficiency.

[0019] The following is a detailed description of the fine-tuning and reasoning method of the large model for drawing recognition provided in the embodiments of the present application in combination with multiple embodiments.

[0020] Figure 1 A flow chart of a method for fine-tuning and reasoning a large model for drawing recognition provided in an embodiment of the present application, referring to Figure 1 As shown, the execution subject of the method may include any electronic device with processing capability, and the method includes: S101, obtaining original sample data, annotation information and task statements.

[0021] The original sample data includes: a plurality of original engineering drawing images, and the task statement is used to indicate the task type of at least one task to be identified in the original sample data.

[0022] Optionally, the original engineering drawing images may be common industrial drawings in the actual production process. The sources of the original engineering drawing images may specifically include: scanned process drawings and construction drawings from actual factory production lines, high-precision CAD export drawings provided by professional design departments or third-party engineering companies, and mixed-format process drawings provided by different industry standards and upstream and downstream enterprises in the supply chain. The original engineering drawing images may include mold drawings, structural parts drawings, assembly drawings, processing process drawings, standard parts drawings, etc.

[0023] Optionally, the task type of the task to be recognized is used to indicate the operation performed when recognizing the original engineering drawing image, such as size specifications, tolerances, geometric shapes, and materials.

[0024] Optionally, the annotation information can be obtained through manual annotation, which may specifically include: geometric shape information, such as: part contour features, hole positions, surface features, cross-sectional views, etc.; specifications and tolerances, such as: length, width, thickness, hole diameter, angle, arc radius, and minimum tolerance and maximum tolerance range and other parameters; material information, such as: part material type (steel, aluminum alloy, plastic, etc.), material brand and corresponding standard code; technical requirements and annotations, such as: specific surface treatment process, roughness requirements, assembly requirements, heat treatment conditions, implementation standards or special markings, etc.

[0025] Exemplarily, for different original engineering drawing images, the task statements are: extracting geometric shape information from the original engineering drawing image, extracting specification size and tolerance information from the original engineering drawing image, extracting material information from the original engineering drawing image, and extracting technical requirements and annotations from the original engineering drawing image.

[0026] S102: According to the original sample data and the annotation information, the original sample data is augmented to generate training sample data.

[0027] Optionally, after obtaining the original sample data and the annotation information, the original sample data and the annotation information may be superimposed, and the sample data obtained after the superposition may be augmented to generate training sample data.

[0028] Exemplarily, the sample data obtained after superposition may be augmented by mirror flipping, rotation, translation, scaling, cropping, adding noise, etc. to generate training sample data.

[0029] S103. In the current training round, the training sample data is input into the image processing module in the initial drawing recognition model to obtain an image sequence to be processed, and the task statement is input into the text processing module in the initial drawing recognition model to obtain a text sequence to be processed.

[0030] Optionally, Figure 2 The schematic diagram of the structure of the initial drawing recognition model provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the initial drawing recognition model includes: an image processing module, a text processing module, a feature processing module and a multimodal recognition module. The output end of the image processing module and the output end of the text processing module are connected to the input end of the feature processing module, the output end of the feature processing module is connected to the input end of the multimodal recognition module, and the output end of the multimodal recognition module is used to output the recognition result.

[0031] Optionally, continue to refer to Figure 2 As shown, the image processing module may include a first sub-image processing module and a second sub-image processing module. The first sub-image processing module is used to extract image features in the training sample data, and the second sub-image processing module is used to restore local details of the extracted image features to obtain an image sequence to be processed.

[0032] Optionally, continue to refer to Figure 2 As shown, the text processing module is used to perform word segmentation and vector generation processing on the input task sentence, so as to generate a text sequence to be processed.

[0033] Exemplarily, the first sub-image processing module can be implemented based on a visual basic processing model, for example, the visual basic processing model can be InternViT-6B, the second sub-image processing module can be implemented based on pixel unshuffle, and the text processing module can be implemented based on a text tokenizer.

[0034] S104: The feature processing module adjusts the parameters of the feature processing module based on a preset first enhancement factor and a loss result of a previous training round, and performs feature processing on the image sequence to be processed and the text sequence to be processed to obtain a feature processing result.

[0035] Optionally, the feature processing module may include at least one first adapter. When the feature processing module receives the image sequence to be processed and the text sequence to be processed, the parameters of the feature processing module can be adjusted based on a preset first enhancement factor and the loss result of the previous training round, so that the adjusted feature processing module can perform feature processing such as linear change and activation on the image sequence to be processed and the text sequence to be processed to obtain a feature processing result.

[0036] The parameters of the feature processing module at least include: parameters of the first adapter.

[0037] Exemplarily, the feature processing module can be implemented based on a multi-layer perceptron (MLP for short), and a first adapter is introduced into the multi-layer perceptron.

[0038] Exemplarily, after the feature processing module receives the image sequence to be processed and the text sequence to be processed, the parameters of the first adapter in the feature processing module and other parameters in the feature processing module other than the first adapter can be adjusted based on the preset first enhancement factor and the loss result of the previous training round, so that the adjusted feature processing module can perform feature processing such as linear change and activation on the image sequence to be processed and the text sequence to be processed to obtain the feature processing result. Among them, the first enhancement factor can be used to control the enhancement amplitude of the intermediate layer in the feature processing module.

[0039] Exemplarily, the first adapter may be a residual adapter network (Residual adapter_1s), which includes a small number of parameters and is connected to the original network layer in the feature processing module through a residual connection to achieve fine-tuning of the network output.

[0040] S105. The multimodal recognition module adjusts the parameters of the multimodal recognition module based on a preset second enhancement factor, a preset third enhancement factor, and a loss result of a previous training round, and the multimodal recognition module performs multimodal recognition on the feature processing result to obtain a recognition result.

[0041] Optionally, the multimodal recognition module may include at least one second adapter. When the multimodal recognition module receives the feature processing result output by the feature processing module, the parameters of the multimodal recognition module may be adjusted based on a preset second enhancement factor, a preset third enhancement factor and the loss result of the previous training round, so that the adjusted multimodal recognition module can perform multimodal recognition on the feature recognition result to obtain the recognition result of the current training round.

[0042] Among them, the parameters of the multimodal recognition module include at least: parameters of the second adapter, a preset second enhancement factor and a preset third enhancement factor, both of which are used to control the enhancement amplitude of the intermediate layer in the multimodal recognition module, specifically, the second enhancement factor and the third enhancement factor are used to control the enhancement amplitude of different intermediate layers.

[0043] Exemplarily, the multimodal recognition module can be implemented based on the Transformer architecture, and a second adapter network is introduced into the Transformer architecture.

[0044] Exemplarily, after the multimodal recognition module receives the feature processing result output by the feature processing module, it can adjust the parameters of the second adapter in the multimodal recognition module and other parameters in the multimodal recognition module except the second adapter based on a preset second enhancement factor, a preset third enhancement factor and the loss result of the previous training round, so that the adjusted multimodal recognition module can perform multimodal recognition on the feature recognition result to obtain the recognition result of the current training round.

[0045] Exemplarily, the second enhancement factor and the third enhancement factor are used to control the enhancement amplitude of different intermediate layers in the Transformer architecture.

[0046] S106. Determine the loss result according to the recognition result, the true value and the preset loss function, iteratively fine-tune the initial drawing recognition model according to the loss result, obtain the target drawing recognition model, and use the target drawing recognition model for reasoning.

[0047] Optionally, after obtaining the recognition result of the current round, the loss result of the current round may be calculated based on the recognition result of the current round, the true value and a preset loss function.

[0048] Optionally, the loss result of the current round is compared with the preset iteration end condition. If the loss result of the current round meets the preset iteration end condition, the training is terminated and the initial drawing recognition model of the current round is used as the target drawing recognition model so that the target drawing recognition model can be applied for reasoning.

[0049] Optionally, if the loss result of the current round does not meet the preset iteration end condition, the above steps S104-S105 are executed to fine-tune the initial drawing recognition model according to the loss result of the current round for the next round of training.

[0050] In this embodiment, by acquiring original sample data, annotation information and task statements, and performing augmentation processing on the original sample data according to the original sample data and the annotation information to generate training sample data, the diversity of the training sample data can be improved, and in the current training round, the training sample data is input into the image processing module in the initial drawing recognition model to obtain an image sequence to be processed, and the task statement is input into the text processing module in the initial drawing recognition model to obtain a text sequence to be processed, and the feature processing module adjusts the parameters of the feature processing module based on a preset first enhancement factor and a loss result of the previous training round, and the feature processing module performs augmentation processing on the image sequence to be processed and the text sequence to be processed. Feature processing is performed to obtain a feature processing result. The multimodal recognition module adjusts the parameters of the multimodal recognition module based on a preset second enhancement factor, a preset third enhancement factor and a loss result of a previous training round, and the multimodal recognition module performs multimodal recognition on the feature processing result to obtain a recognition result, so that the loss result can be determined according to the recognition result, the true value and a preset loss function, and the initial drawing recognition model is iteratively fine-tuned according to the loss result to obtain a target drawing recognition model, and the target drawing recognition model is used for reasoning, which can improve the generalization ability and robustness of the target drawing recognition model, so that the target drawing recognition model can accurately locate and understand the information required by the user in complex engineering drawing images.

[0051] At the same time, by introducing the adapter, cross-task or cross-domain transfer learning can be achieved while keeping most of the structures of the feature processing module and the multimodal recognition module unchanged, so that the recognition effect of the target drawing recognition model is better. It also reduces the complexity and computational complexity of the feature processing module and the multimodal recognition module without sacrificing too much performance, thereby improving training efficiency.

[0052] In addition, by controlling the enhancement amplitude of the middle layer of the feature processing module and the multimodal recognition module through the enhancement factor, the output of the middle layer of the feature processing module and the multimodal recognition module can be adjusted more flexibly, thereby generating more diverse feature representations, which helps the feature processing module and the multimodal recognition module to learn a wider feature space during the training process, thereby improving the generalization ability of the model, and preventing overfitting during the training process, thereby improving the robustness of the feature processing module and the multimodal recognition module. It can also enable the feature processing module and the multimodal recognition module to better process sample data carrying noise, thereby improving the anti-interference ability of the feature processing module and the multimodal recognition module. Moreover, by introducing the enhancement factor, the training effect of the feature processing module and the multimodal recognition module can be optimized by adjusting the value of the enhancement factor, which helps to steadily improve the performance of the feature processing module and the multimodal recognition module.

[0053] In one possible implementation, Figure 3 A flow chart of generating training sample data in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Figure 3 As shown, the above S102 performs augmentation processing on the original sample data according to the original sample data and the annotation information to generate training sample data, including: S301: Add annotation information to the corresponding position in the original sample data to obtain annotated sample data.

[0054] Optionally, the annotation information is added to the corresponding position in the original sample data to obtain the annotated sample data.

[0055] S302 : Based on a preset rotation matrix, a preset scaling matrix, and a preset cropping matrix, the labeled sample data is randomly rotated, randomly scaled, and randomly cropped in sequence to obtain randomly processed labeled sample data.

[0056] Optionally, after obtaining the labeled sample data, the labeled sample data may be randomly rotated based on a preset rotation matrix to obtain the randomly rotated labeled sample data.

[0057] Optionally, after obtaining the labeled sample data, the labeled sample data may be randomly scaled based on a preset scaling matrix to obtain the randomly scaled labeled sample data.

[0058] Optionally, after obtaining the labeled sample data, the labeled sample data may be randomly cropped based on a preset cropping matrix to obtain the randomly cropped labeled sample data.

[0059] Optionally, the randomly rotated labeled sample data, the randomly scaled labeled sample data, and the randomly cropped labeled sample data are all used as randomly processed labeled sample data.

[0060] Optionally, the randomly rotated labeled sample data, the randomly scaled labeled sample data, and the randomly cropped labeled sample data may be randomly combined to obtain randomly processed labeled sample data.

[0061] For example, assuming that the original coordinates of the image in the labeled sample data are (x, y), and the rotation angle is θ (randomly selected in the range [-θ_max, θ_max]), the randomly rotated coordinates (x', y') of the image in the labeled sample data can be expressed as follows: x' = cos(θ) * x - sin(θ) * y y' = sin(θ) * x + cos(θ) * y Among them, θ is the rotation angle, and the random selection range is [-θ_max, θ_max], which can be selected through the random selection function Uniform(), specifically θ = Uniform(-θ_max, θ_max).

[0062] For example, assuming that the original coordinates of the image in the labeled sample data are (x, y), and the image scaling factor is s (randomly selected in the range [s_min, s_max]), the randomly scaled coordinates (x', y') of the image in the labeled sample data can be expressed as follows: x' = s * x y' = s * y Among them, s is the scaling factor, and the random selection range is [s_min, s_max], which can be selected by the random selection function Uniform(), specifically s = Uniform(s_min, s_max).

[0063] For example, assuming that the original area of ​​the image in the labeled sample data is W x H, the original area W x H of the image in the labeled sample data is cropped into a sub-rectangular area W_c x H_c according to the random offset, and the offset (Δx, Δy) is used to represent the starting point of the cropping as follows: Δx = Uniform(0, W - W_c) Δy = Uniform(0, H - H_c) The cropped image region is (x, y) ∈ [Δx, Δx + W_c] × [Δy, Δy + H_c] Among them, W and H are the original width and height of the image in the labeled sample data, W_c and H_c are the cropped width and height of the image in the labeled sample data, and Δx and Δy are the cropping offsets.

[0064] S303: Perform consistency processing on the labeling information in the randomly processed labeling sample data to obtain target sample data.

[0065] Optionally, after obtaining the randomly processed labeled sample data, a consistency check may be performed on the randomly processed labeled sample data to obtain target sample data, so as to ensure consistency between the randomly processed labeled sample data and the labeling information.

[0066] S304: Perform image quality adjustment on the target sample data to obtain training sample data.

[0067] Optionally, after obtaining the target sample data, data augmentation and enhancement processing are performed on the target sample data, the target sample data is expanded to obtain training sample data.

[0068] By adding the annotation information to the corresponding position in the original sample data, the annotation sample data is obtained, and based on the preset rotation matrix, the preset scaling matrix and the preset cropping matrix, the annotation sample data is randomly rotated, randomly scaled and randomly cropped in turn to obtain the randomly processed annotation sample data, and the annotation information in the randomly processed annotation sample data is processed for consistency to obtain the target sample data, and then the image quality of the target sample data is adjusted to obtain the training sample data, which can increase the diversity of the training sample data, help the model learn richer features, reduce the risk of overfitting, and improve the generalization performance of the model. In addition, it can also save data collection costs and improve the applicability and flexibility of the model.

[0069] In one possible implementation, Figure 4 A flow chart of obtaining target sample data in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Figure 4 As shown, the above S303 performs consistency processing on the labeling information in the randomly processed labeling sample data to obtain target sample data, including: S401: Determine whether the first annotation text in the annotation information in the randomly processed annotation sample data is complete.

[0070] Optionally, each annotation text in the annotation information in the randomly processed annotation sample data is traversed, and for the current annotation text traversed, that is, the first annotation text, it is determined whether the first annotation text is complete. The first annotation text is any annotation text in the annotation information.

[0071] Optionally, the position of the text box of the first annotated text may be compared with the region of the image in the annotated sample data to determine whether the first annotated text is complete.

[0072] S402: If yes, retain the first annotation text in the randomly processed annotation sample data; if no, delete the first annotation text in the randomly processed annotation sample data.

[0073] Optionally, if the first annotation text is complete, the first annotation text is retained; if the first annotation text is incomplete, the first annotation text is deleted.

[0074] Exemplarily, whether to delete the first annotated text may be determined by judging whether the first annotated text is completely located within the image region of the randomly processed annotated sample data.

[0075] Exemplarily, assuming that the image area of ​​the randomly processed annotated sample data is [x_min, x_max] ×[y_min, y_max], the bounding box of the first annotated text is [x1, x2] × [y1, y2], and the condition for deleting the annotation is: x1>= x_max or x2<= x_min or y1>= y_max or y2<= y_min. If this condition is met, the first annotated text is deleted.

[0076] Among them, [x_min, x_max] is the range of the image area of ​​the randomly processed labeled sample data on the x-axis, [y_min, y_max] is the range of the image area of ​​the randomly processed labeled sample data on the y-axis, [x1, x2] is the range of the bounding box of the first labeled text on the x-axis, and [y1, y2] is the range of the bounding box of the first labeled text on the y-axis.

[0077] Optionally, if the above conditions are not met, the first annotation text may be deleted.

[0078] Optionally, if the above conditions are not met, the first annotation text can also be relocated according to the position of the first annotation text and the randomly processed annotation sample data to obtain the target position of the first annotation text, and the position of the first annotation text can be updated to the target position in the randomly processed annotation sample data.

[0079] Exemplarily, when the incomplete first annotated text needs to be retained, the bounding box of the first annotated text may be relocated to accommodate the randomly processed annotated sample data.

[0080] For example, taking random processing as only random cropping as an example, assuming that the cropped area of ​​the image in the labeled sample data is [x_min, x_max]×[y_min, y_max], the bounding box of the first labeled text before cropping is [x1, x2]×[y1,y2], and the new bounding box after cropping is [x1', x2']×[y1', y2'], then the area of ​​the cropped first labeled text can be expressed by the following formula: x1' = max(x1, x_min) x2' = min(x2, x_max) y1' = max(y1, y_min) y2' =min(y2, y_max) Among them, [x1', x2'] is the new range of the first annotation text on the x-axis after cropping, [y1', y2'] is the new range of the first annotation text on the y-axis after cropping, max(a, b) is the larger value of a and b, and min(a, b) is the smaller value of a and b.

[0081] Exemplarily, in the randomly processed annotated sample data, the position of the bounding box of the first annotated text is updated to the target position [x1', x2']×[y1', y2'].

[0082] Optionally, after each first annotated text is updated, a consistency check may be performed on the annotated sample data to obtain target sample data.

[0083] Exemplarily, taking random processing as only random cropping as an example, if the area of ​​the new bounding box [x1', x2']×[y1', y2'] after cropping the first annotation text is 0, the first annotation text is deleted, thereby ensuring the consistency of the annotation information in the target sample data and the image.

[0084] S403: After all randomly processed labeled sample data are processed, target sample data are obtained.

[0085] Optionally, after all randomly processed labeled sample data have been executed according to the above steps S401 - S402 , that is, the consistency processing is completed, the target sample data can be obtained.

[0086] In one possible implementation, Figure 5 A flow chart of obtaining training sample data in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Figure 5 As shown, the above S304 performs image quality adjustment on the target sample data to obtain training sample data, including: S501 , performing illumination adjustment on target sample data to obtain pixel-adjusted target sample data.

[0087] Optionally, image pixel value adjustment may be performed on each sample data in the target sample data to achieve illumination adjustment and obtain pixel-adjusted target sample data.

[0088] For example, assuming that the original pixel value of a sample data is I(x, y), and the adjusted pixel value is I'(x, y), the adjusted pixel value can be expressed using the following formula: I'(x, y) = α * I(x, y) + β Among them, I(x, y) is the original pixel value of a sample data at position (x, y), I'(x, y) is the pixel value of a sample data at position (x, y) after the lighting changes, α is the brightness scaling factor, which is used to control the overall brightness change, and β is the brightness offset, which is used to simulate different lighting conditions.

[0089] S502. Perform contrast adjustment on the target sample data to obtain the contrast-adjusted target sample data.

[0090] Optionally, the pixel values of each sample data in the target sample data can be stretched or compressed through linear transformation to achieve contrast adjustment, so as to obtain the contrast-adjusted target sample data.

[0091] Exemplarily, assuming that the original pixel value of a certain sample data is I(x, y) and the adjusted pixel value is I'(x, y), the adjusted pixel value can be expressed by the following formula: I'(x, y) = (I(x, y) - μ) * c + μ Where, I(x, y) is the original pixel value of a certain sample data at the position (x, y), I'(x, y) is the contrast-adjusted pixel value of a certain sample data at the position (x, y), μ is the average pixel value of a certain sample data, defined as μ = (1 / N) * Σ I(x, y), where N is the total number of pixels of this sample data, and c is the contrast adjustment coefficient. c>1 represents enhanced contrast, and 0<c<1 represents reduced contrast.

[0092] S503. Perform blur adjustment on the target sample data to obtain the blur-adjusted target sample data.

[0093] Optionally, the image blur adjustment can be performed on each sample data in the target sample data through convolution operation to obtain the blur-adjusted target sample data.

[0094] Exemplarily, assuming that the original pixel value of a certain sample data is I(x, y) and the adjusted pixel value is I'(x, y), the adjusted pixel value can be expressed by the following formula: I'(x, y) = ΣI(x - u, y - v) * K(u, v) Where, the range of u and v is from -k to k, I(x, y) is the original pixel value of a certain sample data at the position (x, y), I'(x, y) is the blur-adjusted pixel value of a certain sample data at the position (x, y), K(u, v) is the convolution kernel, used to define the blur degree, such as the Gaussian blur kernel, and k is the radius of the convolution kernel, controlling the blur range.

[0095] S504. Generate training sample data according to the pixel-adjusted target sample data, the contrast-adjusted target sample data, and the blur-adjusted target sample data.

[0096] Optionally, after obtaining the pixel-adjusted target sample data, the contrast-adjusted target sample data and the blur-adjusted target sample data, the pixel-adjusted target sample data, the contrast-adjusted target sample data and the blur-adjusted target sample data may be used as training sample data.

[0097] By performing illumination adjustment on the target sample data to obtain pixel-adjusted target sample data, performing contrast adjustment on the target sample data to obtain contrast-adjusted target sample data, and performing blur adjustment on the target sample data to obtain blur-adjusted target sample data, training sample data can be generated based on the pixel-adjusted target sample data, the contrast-adjusted target sample data, and the blur-adjusted target sample data, so that the obtained training sample data can cover drawing inputs of various resolutions, directions and clarity, so that the obtained target drawing recognition model has stronger robustness and the ability to handle diversified inputs when facing complex real application scenarios.

[0098] It can be understood that the above is an exemplary description of the processing process of obtaining training sample data in the fine-tuning and reasoning method of the large drawing recognition model provided in the embodiment of the present application, and the following is an exemplary description of the process of training the initial drawing recognition model based on the training sample data.

[0099] It is worth noting that during the training process of the initial drawing recognition model, the initial drawing recognition model can be trained in a fine-tuning manner. However, when using general fine-tuning methods, such as low-rank adaptation methods such as LORA (Low-Rank Adaptation), the weight updates of different layers show obvious inclination, that is, the bottom layer (such as the embedding layer) and the top layer (such as the head layer of the language model) will occupy most of the weight updates during fine-tuning, while the weight updates of the middle layers are relatively small. This causes the initial drawing recognition model to have performance deviations after fine-tuning, because the middle layers are equally important to the overall performance of the model.

[0100] In order to narrow the weight update gap between the middle layer and the bottom and top layers during fine-tuning, enhance the importance of the middle layer in the fine-tuning process, and narrow the performance gap with full fine-tuning, the following multiple embodiments are used to explain in detail the structure and training process of the initial drawing recognition model in the fine-tuning and reasoning method of the large drawing recognition model provided in this application.

[0101] In one possible implementation, Figure 6 This is a schematic diagram of the structure of the feature processing module in the initial drawing recognition model provided in the embodiment of the present application, referring to Figure 6As shown, the feature processing module in the initial drawing recognition model includes: a splicing layer and at least one perception module connected in sequence; the perception module includes: a first fully connected layer and an activation layer; a first adapter is deployed in each first fully connected layer of each perception module, and each first fully connected layer has a weight matrix.

[0102] Optionally, the splicing layer is used to splice the image sequence to be processed and the text sequence to be processed input into the feature processing module, and input the spliced ​​feature sequence into each perception module for feature processing.

[0103] Optionally, the feature processing module includes multiple perception modules, and the multiple perception modules are connected in sequence. A perception module can be a layer in a multi-layer perceptron (Multi-Layer Perceptron, MLP for short), and each perception module includes a first fully connected layer and a first adapter.

[0104] Among them, the first adapter can be a residual adapter network (Residual adapters), which contains a small number of parameters and is connected to the first fully connected layer in the feature processing module through a residual connection to achieve fine-tuning of the output of the perception module.

[0105] In one possible implementation, Figure 7 A flow chart of adjusting the parameters of the feature processing module in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Figure 6 as well as Figure 7 As shown, in the above S104, the parameters of the feature processing module are adjusted based on the preset first enhancement factor and the loss result of the previous training round, including: S701. Determine first parameter adjustment information of each first fully connected layer according to the first enhancement factor, the loss result of the previous training round, the weight matrix of each first fully connected layer in the previous training round, and the weight matrix of the first adapter of each first fully connected layer in the previous training round.

[0106] It can be understood that the first adapters in each first fully connected layer are connected based on each first fully connected layer by residual connection. When each first fully connected layer is adjusted, that is, the weight matrix of each first fully connected layer is adjusted, then the weight matrix of the first adapter of each first fully connected layer in the previous round can be adjusted, and the weight matrix of each first fully connected layer in the previous training round can be adjusted, thereby realizing the adjustment of each first fully connected layer, so that the input data can be processed through the adjusted weight matrix of each first fully connected layer.

[0107] Optionally, when adjusting the parameters of the feature processing module, the first adapter parameter adjustment information corresponding to the first adapter of each first fully connected layer can be obtained based on the loss result of the previous training round. On this basis, the first parameter adjustment information of each first fully connected layer can be obtained based on the first enhancement factor, the loss result of the previous training round and the first adapter parameter adjustment information.

[0108] Optionally, when adjusting the parameters of the feature processing module, the first adapter parameter adjustment information corresponding to the first adapter of each first fully connected layer can be obtained based on the first enhancement factor and the loss result of the previous training round. On this basis, the first parameter adjustment information of each first fully connected layer can be obtained based on the loss result of the previous training round and the first adapter parameter adjustment information.

[0109] The first parameter adjustment information is used to indicate the weight matrix of the first fully connected layer and the adjustment information of each parameter in the first adapter in the first fully connected layer.

[0110] S702: According to each of the first parameter adjustment information, adjust the weight matrix of each of the first fully connected layers to obtain the weight matrix of each of the first fully connected layers in the current training round.

[0111] Optionally, after obtaining the first parameter adjustment information, the weight matrices of each first fully connected layer can be adjusted separately according to the first parameter adjustment information of each first fully connected layer, so as to obtain the weight matrices of each first fully connected layer in the current training round, so that the weight matrices of each first fully connected layer in the current training round can be used to perform feature processing on the input data to obtain the output of each first fully connected layer.

[0112] The first parameter adjustment information of each first fully connected layer is determined by the first enhancement factor, the loss result of the previous training round, the weight matrix of each first fully connected layer in the previous training round, and the weight matrix of the first adapter of each first fully connected layer in the previous training round. The weight matrix of each first fully connected layer is adjusted through the first parameter adjustment information to obtain the weight matrix of each first fully connected layer in the current training round, so that the weight matrix of each first fully connected layer can be finely adjusted, so that the parameters to be adjusted can be more accurately located, the weight update gap between the middle layer and the bottom layer and the top layer during fine-tuning is narrowed, the importance of the middle layer in the fine-tuning process is enhanced, and the performance gap with full-scale fine-tuning is narrowed. At the same time, unnecessary computing overhead is avoided, computing resources are used more efficiently, and overfitting of input data by each first fully connected layer can be reduced, and the generalization ability of each first fully connected layer on unseen data can be improved.

[0113] In one possible implementation, Figure 8A flow chart of determining the first parameter adjustment information of each first fully connected layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Figure 8 As shown, the above S701 determines the first parameter adjustment information of each first fully connected layer according to the first enhancement factor, the loss result of the previous training round, the weight matrix of each first fully connected layer in the previous training round, and the weight matrix of the first adapter of each first fully connected layer in the previous training round, including: S801. Determine a first low-rank transformation matrix and a second low-rank transformation matrix according to the image dimension in the training sample data.

[0114] Optionally, a low-rank decomposition method may be used to decompose the image dimensions in the training sample data to obtain a first low-rank transformation matrix and a second low-rank transformation matrix.

[0115] The dimension of the multiplication of the first low-rank transformation matrix and the second low-rank transformation matrix matches the dimension of the image. The low-rank decomposition method may include singular value decomposition (SVD) and non-negative matrix decomposition (NMF).

[0116] Exemplarily, in the initial case, a preconfigured target rank can be used to decompose the image dimensions in the training sample data to obtain a first low-rank transformation matrix and a second low-rank transformation matrix. During the training process, the target rank can be optimized by methods such as singular value decomposition (SVD) and non-negative matrix decomposition (NMF) to achieve optimization of the first low-rank transformation matrix and the second low-rank transformation matrix.

[0117] For example, assume that the image dimensions in the training sample data are ,in is the height, As width, the first low-rank transformation matrix U1 can be obtained as And the second low-rank transformation matrix V1^T is ,in, is the current target rank.

[0118] S802: Determine a weight matrix of the first adapter in a current training round according to a loss result of a previous training round and a weight matrix of a first adapter of a first fully connected layer in a previous training round.

[0119] Optionally, taking the Lth first fully connected layer as an example, Fig. 9 A schematic diagram of determining the first parameter adjustment information of the first fully connected layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Fig. 9As shown, x1_L is the input of the Lth first fully connected layer, W1_L is the original trainable weight of the Lth first fully connected layer, U1_L is the first low-rank transformation matrix of the Lth first fully connected layer, V1^T_L is the second low-rank transformation matrix of the Lth first fully connected layer, the first low-rank transformation matrix U1_L can be initialized to 0, and the second low-rank transformation matrix V1^T_L can be initialized to N1_L (0, σ²), that is, randomly initialized from a normal distribution with a mean of 0 and a variance of σ², adapter1_L is the first adapter of the Lth first fully connected layer, h1_L is the differentiable function in the Lth first fully connected layer, which is used to achieve regularization or transformation to help the model better adapt to new tasks or data sets, and α1 is the first enhancement factor.

[0120] The first enhancement factor α1 is used to control a parameter of the enhancement amplitude. For example, when α1=1.5, the enhancement factor of the middle layer increases to 1.5 times.

[0121] Optionally, continue to refer to Fig. 9 As shown, the weight matrix of the first adapter in the current training round can be determined according to the loss result of the previous training round and the weight matrix of the first adapter of the first fully connected layer in the previous training round.

[0122] Exemplarily, the gradient can be calculated according to the loss result of the previous training round and the weight matrix of the first adapter of the first fully connected layer in the previous training round to adjust the weight matrices W1_1 and W1_2 of the first adapter through the back propagation algorithm.

[0123] S803: Determine the first adapter in the current training round according to the weight matrix of the first adapter in the current training round and the second low-rank transformation matrix.

[0124] Optionally, taking the Lth first fully connected layer as an example, the first adapter in the current training round can be calculated based on the weight matrices W1_1 and W1_2 of the first adapter in the current training round and the second low-rank transformation matrix V1_L^T of the Lth first fully connected layer.

[0125] Exemplarily, the first adapter adapter1_L(V1_L^T) of the Lth first fully connected layer can be expressed as the following formula: adapter1_L(V1_L^T) = σ(W1_2 * ReLU1(W1_1 * V1_L^T)) Exemplarily, after obtaining the weight matrices W1_1 and W1_2 of the first adapter in the current training round, W1_1 and W1_2 and the second low-rank transformation matrix V1_L^T of the Lth first fully connected layer can be input into the above formula to obtain the first adapter adapter1_L(V1_L^T) of the Lth first fully connected layer in the current training round.

[0126] Where V1_L^T is the second low-rank transformation matrix of the Lth first fully connected layer, W1_1 and W1_2 are the weight matrices of the first adapter, ReLU1 is the rectified linear unit activation function corresponding to the first fully connected layer, defined as ReLU1 (x) = max(0, x), and σ is the activation function (such as Sigmoid or other nonlinear functions).

[0127] S804: Determine first parameter adjustment information of a first fully connected layer according to the first enhancement factor, the first adapter in the current training round, and the first low-rank transformation matrix.

[0128] Exemplarily, taking the Lth first fully connected layer as an example, after obtaining the first adapter adapter1_L(V1_L^T) in the current training round, the first parameter adjustment information ΔW1_L of the Lth first fully connected layer can be obtained according to the first enhancement factor α1, the first adapter adapter1_L(V1_L^T) in the current training round and the first low-rank transformation matrix U1_L.

[0129] Exemplarily, the first parameter adjustment information ΔW1_L of the Lth first fully connected layer can be expressed using the following formula: ΔW1_L=α1 *U1_L*adapter_1(V1_L^T) Exemplarily, taking the Lth first fully connected layer as an example, after obtaining the first parameter adjustment information ΔW1_L of the Lth first fully connected layer, the first parameter adjustment information ΔW1_L can be superimposed on the weight matrix W1_L of the Lth first fully connected layer to obtain the weight matrix W1'_L of the Lth first fully connected layer in the current training round, where W1'_L=W1_L+ΔW1_L.

[0130] The first low-rank transformation matrix and the second low-rank transformation matrix are determined by the image dimension in the training sample data, and the weight matrix of the first adapter in the current training round is determined according to the loss result of the previous training round and the weight matrix of the first adapter of the first fully connected layer in the previous training round, so as to determine the first adapter in the current training round according to the weight matrix of the first adapter in the current training round and the second low-rank transformation matrix, so that the first parameter adjustment information of the first fully connected layer can be determined according to the first enhancement factor, the first adapter in the current training round and the first low-rank transformation matrix, so that the first fully connected layer can be continuously updated by fine-tuning during the training process, and the first fully connected layer can update the intermediate layer more evenly when updating, which helps to reduce the performance deviation in the fine-tuning process, improve the generalization ability and overall performance of the target drawing recognition model, and also enable the initial drawing recognition model to converge faster, so as to obtain the target drawing recognition model more efficiently.

[0131] In one possible implementation, Fig.10 This is a schematic diagram of the structure of the multimodal recognition module in the initial drawing recognition model provided in the embodiment of the present application, referring to Fig.10 As shown, the multimodal recognition module includes: a plurality of feature recognition modules connected in sequence, each feature recognition module includes: a self-attention layer and a second fully connected layer connected in sequence; a plurality of second adapters are respectively deployed in each attention layer, and a third adapter is respectively deployed in each second fully connected layer.

[0132] Among them, the second adapter can be a residual adapter network (Residual adapters), which contains a small number of parameters and is connected to each attention layer through a residual connection to achieve fine-tuning of the output of each attention layer.

[0133] The third adapter may be a residual adapter network, which includes a small number of parameters and is connected to each second fully connected layer through a residual connection to achieve fine-tuning of the output of each second fully connected layer.

[0134] In one possible implementation, Fig.11 A flow chart of adjusting the parameters of the multimodal recognition module in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Fig.10 as well as Fig.11 As shown, the above S105 adjusts the parameters of the multimodal recognition module based on the preset second enhancement factor, the preset third enhancement factor and the loss result of the previous training round, including: S1101. Determine the second parameter adjustment information of each attention layer according to the second enhancement factor, the loss result of the previous training round, the weight matrix of each attention layer in the previous round, and the weight matrix of each second adapter of each attention layer in the previous training round.

[0135] It can be understood that the second adapters in each attention layer are connected based on the respective attention layer by residual connection. When the respective attention layer is adjusted, that is, the weight matrix of the respective attention layer is adjusted, then the weight matrix of the second adapter of the respective attention layer in the previous round can be adjusted, and the weight matrix of the respective attention layer in the previous training round can be adjusted, thereby realizing the adjustment of the respective attention layer, so that the input data can be processed through the adjusted weight matrix of the respective attention layer.

[0136] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the second adapters of the respective attention layers can be obtained based on the loss results of the previous training round. On this basis, the second parameter adjustment information of the respective attention layers can be obtained based on the second enhancement factor, the loss results of the previous training round and the second adapter parameter adjustment information.

[0137] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the second adapters of the respective attention layers can be obtained based on the second enhancement factor and the loss result of the previous training round. On this basis, the second parameter adjustment information of the respective attention layers can be obtained based on the loss result of the previous training round and the parameter adjustment information of the second adapters.

[0138] Among them, the second parameter adjustment information is used to indicate the weight matrix of the self-attention layer and the adjustment information of each parameter in each second adapter in the self-attention layer.

[0139] S1102. According to each second parameter adjustment information, adjust the weight matrix of each attention layer to obtain the weight matrix of each attention layer in the current training round.

[0140] Optionally, after obtaining the second parameter adjustment information, the weight matrices of the respective attention layers can be adjusted respectively according to the second parameter adjustment information of the respective attention layers, so as to obtain the weight matrices of the respective attention layers in the current training round, so that the input data can be feature processed through the weight matrices of the respective attention layers in the current training round to obtain the output of the respective attention layers, and the weight matrices of the respective attention layers can be finely adjusted, so as to more accurately locate the parameters that need to be adjusted, narrowing the weight update gap between the middle layer and the bottom and top layers during fine-tuning, enhancing the importance of the middle layer in the fine-tuning process, and narrowing the performance gap with full fine-tuning. At the same time, unnecessary computational overhead is avoided, computing resources are used more efficiently, and overfitting of the input data by the respective attention layers can be reduced, thereby improving the generalization ability of the respective attention layers on unseen data.

[0141] S1103. Determine third parameter adjustment information of each second fully connected layer according to the third enhancement factor, the loss result of the previous training round, the weight matrix of each second fully connected layer in the previous training round, and the weight matrix of the third adapter of each second fully connected layer in the previous training round.

[0142] Optionally, when adjusting the parameters of the multimodal recognition module, the third adapter parameter adjustment information corresponding to the third adapter of each second fully connected layer can be obtained based on the loss result of the previous training round. On this basis, the first parameter adjustment information of each second fully connected layer can be obtained based on the third enhancement factor, the loss result of the previous training round and the third adapter parameter adjustment information.

[0143] Optionally, when adjusting the parameters of the multimodal recognition module, the third adapter parameter adjustment information corresponding to the third adapter of each second fully connected layer can be obtained based on the third enhancement factor and the loss result of the previous training round. On this basis, the first parameter adjustment information of each second fully connected layer can be obtained based on the loss result of the previous training round and the third adapter parameter adjustment information.

[0144] S1104. According to each third parameter adjustment information, adjust the weight matrix of each second fully connected layer to obtain the weight matrix of each second fully connected layer in the current training round.

[0145] Optionally, after obtaining the third parameter adjustment information, the weight matrix of each second fully connected layer can be adjusted separately according to the third parameter adjustment information of each second fully connected layer, so as to obtain the weight matrix of each second fully connected layer in the current training round, so that the input data can be feature processed through the weight matrix of each second fully connected layer in the current training round to obtain the output of each second fully connected layer, and the weight matrix of each second fully connected layer can be finely adjusted, so as to more accurately locate the parameters that need to be adjusted, narrow the weight update gap between the middle layer and the bottom layer and the top layer during fine-tuning, enhance the importance of the middle layer in the fine-tuning process, and narrow the performance gap with full fine-tuning. At the same time, unnecessary computational overhead is avoided, computing resources are used more efficiently, and overfitting of input data by each second fully connected layer can be reduced, thereby improving the generalization ability of each second fully connected layer on unseen data.

[0146] In a possible implementation, the weight matrix of each attention layer includes a query matrix, a key matrix and a value matrix, the second enhancement factor includes a query enhancement factor, a key enhancement factor and a value enhancement factor, and each second adapter of each attention layer includes a second adapter corresponding to the query matrix, a second adapter corresponding to the key matrix and a second adapter corresponding to the value matrix. Fig.12 A flow chart of determining the second parameter adjustment information of each attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Fig.12 As shown, the above S1101 determines the second parameter adjustment information of each attention layer according to the second enhancement factor, the loss result of the previous training round, the weight matrix of each attention layer in the previous round, and the weight matrix of each second adapter of each attention layer in the previous training round, including: S1201. Determine the query parameter adjustment information of each attention layer according to the query enhancement factor, the loss result of the previous training round, the query matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the query matrix of each attention layer in the previous round.

[0147] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the query matrix of each second adapter of each attention layer can be obtained based on the loss result of the previous training round. On this basis, the query parameter adjustment information of each attention layer can be obtained based on the query enhancement factor, the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the query matrix of each second adapter.

[0148] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the query matrix of each second adapter of each attention layer can be obtained based on the query enhancement factor and the loss result of the previous training round. On this basis, the query parameter adjustment information of each attention layer can be obtained based on the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the query matrix of each second adapter.

[0149] S1202. Determine the key parameter adjustment information of each attention layer according to the key enhancement factor, the loss result of the previous training round, the key matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the key matrix of each attention layer in the previous round.

[0150] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the key matrix of each second adapter of each attention layer can be obtained based on the loss result of the previous training round. On this basis, the key parameter adjustment information of each attention layer can be obtained based on the key enhancement factor, the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the key matrix of each second adapter.

[0151] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the key matrix of each second adapter of each attention layer can be obtained according to the key enhancement factor and the loss result of the previous training round. On this basis, the key parameter adjustment information of each attention layer can be obtained according to the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the key matrix of each second adapter.

[0152] S1203. Determine the value parameter adjustment information of each attention layer according to the value enhancement factor, the loss result of the previous training round, the value matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the value matrix of each attention layer in the previous round.

[0153] Optionally, when adjusting the parameters of the multimodal recognition module, the second adapter parameter adjustment information corresponding to the value matrix of each second adapter of each attention layer can be obtained based on the loss result of the previous training round. On this basis, the value parameter adjustment information of each attention layer can be obtained based on the value enhancement factor, the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the value matrix of each second adapter.

[0154] Optionally, when adjusting the parameters of the multimodal recognition module, the value enhancement factor and the loss result of the previous training round can be used to obtain the second adapter parameter adjustment information corresponding to the value matrix of each second adapter of each attention layer. On this basis, the value parameter adjustment information of each attention layer can be obtained based on the loss result of the previous training round and the second adapter parameter adjustment information corresponding to the value matrix of each second adapter.

[0155] S1204. Determine the second parameter adjustment information of each attention layer according to the query parameter adjustment information of each attention layer, the key parameter adjustment information of each attention layer, and the value parameter adjustment information of each attention layer.

[0156] Optionally, after obtaining the query parameter adjustment information of each attention layer, the key parameter adjustment information of each attention layer and the value parameter adjustment information of each attention layer, the query parameter adjustment information of each attention layer, the key parameter adjustment information of each attention layer and the value parameter adjustment information of each attention layer can be integrated to obtain the second parameter adjustment information of each attention layer.

[0157] Exemplarily, the query parameter adjustment information of each attention layer, the key parameter adjustment information of each attention layer, and the value parameter adjustment information of each attention layer are weighted averaged, concatenated, or combined in other forms to obtain the second parameter adjustment information of each attention layer.

[0158] In a possible implementation, the above S1201 determines the query parameter adjustment information of each attention layer according to the query enhancement factor, the loss result of the previous training round, the query matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the query matrix of each attention layer in the previous round, including: S1301. Determine a third low-rank transformation matrix and a fourth low-rank transformation matrix according to the image dimension in the training sample data.

[0159] Among them, the dimension after multiplying the third low-rank transformation matrix and the fourth low-rank transformation matrix matches the image dimension.

[0160] Optionally, a low-rank decomposition method may be used to decompose the image dimensions in the training sample data to obtain a third low-rank transformation matrix and a fourth low-rank transformation matrix.

[0161] The dimension of the product of the third low-rank transformation matrix and the fourth low-rank transformation matrix matches the dimension of the image. The low-rank decomposition method may include singular value decomposition (SVD) and non-negative matrix decomposition (NMF).

[0162] Exemplarily, in the initial case, a preconfigured target rank can be used to decompose the image dimensions in the training sample data to obtain a third low-rank transformation matrix and a fourth low-rank transformation matrix. During the training process, the target rank can be optimized by methods such as singular value decomposition (SVD) and non-negative matrix decomposition (NMF) to achieve optimization of the third low-rank transformation matrix and the fourth low-rank transformation matrix.

[0163] For example, assume that the image dimensions in the training sample data are ,in is the height, As width, the third low-rank transformation matrix U3 can be obtained as And the fourth low-rank transformation matrix V4 ^T is ,in, is the current target rank.

[0164] S1302. Determine the weight matrix of the second adapter corresponding to the query matrix in the current training round based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the query matrix of the self-attention layer in the previous training round.

[0165] Optionally, taking the Lth self-attention layer as an example, Fig.13 A schematic diagram of determining the query parameter adjustment information of the self-attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Fig.13 As shown, x2q_L is the input corresponding to the query matrix of the L-th self-attention layer, W2q_L is the original trainable weight corresponding to the query matrix of the L-th self-attention layer, U3_L is the third low-rank transformation matrix corresponding to the query matrix of the L-th self-attention layer, V4^T_L is the fourth low-rank transformation matrix corresponding to the query matrix of the L-th self-attention layer, the third low-rank transformation matrix U3_L can be initialized to 0, and the fourth low-rank transformation matrix V4^T_L can be initialized to N2q_L (0, σ²), that is, randomly initialized from a normal distribution with a mean of 0 and a variance of σ², adapter2q_L is the second adapter corresponding to the query matrix of the L-th self-attention layer, h2q_L is the differentiable function corresponding to the query matrix in the L-th self-attention layer, which is used to achieve regularization or transformation to help the model better adapt to new tasks or data sets, and αq is the query enhancement factor.

[0166] The query enhancement factor αq is used to control the parameters of the enhancement amplitude. For example, when αq=1.5, the enhancement factor of the middle layer is increased to 1.5 times.

[0167] For example, continue to refer to Fig.13As shown, the gradient can be calculated based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the query matrix of the self-attention layer in the previous training round, so as to adjust the weight matrices Wq_1 and Wq_2 of the second adapter corresponding to the query matrix in the current training round through the back-propagation algorithm.

[0168] S1303: Determine the second adapter corresponding to the query matrix in the current training round according to the weight matrix of the second adapter corresponding to the query matrix in the current training round and the fourth low-rank transformation matrix.

[0169] Optionally, taking the L-th self-attention layer as an example, the second adapter corresponding to the query matrix in the current training round can be calculated based on the weight matrices Wq_1 and Wq_2 of the second adapter corresponding to the query matrix in the current training round and the fourth low-rank transformation matrix V4^T_L corresponding to the query matrix of the L-th self-attention layer.

[0170] Exemplarily, the second adapter adapter2q_L corresponding to the query matrix of the L-th self-attention layer can be expressed as follows: adapter2q _L(V4_L^T) = σ(Wq_2* ReLU2q(Wq_1*V4^T_L)) Where V4^T_L is the fourth low-rank transformation matrix corresponding to the query matrix of the L-th self-attention layer, Wq_1 and Wq_2 are the weight matrices of the second adapter corresponding to the query matrix in the current training round, ReLU2q is the rectified linear unit activation function corresponding to the query matrix, defined as ReLU2q (x) = max(0, x), and σ is the activation function (such as Sigmoid or other nonlinear functions).

[0171] Exemplarily, after obtaining the weight matrices Wq_1 and Wq_2 of the second adapter corresponding to the query matrix in the current training round, Wq_1 and Wq_2 and the fourth low-rank transformation matrix V4^T_L corresponding to the query matrix of the Lth self-attention layer can be input into the above formula to obtain the second adapter adapter2q_L (V4^T_L) corresponding to the query matrix of the Lth self-attention layer.

[0172] S1304: Determine query parameter adjustment information of the self-attention layer according to the query enhancement factor, the second adapter corresponding to the query matrix in the current training round, and the third low-rank transformation matrix.

[0173] Exemplarily, taking the Lth self-attention layer as an example, after obtaining the second adapter adapter2q_L (V4^T_L) corresponding to the query matrix in the current training round, the query parameter adjustment information ΔWq_L of the Lth self-attention layer can be obtained according to the query enhancement factor αq, the second adapter adapter2q_L (V4^T_L) corresponding to the query matrix in the current training round, and the third low-rank transformation matrix U3_L corresponding to the query matrix.

[0174] Exemplarily, the query parameter adjustment information ΔWq_L of the Lth self-attention layer can be expressed using the following formula: ΔWq_L=αq *U3_L*adapter2q_1(V4_L^T) Exemplarily, taking the Lth self-attention layer as an example, after obtaining the query parameter adjustment information ΔWq_L of the Lth self-attention layer, the query parameter adjustment information ΔWq_L can be superimposed on the weight matrix Wq_L corresponding to the query matrix of the Lth self-attention layer to obtain the weight matrix Wq'_L corresponding to the query matrix of the Lth self-attention layer in the current training round, where Wq'_L=Wq_L+ΔWq_L.

[0175] In a possible implementation, the above S1202 determines the key parameter adjustment information of each attention layer according to the key enhancement factor, the loss result of the previous training round, the key matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the key matrix of each attention layer in the previous round, including: S1401. Determine a fifth low-rank transformation matrix and a sixth low-rank transformation matrix according to the image dimension in the training sample data, and the dimension after multiplying the fifth low-rank transformation matrix by the sixth low-rank transformation matrix matches the image dimension.

[0176] Among them, the dimension after multiplying the fifth low-rank transformation matrix and the sixth low-rank transformation matrix matches the image dimension.

[0177] Optionally, a low-rank decomposition method may be used to decompose the image dimensions in the training sample data to obtain a fifth low-rank transformation matrix and a sixth low-rank transformation matrix.

[0178] The dimension of the product of the fifth low-rank transformation matrix and the sixth low-rank transformation matrix matches the dimension of the image. The low-rank decomposition method may include singular value decomposition (SVD) and non-negative matrix decomposition (NMF).

[0179] Exemplarily, in the initial case, a preconfigured target rank can be used to decompose the image dimensions in the training sample data to obtain a fifth low-rank transformation matrix and a sixth low-rank transformation matrix. During the training process, the target rank can be optimized by methods such as singular value decomposition (SVD) and non-negative matrix decomposition (NMF) to achieve optimization of the fifth low-rank transformation matrix and the sixth low-rank transformation matrix.

[0180] For example, assume that the image dimensions in the training sample data are ,in is the height, As width, the fifth low-rank transformation matrix U5 can be obtained as And the sixth low-rank transformation matrix V6 ^T is ,in, is the current target rank.

[0181] S1402. Determine the weight matrix of the second adapter corresponding to the key matrix in the current training round based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the key matrix of the self-attention layer in the previous training round.

[0182] Optionally, taking the Lth self-attention layer as an example, Fig.14 A schematic diagram of determining the key parameter adjustment information of the self-attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, with reference to Fig.14 As shown, x2k_L is the input corresponding to the key matrix of the L-th self-attention layer, W2k_L is the original trainable weight corresponding to the key matrix of the L-th self-attention layer, U5_L is the fifth low-rank transformation matrix corresponding to the key matrix of the L-th self-attention layer, V6^T_L is the sixth low-rank transformation matrix corresponding to the key matrix of the L-th self-attention layer, the fifth low-rank transformation matrix U5_L can be initialized to 0, and the sixth low-rank transformation matrix V6^T_L can be initialized to N2k_L (0, σ²), that is, randomly initialized from a normal distribution with a mean of 0 and a variance of σ², adapter2k_L is the second adapter corresponding to the key matrix of the L-th self-attention layer, h2k_L is the differentiable function corresponding to the key matrix in the L-th self-attention layer, which is used to achieve regularization or transformation to help the model better adapt to new tasks or data sets, and αk is the key enhancement factor.

[0183] The key enhancement factor αk is used to control the parameters of the enhancement amplitude. For example, when αk=1.5, the enhancement factor of the middle layer increases to 1.5 times.

[0184] For example, continue to refer to Fig.14As shown, the gradient can be calculated based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the key matrix of the self-attention layer in the previous training round, so as to adjust the weight matrices Wk_1 and Wk_2 of the second adapter corresponding to the key matrix in the current training round through the back-propagation algorithm.

[0185] S1403. Determine the second adapter corresponding to the key matrix in the current training round according to the weight matrix of the second adapter corresponding to the key matrix in the current training round and the sixth low-rank transformation matrix.

[0186] Optionally, taking the L-th self-attention layer as an example, the second adapter corresponding to the key matrix in the current training round can be calculated based on the weight matrices Wk_1 and Wk_2 of the second adapter corresponding to the key matrix in the current training round and the sixth low-rank transformation matrix V6^T_L corresponding to the key matrix of the L-th self-attention layer.

[0187] Exemplarily, the second adapter adapter2k_L corresponding to the key matrix of the L-th self-attention layer can be expressed as follows: adapter2k _L(V6_L^T) = σ(Wk_2* ReLU2k(Wk_1*V6^T_L)) Where V6^T_L is the sixth low-rank transformation moment corresponding to the key matrix of the L-th self-attention layer, Wk_1 and Wk_2 are the weight matrices of the second adapter corresponding to the key matrix in the current training round, ReLU2k is the rectified linear unit activation function corresponding to the key matrix, defined as ReLU2k (x) = max(0, x), and σ is the activation function (such as Sigmoid or other nonlinear functions).

[0188] Exemplarily, after obtaining the weight matrices Wk_1 and Wk_2 of the second adapter corresponding to the key matrix in the current training round, Wk_1 and Wk_2 and the sixth low-rank transformation matrix V6^T_L corresponding to the key matrix of the L-th self-attention layer can be input into the above formula to obtain the second adapter adapter2k_L (V6^T_L) corresponding to the key matrix of the L-th self-attention layer.

[0189] S1404. Determine key parameter adjustment information of the self-attention layer according to the key enhancement factor, the second adapter corresponding to the key matrix in the current training round, and the fifth low-rank transformation matrix.

[0190] Exemplarily, taking the Lth self-attention layer as an example, after obtaining the second adapter adapter2k_L (V6^T_L) corresponding to the key matrix in the current training round, the key parameter adjustment information ΔWk_L of the Lth self-attention layer can be obtained according to the key enhancement factor αk, the second adapter adapter2k_L (V6^T_L) corresponding to the key matrix in the current training round, and the fifth low-rank transformation matrix U5_L corresponding to the key matrix.

[0191] Exemplarily, the key parameter adjustment information ΔWk_L of the Lth self-attention layer can be expressed using the following formula: ΔWk_L=αk *U5_L*adapter2k_1(V6_L^T) Exemplarily, taking the L-th self-attention layer as an example, after obtaining the key parameter adjustment information ΔWk_L of the L-th self-attention layer, the key parameter adjustment information ΔWk_L can be superimposed on the weight matrix Wk_L corresponding to the key matrix of the L-th self-attention layer to obtain the weight matrix Wk'_L corresponding to the key matrix of the L-th self-attention layer in the current training round, where Wk'_L=Wk_L+ΔWk_L.

[0192] In a possible implementation, the above S1203 determines the value parameter adjustment information of each attention layer according to the value enhancement factor, the loss result of the previous training round, the value matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the value matrix of each attention layer in the previous round, including: S1501. Determine a seventh low-rank transformation matrix and an eighth low-rank transformation matrix according to the image dimension in the training sample data, and the dimension after multiplying the seventh low-rank transformation matrix by the eighth low-rank transformation matrix matches the image dimension.

[0193] Among them, the dimension after multiplying the seventh low-rank transformation matrix and the eighth low-rank transformation matrix matches the image dimension.

[0194] Optionally, a low-rank decomposition method may be used to decompose the image dimensions in the training sample data to obtain a seventh low-rank transformation matrix and an eighth low-rank transformation matrix.

[0195] The dimension of the product of the seventh low-rank transformation matrix and the eighth low-rank transformation matrix matches the dimension of the image. The low-rank decomposition method may include singular value decomposition (SVD) and non-negative matrix decomposition (NMF).

[0196] Exemplarily, in the initial case, a preconfigured target rank can be used to decompose the image dimensions in the training sample data to obtain the seventh low-rank transformation matrix and the eighth low-rank transformation matrix. During the training process, the target rank can be optimized by methods such as singular value decomposition (SVD) and non-negative matrix decomposition (NMF) to achieve the optimization of the seventh low-rank transformation matrix and the eighth low-rank transformation matrix.

[0197] For example, assume that the image dimensions in the training sample data are ,in is the height, As width, the seventh low-rank transformation matrix U7 can be obtained as And the eighth low-rank transformation matrix V8 ^T is ,in, is the current target rank.

[0198] S1502. Determine the weight matrix of the second adapter corresponding to the value matrix in the current training round based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the value matrix of the self-attention layer in the previous training round.

[0199] Optionally, taking the Lth self-attention layer as an example, Fig.15 A schematic diagram of determining the value parameter adjustment information of the self-attention layer in the fine-tuning and reasoning method of the drawing recognition large model provided in the embodiment of the present application, referring to Fig.15 As shown, x2v_L is the input corresponding to the value matrix of the L-th self-attention layer, W2 v _L is the original trainable weight corresponding to the value matrix of the L-th self-attention layer, U7_L is the seventh low-rank transformation matrix corresponding to the value matrix of the L-th self-attention layer, V8^T_L is the eighth low-rank transformation matrix corresponding to the value matrix of the L-th self-attention layer, the seventh low-rank transformation matrix U7_L can be initialized to 0, and the eighth low-rank transformation matrix V8^T_L can be initialized to N2v_L (0, σ²), that is, randomly initialized from a normal distribution with a mean of 0 and a variance of σ², adapter2v_L is the second adapter corresponding to the value matrix of the L-th self-attention layer, h2v_L is the differentiable function corresponding to the value matrix of the L-th self-attention layer, which is used to achieve regularization or transformation to help the model better adapt to new tasks or data sets, and αv is the value enhancement factor.

[0200] The value enhancement factor αv is used to control the parameters of the enhancement amplitude. For example, when αv=1.5, the enhancement factor of the middle layer increases to 1.5 times.

[0201] For example, continue to refer to Fig.15As shown, the gradient can be calculated based on the loss result of the previous training round and the weight matrix of the second adapter corresponding to the value matrix of the self-attention layer in the previous training round, so as to adjust the weight matrices Wv_1 and Wv_2 of the second adapter corresponding to the value matrix in the current training round through the back propagation algorithm.

[0202] S1503: Determine the second adapter corresponding to the value matrix in the current training round according to the weight matrix of the second adapter corresponding to the value matrix in the current training round and the eighth low-rank transformation matrix.

[0203] Optionally, taking the Lth self-attention layer as an example, the second adapter corresponding to the value matrix in the current training round can be calculated based on the weight matrices Wv_1 and Wv_2 of the second adapter corresponding to the value matrix in the current training round and the eighth low-rank transformation matrix V8^T_L corresponding to the value matrix of the Lth self-attention layer.

[0204] Exemplarily, the second adapter adapter2v_L corresponding to the value matrix of the L-th self-attention layer can be expressed as follows: adapter2v _L(V8_L^T) = σ(Wv_2* ReLU2v(Wv_1*V8^T_L)) Among them, V8^T_L is the eighth low-rank transformation moment corresponding to the value matrix of the L-th self-attention layer, Wv_1 and Wv_2 are the weight matrices of the second adapter in the current training round corresponding to the value matrix, ReLU2v is the rectified linear unit activation function corresponding to the value matrix, defined as ReLU2v (x) = max(0, x), and σ is the activation function (such as Sigmoid or other nonlinear functions).

[0205] Exemplarily, after obtaining the weight matrices Wv_1 and Wv_2 of the second adapter corresponding to the value matrix in the current training round, Wv_1 and Wv_2 and the eighth low-rank transformation matrix V8^T_L corresponding to the value matrix of the Lth self-attention layer can be input into the above formula to obtain the second adapter adapter2v_L (V8^T_L) corresponding to the value matrix of the Lth self-attention layer.

[0206] S1504. Determine the value parameter adjustment information of the self-attention layer according to the value enhancement factor, the second adapter corresponding to the median matrix of the current training round, and the seventh low-rank transformation matrix.

[0207] Exemplarily, taking the Lth self-attention layer as an example, after obtaining the second adapter adapter2v_L (V8^T_L) corresponding to the median matrix of the current training round, the value parameter adjustment information ΔWv_L of the Lth self-attention layer can be obtained according to the value enhancement factor αv, the second adapter adapter2v_L (V8^T_L) corresponding to the median matrix of the current training round, and the seventh low-rank transformation matrix U7_L corresponding to the value matrix.

[0208] Exemplarily, the value parameter adjustment information ΔWv_L of the Lth self-attention layer can be expressed using the following formula: ΔWv_L=αv *U7_L*adapter2v_1(V8_L^T) Exemplarily, taking the L-th self-attention layer as an example, after obtaining the value parameter adjustment information ΔWv_L of the L-th self-attention layer, the value parameter adjustment information ΔWv_L can be superimposed on the weight matrix Wv_L corresponding to the value matrix of the L-th self-attention layer to obtain the weight matrix Wv'_L corresponding to the value matrix of the L-th self-attention layer in the current training round, where Wv'_L=Wv_L+ΔWv_L.

[0209] Through the above processing, after freezing the weights of the pre-trained model, the trainable low-rank decomposition matrix can be injected into each layer of the Transformer architecture, thereby greatly reducing the amount of trainable parameters on downstream tasks and enabling the self-attention layer to update the intermediate layers in the self-attention layer more evenly during updating, thereby reducing the performance deviation in the fine-tuning process, improving the generalization ability and overall performance of the target drawing recognition model, and enabling the initial drawing recognition model to converge faster, thereby obtaining the target drawing recognition model more efficiently.

[0210] In a possible implementation, the above S1103 determines the third parameter adjustment information of each second fully connected layer according to the third enhancement factor, the loss result of the previous training round, the weight matrix of each second fully connected layer in the previous training round, and the weight matrix of the third adapter of each second fully connected layer in the previous training round, including: S1901. Determine a ninth low-rank transformation matrix and a tenth low-rank transformation matrix according to the image dimension in the training sample data.

[0211] Optionally, the specific processing process of S1901 can be implemented with reference to the processing process of S801 above. Specifically, the ninth low-rank transformation matrix can be regarded as the first low-rank transformation matrix in S801 above, and the tenth low-rank transformation matrix can be regarded as the second low-rank transformation matrix in S801 above.

[0212] S1902: Determine a weight matrix of the third adapter in the current training round according to the loss result of the previous training round and the weight matrix of the third adapter of the second fully connected layer in the previous training round.

[0213] Optionally, the specific processing process of S1902 can be implemented with reference to the processing process of S802 above. Specifically, the second fully connected layer can be regarded as the first fully connected layer in S802 above, and the third adapter can be regarded as the first adapter in S802 above, so as to determine the weight matrix of the third adapter in the current training round. The second fully connected layer and the first fully connected layer can have the same or different original trainable weights, adapters, differentiable functions and weight matrices, and the third enhancement factor α3 can be the same or different from the first enhancement factor α1.

[0214] S1903: Determine the third adapter in the current training round according to the weight matrix of the third adapter in the current training round and the ninth low-rank transformation matrix.

[0215] Optionally, the specific processing process of S1903 may be implemented by referring to the processing process of S803, so as to calculate and obtain the third adapter adapter3_L (V10_L^T) in the current training round.

[0216] S1904. Determine third parameter adjustment information of the second fully connected layer according to the third enhancement factor, the third adapter in the current training round, and the tenth low-rank transformation matrix.

[0217] Optionally, the specific processing process of S1904 can be implemented with reference to the processing process of S804 above, so as to obtain the third parameter adjustment information ΔW3_L of the Lth second fully connected layer.

[0218] Exemplarily, taking the Lth second fully connected layer as an example, after obtaining the third parameter adjustment information ΔW3_L of the Lth second fully connected layer, the third parameter adjustment information ΔW3_L can be superimposed on the weight matrix W3_L of the Lth second fully connected layer to obtain the weight matrix W3'_L of the Lth second fully connected layer in the current training round, where W3'_L=W3_L+ΔW3_L.

[0219] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for fine-tuning and reasoning the large model of drawing recognition are executed.

[0220] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A fine-tuning and reasoning method for a large model of drawing recognition, characterized in that: include: Acquire original sample data, annotation information, and a task statement, wherein the original sample data includes: a plurality of original engineering drawing images, and the task statement is used to indicate a task type of at least one task to be identified in the original sample data; According to the original sample data and the annotation information, the original sample data is augmented to generate training sample data; In the current training round, the training sample data is input into the image processing module in the initial drawing recognition model to obtain an image sequence to be processed, and the task statement is input into the text processing module in the initial drawing recognition model to obtain a text sequence to be processed, wherein the initial drawing recognition model includes: an image processing module, a text processing module, a feature processing module and a multimodal recognition module; The feature processing module adjusts the parameters of the feature processing module based on a preset first enhancement factor and a loss result of a previous training round, and performs feature processing on the image sequence to be processed and the text sequence to be processed to obtain a feature processing result; The multimodal recognition module adjusts the parameters of the multimodal recognition module based on a preset second enhancement factor, a preset third enhancement factor, and a loss result of a previous training round, and the multimodal recognition module performs multimodal recognition on the feature processing result to obtain a recognition result; According to the recognition result, the true value and the preset loss function, a loss result is determined, and the initial drawing recognition model is iteratively fine-tuned according to the loss result to obtain a target drawing recognition model, and the target drawing recognition model is used for reasoning.

2. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 1 is characterized in that: The augmenting the original sample data according to the original sample data and the annotation information to generate the training sample data includes: Adding the annotation information to the corresponding position in the original sample data to obtain the annotated sample data; Based on a preset rotation matrix, a preset scaling matrix, and a preset cropping matrix, the labeled sample data is randomly rotated, randomly scaled, and randomly cropped in sequence to obtain randomly processed labeled sample data; Performing consistency processing on the labeling information in the randomly processed labeling sample data to obtain target sample data; Image quality adjustment is performed on the target sample data to obtain the training sample data.

3. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 2 is characterized in that: The performing consistency processing on the label information in the randomly processed label sample data to obtain target sample data includes: Determine whether a first annotation text in the annotation information in the randomly processed annotation sample data is complete, wherein the first annotation text is any one of the annotation texts in the annotation information; If yes, retaining the first annotation text in the randomly processed annotation sample data; if no, deleting the first annotation text in the randomly processed annotation sample data; After all randomly processed labeled sample data are processed, the target sample data are obtained.

4. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 2 is characterized in that: The step of adjusting the image quality of the target sample data to obtain the training sample data includes: Performing illumination adjustment on the target sample data to obtain pixel-adjusted target sample data; Performing comparison and adjustment on the target sample data to obtain the compared and adjusted target sample data; Performing fuzzy adjustment on the target sample data to obtain fuzzy adjusted target sample data; The training sample data is generated according to the pixel-adjusted target sample data, the contrast-adjusted target sample data and the blur-adjusted target sample data.

5. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 1 is characterized in that: The feature processing module in the initial drawing recognition model includes: a splicing layer and at least one perception module connected in sequence; the perception module includes: a first fully connected layer and an activation layer; a first adapter is respectively deployed in each first fully connected layer of each perception module, and each first fully connected layer has a weight matrix; The adjusting the parameters of the feature processing module based on the preset first enhancement factor and the loss result of the previous training round includes: Determine first parameter adjustment information of each first fully connected layer according to the first enhancement factor, the loss result of the previous training round, the weight matrix of each first fully connected layer in the previous training round, and the weight matrix of the first adapter of each first fully connected layer in the previous training round; According to each of the first parameter adjustment information, the weight matrix of each of the first fully connected layers is adjusted to obtain the weight matrix of each of the first fully connected layers in the current training round.

6. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 5 is characterized in that: The determining, according to the first enhancement factor, the loss result of the previous training round, the weight matrix of each first fully connected layer in the previous training round, and the weight matrix of the first adapter of each first fully connected layer in the previous training round, the first parameter adjustment information of each first fully connected layer includes: Determine a first low-rank transformation matrix and a second low-rank transformation matrix according to the image dimension in the training sample data, wherein the dimension after multiplying the first low-rank transformation matrix by the second low-rank transformation matrix matches the image dimension; Determine a weight matrix of the first adapter in a current training round according to a loss result of a previous training round and a weight matrix of the first adapter of the first fully connected layer in the previous training round; Determining the first adapter in the current training round according to the weight matrix of the first adapter in the current training round and the second low-rank transformation matrix; Determine first parameter adjustment information of the first fully connected layer according to the first enhancement factor, the first adapter in the current training round, and the first low-rank transformation matrix.

7. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 1 is characterized in that: The multimodal recognition module includes: a plurality of feature recognition modules connected in sequence, each feature recognition module includes: a self-attention layer and a second fully connected layer connected in sequence; a plurality of second adapters are respectively deployed in each attention layer, and a third adapter is respectively deployed in each second fully connected layer; The adjusting the parameters of the multimodal recognition module based on the preset second enhancement factor, the preset third enhancement factor and the loss result of the previous training round includes: Determine second parameter adjustment information of the respective attention layers according to the second enhancement factor, the loss result of the previous training round, the weight matrix of the respective attention layers in the previous training round, and the weight matrix of each second adapter of the respective attention layers in the previous training round; According to each of the second parameter adjustment information, the weight matrix of each attention layer is adjusted to obtain the weight matrix of each attention layer in the current training round; Determining third parameter adjustment information of each second fully connected layer according to the third enhancement factor, the loss result of the previous training round, the weight matrix of each second fully connected layer in the previous training round, and the weight matrix of the third adapter of each second fully connected layer in the previous training round; According to each of the third parameter adjustment information, the weight matrix of each second fully connected layer is adjusted to obtain the weight matrix of each second fully connected layer in the current training round.

8. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 7 is characterized in that: The weight matrix of each attention layer includes a query matrix, a key matrix and a value matrix, the second enhancement factor includes a query enhancement factor, a key enhancement factor and a value enhancement factor, and each second adapter of each attention layer includes a second adapter corresponding to the query matrix, a second adapter corresponding to the key matrix and a second adapter corresponding to the value matrix; The determining the second parameter adjustment information of each attention layer according to the second enhancement factor, the loss result of the previous training round, the weight matrix of each attention layer in the previous training round, and the weight matrix of each second adapter of each attention layer in the previous training round includes: Determine query parameter adjustment information of each attention layer according to the query enhancement factor, the loss result of the previous training round, the query matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the query matrix of each attention layer in the previous round; Determine key parameter adjustment information of each attention layer according to the key enhancement factor, the loss result of the previous training round, the key matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the key matrix of each attention layer in the previous round; Determine value parameter adjustment information of each attention layer according to the value enhancement factor, the loss result of the previous training round, the value matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the value matrix of each attention layer in the previous round; Determine the second parameter adjustment information of each attention layer according to the query parameter adjustment information of each attention layer, the key parameter adjustment information of each attention layer and the value parameter adjustment information of each attention layer.

9. The fine-tuning and reasoning method of the large model for drawing recognition according to claim 8 is characterized in that: Determining query parameter adjustment information of each attention layer according to the query enhancement factor, the loss result of the previous training round, the query matrix of each attention layer in the previous round, and the weight matrix of the second adapter corresponding to the query matrix of each attention layer in the previous round includes: Determine a third low-rank transformation matrix and a fourth low-rank transformation matrix according to the image dimension in the training sample data, wherein the dimension after multiplying the third low-rank transformation matrix by the fourth low-rank transformation matrix matches the image dimension; Determine a weight matrix of the second adapter in the current training round according to the loss result of the previous training round and the weight matrix of the second adapter in the previous training round corresponding to the query matrix of the self-attention layer; Determining the second adapter in the current training round according to the weight matrix of the second adapter in the current training round and the third low-rank transformation matrix; Determine query parameter adjustment information of the self-attention layer according to the query enhancement factor, the second adapter in the current training round, and the fourth low-rank transformation matrix.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for fine-tuning and reasoning a large model for drawing recognition as described in any one of claims 1 to 9.

Images