Drawing recognition method and device, electronic equipment and storage medium

Abstract

The invention discloses a drawing recognition method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a to-be-recognized drawing; performing feature extraction on the to-be-identified drawing by using a multi-modal large model to obtain a multi-modal feature vector; according to a fusion strategy, performing feature fusion processing on the multi-modal feature vector to obtain a target fusion feature vector; identifying the target fusion feature vector to obtain a drawing identification result; the model parameters of the multi-modal large model and / or the fusion strategy are / is obtained by performing adaptive learning optimization on different drawings. According to the method, the to-be-recognized drawing is recognized by using the multi-modal large model and / or the fusion strategy of the adaptive learning optimization, so that the accuracy of drawing recognition is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a drawing recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the field of drawing recognition, although traditional optical character recognition (OCR) technology and single-modal text processing models can extract text information, they are difficult to accurately recognize architectural structures and layouts in the face of complex graphics, lines and other visual elements in drawings. At the same time, although image recognition models can recognize graphic elements, they lack understanding of text information such as text descriptions on drawings. In short, OCR technology, single-modal text processing models and image recognition models cannot comprehensively integrate image and text information in drawings, resulting in low recognition accuracy of drawings. SUMMARY

[0003] The main purpose of the present application is to provide a drawing recognition method, device, electronic equipment and storage medium, which aims to improve the accuracy of drawing recognition.

[0004] To achieve the above purpose, the present application provides a drawing recognition method, comprising:

[0005] obtaining a to-be-recognized drawing;

[0006] extracting features of the to-be-recognized drawing by using a multi-modal large model to obtain a multi-modal feature vector;

[0007] performing feature fusion processing on the multi-modal feature vector according to a fusion strategy to obtain a target fusion feature vector;

[0008] recognizing the target fusion feature vector to obtain a drawing recognition result;

[0009] The model parameters of the multi-modal large model and / or the fusion strategy are obtained by self-adaptive learning and optimization of different drawings.

[0010] In an embodiment, the multi-modal feature vector comprises a target image feature vector and a target text feature vector.

[0011] The feature fusion processing on the multi-modal feature vector according to the fusion strategy to obtain the target fusion feature vector comprises:

[0012] normalizing the target image feature vector and the target text feature vector, and mapping the normalized target image feature vector and the target text feature vector to the same semantic space;

[0013] According to the fusion strategy, the target image feature vector and the target text feature vector of the same semantic space are subjected to feature fusion processing to obtain the target fusion feature vector.

[0014] In an embodiment, the multi-modal large model comprises a first target large model and a second target large model, and the first target large model and the second target large model transmit feature vectors through a preset protocol; the first target large model is used to extract the target image feature vector of the to-be-recognized drawing; and the second target large model is used to extract the target text feature vector of the to-be-recognized drawing.

[0015] In an embodiment, the adaptive learning process of the model parameters of the multi-modal large model and / or the fusion strategy comprises the following steps:

[0016] Feedback information of a user is obtained, wherein the feedback information is information of the user correcting and labeling the drawing recognition result;

[0017] According to the feedback information and the drawing recognition result, the error rate and the accuracy rate of the multi-modal large model recognition are determined;

[0018] According to the error rate and the accuracy rate of the multi-modal large model recognition, the model parameters of the multi-modal large model and / or the fusion strategy are dynamically adjusted.

[0019] In an embodiment, the adaptive learning process of the model parameters of the multi-modal large model comprises the following steps:

[0020] Different types, styles and qualities of drawing samples are obtained;

[0021] The multi-model large model is used to extract features of each drawing sample to obtain a to-be-trained image feature vector and a to-be-trained text feature vector of each drawing sample;

[0022] According to the sample label, the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample, the model parameters of the multi-modal large model are iteratively optimized.

[0023] In an embodiment, the adaptive learning process of the fusion strategy comprises the following steps:

[0024] Different fusion strategies are used to perform feature fusion processing on the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample to obtain a to-be-trained fusion feature vector corresponding to each fusion strategy;

[0025] The to-be-trained fusion feature vector of each fusion strategy is subjected to recognition to obtain a target recognition result;

[0026] According to the sample label of each drawing sample and the target recognition result corresponding to the different fusion strategies, the fusion strategies are optimized.

[0027] In an embodiment, the feature extraction of each drawing sample by using the multi-model large model to obtain the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample comprises:

[0028] The local feature information and the global structure feature information of each drawing sample are recognized by using a pre-constructed feature enhancement module.

[0029] The to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample are extracted by using the multi-model large model in combination with the local feature information and the global structure feature information of each drawing sample.

[0030] In an embodiment, the recognition of the target fusion feature vector to obtain a drawing recognition result comprises:

[0031] The target fusion feature vector is classified to obtain a classification result.

[0032] The target fusion feature vector is content-recognized to obtain a content recognition result.

[0033] The drawing recognition result is generated according to the classification result and the content recognition result.

[0034] In an embodiment, before the feature extraction of the to-be-recognized drawing by using the multi-modal large model to obtain the multi-modal feature vector, the method further comprises:

[0035] The to-be-recognized drawing is analyzed to obtain the overall layout and element information of the to-be-recognized drawing.

[0036] According to the overall layout and element information of the to-be-recognized drawing, it is determined whether the to-be-recognized drawing belongs to a preset drawing type.

[0037] If yes, the step of the feature extraction of the to-be-recognized drawing by using the multi-modal large model to obtain the multi-modal feature vector is executed.

[0038] In addition, to achieve the above-mentioned purpose, the present application further provides a drawing recognition device, which comprises:

[0039] The acquisition module is configured to acquire a to-be-recognized drawing.

[0040] The feature extraction module is configured to extract features of the to-be-recognized drawing by using a multi-modal large model to obtain a multi-modal feature vector.

[0041] a feature fusion module, configured to perform feature fusion processing on the multi-modal feature vector according to a fusion strategy, to obtain a target fusion feature vector;

[0042] a recognition module, configured to recognize the target fusion feature vector, to obtain a drawing recognition result;

[0043] The model parameters of the multi-modal large model and / or the fusion strategy are obtained by adaptive learning and optimization on different drawings.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the drawing recognition method as described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the drawing recognition method as described above.

[0046] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the drawing recognition method as described above.

[0047] The present application provides a drawing recognition method, device, electronic device and storage medium, the drawing recognition method comprises obtaining a drawing to be recognized; using a multi-modal large model to extract features of the drawing to be recognized, to obtain a multi-modal feature vector; performing feature fusion processing on the multi-modal feature vector according to a fusion strategy, to obtain a target fusion feature vector; recognizing the target fusion feature vector, to obtain a drawing recognition result; the model parameters of the multi-modal large model and / or the fusion strategy are obtained by adaptive learning and optimization on different drawings. By adaptive learning and optimization of multi-modal large model and fusion strategy according to different drawings, the multi-modal large model is adapted to the recognition scene of various complex drawings. And with the help of powerful visual feature extraction and text semantic analysis ability of the multi-modal large model, the image features and text features of the drawing can be comprehensively understood, and then the multi-modal feature vector is deeply fused according to the fusion strategy, which can more comprehensively capture the information in the drawing, and effectively improve the accuracy of drawing recognition. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings are only for the purpose of illustrating the embodiments of the present application, and for the person skilled in the art, other drawings can also be obtained without creative labor.

[0050] Figure 1 The flowchart provided for the first embodiment of the drawing identification method of the present application is shown in the figure.

[0051] Figure 2 The flowchart provided for the second embodiment of the drawing identification method of the present application is shown in the figure.

[0052] Figure 3 The model architecture provided by an embodiment of the present application is shown in the figure.

[0053] Figure 4 The flowchart provided for the third embodiment of the drawing identification method of the present application is shown in the figure.

[0054] Figure 5 The flowchart provided for the fourth embodiment of the drawing identification method of the present application is shown in the figure.

[0055] Figure 6 The flowchart provided for the fifth embodiment of the drawing identification method of the present application is shown in the figure.

[0056] Figure 7 The flowchart provided for the sixth embodiment of the drawing identification method of the present application is shown in the figure.

[0057] Figure 8 The module structure diagram of the drawing identification device of the present application is shown in the figure.

[0058] Figure 9 The device structure diagram of the hardware running environment involved in the drawing identification method in the embodiment of the present application is shown in the figure.

[0059] The purpose of the present application, the functional characteristics and the advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0061] In order to better understand the technical solutions of the present application, the accompanying drawings and specific embodiments will be described in detail below.

[0062] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a big data service platform, a drawing recognition system, etc. capable of realizing the above functions. The following will take the drawing recognition system as an example to describe the embodiment and the following embodiments.

[0063] Based on this, the embodiment of the present application provides a drawing recognition method, which is described with reference to Figure 1 , Figure 1 The flowchart provided by the first embodiment of the drawing recognition method of the present application. In this embodiment, the drawing recognition method comprises the following steps:

[0064] Step S11, obtaining a to-be-recognized drawing;

[0065] It should be noted that the to-be-recognized drawing can be a completed drawing, a circuit drawing, etc. The completed drawing is a drawing drawn by the construction unit according to the actual situation of construction after the completion of the project, aiming to truly, accurately and systematically reflect the actual situation of the project entity. The circuit drawing is a drawing used to represent the composition, connection mode, working principle and other information of the circuit system. In this embodiment, the completed drawing is taken as an example for illustration. The completed drawing includes the position, size label and text description of each element information such as building contour, wall, door and window, etc.

[0066] In an embodiment, the to-be-recognized drawing can be uploaded to the drawing recognition system by the user. In an embodiment, the drawing recognition system can be integrated with other management systems (such as project management software, document management system, etc.) through API interface to automatically obtain the to-be-recognized drawing stored in the management system.

[0067] Step S12, using a multi-modal large model to extract features of the to-be-recognized drawing to obtain a multi-modal feature vector;

[0068] It should be noted that the multi-modal large model includes a first target large model and a second target large model; wherein the first target large model includes Qwen2-VL, GPT-4V (GPT-4Vision), LLaVA-1.5, etc. large model, and the second target large model includes GLM-4V, CAD-VL, DocVQA (Document Visual Question Answering) etc. large model. Optionally, Qwen2-VL large model and GLM-4V large model are selected to recognize the to-be-recognized drawing. The multi-modal feature vector includes a target image feature vector and a target text feature vector.

[0069] It should be noted that the Qwen2-VL large model can accurately identify the image information such as graphic elements, lines, shapes and the like in the as-built drawing by virtue of its powerful visual feature extraction capability, for example, the outlines and shapes of building components such as walls, columns, doors and windows, and the lines and arrows of size annotations and the like. The Qwen2-VL large model can also preliminarily understand the association between the image and the text, laying a foundation for subsequent deep fusion. The GLM-4V large model can deeply analyze the text semantics and process complex natural language information in terms of language understanding, which is crucial for interpreting the textual descriptions, annotations and the like on the drawing, for example, accurately understanding the engineering meaning expressed by the textual information such as size values, room names, material descriptions and the like. The combination of the Qwen2-VL large model and the GLM-4V large model can achieve comprehensive understanding of the text and image of the drawing.

[0070] In the present embodiment, the Qwen2-VL large model is responsible for visual feature extraction of the as-built drawing to obtain a target image feature vector, and the GLM-4V large model is responsible for text semantic understanding of the as-built drawing to obtain a target text feature vector.

[0071] It should be noted that the first target large model and the second target large model transmit the feature vectors through a preset protocol; that is, the Qwen2-VL large model and the GLM-4V large model communicate and cooperate through the MCP (Multi-Contextual Protocol) protocol and can share image and text information. In a feasible implementation manner, in the feature extraction process, the cooperation between the Qwen2-VL large model and the GLM-4V large model can be guided bidirectionally through the MCP protocol, for example, the GLM-4V analyzes the text features (such as C1 window is arranged on the Q2 shear wall between the 2-axis and the 3-axis, and the hole size is 1800x1500mm), and feeds back the text features to the Qwen2-VL through the MCP protocol. The Qwen2-VL will focus on strengthening the visual feature extraction of the elements corresponding to the text features to more comprehensively capture the information in the drawing.

[0072] In step S13, the multi-modal feature vectors are subjected to feature fusion processing according to a fusion strategy to obtain a target fusion feature vector.

[0073] It should be noted that the fusion strategy includes feature splicing, weighted fusion and the like, wherein the weight parameters in the weighted fusion strategy are not fixed, and the weight parameters can also be adaptively learned and optimized for different types, styles and qualities of drawings.

[0074] In an embodiment, the target image feature vector and the target text feature vector are subjected to feature fusion processing according to a fusion strategy, to obtain a target fusion feature vector. Optionally, in the process of weighted fusion processing, the initial value of the weight parameter can be set according to actual conditions, for example, the initial weight of the target image feature vector is set to 0.5 (image feature) and the text weight of the target image feature vector is set to 0.5, and the subsequent weight parameter is adjusted through adaptive learning. Then, the target image feature vector and the target text feature vector are subjected to feature fusion processing by using the adjusted weight parameter, to obtain the target fusion feature vector. Understandably, the target fusion feature vector = a x target image feature vector + b x target text feature vector, where a and b are the weights of the target image feature vector and the target text feature vector, respectively, and a + b = 1.

[0075] In another embodiment, the target image feature vector of Qwen2-VL and the target text feature vector of GLM-4V are normalized by the MCP protocol, and the target image feature vector and the target text feature vector in the same semantic space are subjected to feature fusion processing according to the fusion strategy, to obtain the target fusion feature vector. In other embodiments, the target image feature vector of Qwen2-VL and the target text feature vector of GLM-4V are mapped to the same semantic space by the MCP protocol, so that the visual information and the text information can interact in depth at the semantic level, and then the target image feature vector and the target text feature vector in the same semantic space are subjected to feature fusion processing according to the fusion strategy, to obtain the target fusion feature vector, so that the model can capture the information in the drawing more comprehensively and improve the recognition accuracy.

[0076] It should be noted that the model parameters and / or the fusion strategy of the multi-modal large model are obtained through adaptive learning and optimization of different drawings, wherein the different drawings include drawings of different types, styles and qualities, etc.

[0077] In an embodiment, the model parameters and / or the fusion strategy of the multi-modal large model are iteratively optimized according to drawings of different types, styles and qualities, so that the model can automatically adjust the fusion strategy and the model parameters according to the data characteristics of different as-built drawings, thereby better adapting to various complex scenarios such as different building types, drawing styles, image quality, etc.

[0078] In an embodiment, in the process of model application, feedback information of users is collected, for example, correction, labeling and other information of the recognition result. Optionally, a user interaction interface is developed to facilitate users to correct and label the recognition result, so as to obtain feedback information, and then the fusion strategy and the model parameters of the multi-modal large model are dynamically adjusted according to the feedback information of the users and the drawing recognition result.

[0079] In an embodiment, the staff periodically collects new drawing samples to fine-tune the model parameters and fusion strategies of the multi-modal large model.

[0080] In other embodiments, common error types and problems in user feedback can also be analyzed, and specific training samples can be constructed to dynamically adjust the model parameters and fusion strategies of the multi-modal large model, thereby improving the recognition accuracy and adaptability of the model.

[0081] In step S14, the target fusion feature vector is recognized to obtain a drawing recognition result.

[0082] In an embodiment, the Qwen2-VL large model can be used to recognize the target fusion feature vector. The Qwen2-VL large model in this embodiment is a visual-linguistic multi-modal large model, which has strong visual feature extraction and text analysis capabilities. By transmitting the fusion feature to the Qwen2-VL large model for recognition, the accuracy and robustness of the completed drawing recognition can be effectively improved. In other embodiments, other models can also be set in the decision layer for recognition.

[0083] In an embodiment, the target fusion feature vector is classified to classify different elements in the drawing, such as identifying building components such as walls, doors and windows, columns, etc., to obtain a classification result.

[0084] In another embodiment, the text and graphics in the drawing are recognized to extract specific size values, room names, and other key information, to obtain a content recognition result.

[0085] In other embodiments, a corresponding description or report can also be generated according to the classification result and the content recognition result to obtain a drawing recognition result, such as generating a layout description of the building plan, a size annotation list, and other reports.

[0086] In this embodiment, the multi-modal large model and / or fusion strategy are obtained by self-adaptive learning and optimization according to different drawings, so that the multi-modal large model adapts to various complex drawing recognition scenarios. Moreover, with the powerful visual feature extraction and text semantic analysis capabilities of the multi-modal large model, the image features and text features of the drawing can be comprehensively understood, and then the multi-modal feature vector can be deeply fused according to the fusion strategy, so that the information in the drawing can be more comprehensively captured, and the accuracy of drawing recognition can be effectively improved.

[0087] In a feasible implementation manner, referring to Figure 2 , Figure 2 is a flowchart provided by the second embodiment of the drawing recognition method of the present application; according to the fusion strategy, the multi-modal feature vector is subjected to feature fusion processing to obtain a target fusion feature vector, including:

[0088] Step S21, normalizing the target image feature vector and the target text feature vector, and mapping the normalized target image feature vector and the target text feature vector to the same semantic space;

[0089] Step S22, performing feature fusion processing on the target image feature vector and the target text feature vector in the same semantic space according to the fusion strategy to obtain the target fusion feature vector.

[0090] It should be noted that, with reference to Figure 3 , Figure 3 is a model architecture diagram provided by an embodiment of the present application, wherein the Qwen2-VL large model and the GLM-4V large model realize fusion processing of the multi-modal feature vector through a preset protocol. Optionally, the preset protocol is an MCP protocol, which is an intelligent collaboration protocol designed for multi-modal models. Specifically, the target image feature vector and the target text feature vector are normalized through the MCP protocol, so that the image feature vector and the text feature vector have the same scale range. Then, the normalized target image feature vector and the target text feature vector are analyzed to learn the semantic correspondence of the target image feature vector and the target text feature vector, so as to map the target image feature vector and the target text feature vector to the same semantic space, so that the model can better understand the association between the image and the text.

[0091] Further, in the feature fusion process, a fixed fusion strategy is mostly used at present, and when facing different styles, scales, clarity, etc. of drawings, the recognition accuracy fluctuates greatly. However, the fusion strategy in the embodiment is self-adaptive learning and optimization for different types, styles and quality of drawings, and the fusion strategy is optimized according to the data characteristics of different drawings, so that the model has stronger adaptability and stability when facing various complex scenes. Specifically, according to the fusion strategy, the target image feature vector and the target text feature vector in the same semantic space are subjected to feature fusion processing to obtain the target fusion feature vector. For example, when the input is a drawing with poor image quality but complete text information, the model will automatically reduce the weight of the target image feature vector and increase the weight of the target text feature vector to rely more on the text information for recognition.

[0092] The embodiment uses the fusion strategy of adaptive learning to deeply fuse the target image feature vector and the target text feature vector. Compared with the simple splicing or weighting fusion scheme, the fusion strategy in the embodiment can make the model more comprehensively capture the information in the drawing, effectively improving the accuracy of drawing recognition.

[0093] In a feasible implementation manner, with reference to Figure 4 , Figure 4A flowchart is provided for the third embodiment of the drawing identification method of the present application. The adaptive learning process of the model parameters of the multi-modal large model and / or the fusion strategy includes the following steps:

[0094] In step S31, feedback information of the user is obtained, wherein the feedback information is information of the user correcting and labeling the drawing identification result.

[0095] In step S32, the error rate and accuracy of the multi-modal large model identification are determined according to the feedback information and the drawing identification result.

[0096] In step S33, the model parameters of the multi-modal large model and / or the fusion strategy are dynamically adjusted according to the error rate and accuracy of the multi-modal large model identification.

[0097] It should be noted that a user interaction interface is pre-constructed. After obtaining the drawing identification result, the user can view the drawing identification result on the user interaction interface, and the user can correct and label the drawing identification result, etc. to obtain the feedback information of the user. According to the difference between the feedback information and the drawing identification result, the error rate and accuracy of the multi-modal large model identification are calculated, wherein the error rate is calculated according to the total number of drawing identification results and the number of user corrections, and the accuracy is calculated according to the number of correct identification in the drawing identification result and the total number of identification. Further, according to the error rate and accuracy of the multi-modal large model identification, the reinforcement learning algorithm is used to dynamically adjust the fusion strategy and / or the model parameters of the multi-modal large model, wherein the reinforcement learning algorithm includes Q learning, deep reinforcement learning, etc. Alternatively, a reward function is defined. When the drawing identification result is correct and the accuracy is high, a positive reward is given to increase the weight of the corresponding feature mode; when the drawing identification result is incorrect, a negative reward is given to reduce the weight of the corresponding feature mode. For example, in a certain as-built drawing identification, the model incorrectly identifies a column Z1 (cross section 500x500mm) as a wall Q1 (thickness 200mm). The user feedbacks the correction as follows: image area: the model correctly frames the position of the column Z1 (i.e., the image feature extraction is correct); text analysis: the model incorrectly analyzes Z1 500x500 as Q1 200 (i.e., the text semantic error), at this time, the weight of the target text feature vector is reduced.

[0098] The present embodiment dynamically adjusts the fusion strategy and / or the model parameters of the multi-modal large model according to the feedback information of the user and the drawing identification result, continuously learns and updates the weights, so that the model can adapt to different types and styles of as-built drawings, and improve the accuracy of drawing identification.

[0099] In a feasible implementation, with reference to Figure 5 , Figure 5A flowchart provided for the fourth embodiment of the drawing identification method of the present application is shown in the figure; the adaptive learning process of the model parameters of the multi-modal large model comprises the following steps:

[0100] Step S41, obtain drawing samples of different types, styles and qualities;

[0101] It should be noted that the types include residential building drawings, commercial building drawings, industrial building drawings, etc. The residential building drawings pay more attention to the wall partition and room function labeling of the house type. The commercial building drawings focus on the identification of the flow line and facility distribution of the public area. For industrial building drawings, the identification of equipment installation location and process flow is strengthened. In the context of as-built drawings, style refers to the differentiated characteristics of as-built drawings in visual presentation form, drawing specification, and information layout logic. Different styles of drawing samples are collected, such as line thickness, type, standardization of professional symbols, text density, and format. Quality refers to the clarity of the drawing.

[0102] In this embodiment, different types, styles and quality of drawing samples are collected through artificial uploading, crawler technology and other methods. Optionally, after obtaining the drawing samples, the drawing samples are preprocessed, for example, using noise reduction algorithms such as Gaussian filtering, median filtering, sharpening algorithms such as Laplacian sharpening, etc. to improve the quality of the drawings.

[0103] In addition, for multi-language drawing samples, that is, the text part of the drawing is written in different languages, but expresses the same or similar engineering meaning, for example, a bedroom (Chinese) and a bedroom (English) are labeled in a room area at the same time. In the preprocessing stage, the multi-language text in the drawing sample is detected and recognized, for example, using optical character recognition and other text recognition technologies, and then the recognized multi-language text is converted into a unified text encoding format, such as Unicode encoding format, so that the multi-modal large model can recognize multi-language as-built drawings.

[0104] In addition, different types, styles and quality of drawing samples need to be classified and labeled so that the model can learn the characteristics and patterns of different drawings.

[0105] Step S42, using the multi-model large model to extract features of each drawing sample, obtaining the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample;

[0106] Step S43, according to the sample label of each drawing sample, the to-be-trained image feature vector and the to-be-trained text feature vector, iteratively optimizing the model parameters of the multi-modal large model.

[0107] In this embodiment, the Qwen2-VL large model is used to extract image features of the drawing samples to obtain the training image feature vectors of the drawing samples. In addition, the GLM-4V large model is used to extract text features of the drawing samples to obtain the training text feature vectors of the drawing samples. For example, the Qwen2-VL large model extracts image features in the drawing samples, such as the edges of walls, the shapes of doors and windows, etc.; and the GLM-4V large model analyzes the text annotations, such as size values, room names, etc.

[0108] Further, according to the sample labels of the drawing samples and the target recognition results, the model loss values of the Qwen2-VL large model and the GLM-4V large model are calculated using a preset loss function, for example, a cross-entropy loss function is used to calculate the model loss values, and then the model parameters of the Qwen2-VL large model and the model parameters of the GLM-4V large model are optimized according to the model loss values.

[0109] It should be noted that in the optimization process of the model parameters of the Qwen2-VL large model and the GLM-4V large model, the method of transfer learning can be used to train and optimize part of the parameters of the large model. Optionally, part of the bottom layer parameters of the model are frozen. The bottom layer parameters retain the general features learned by the pre-trained model on a large amount of data, and only the parameters of the top layer of the model are updated to reduce the amount of calculation and avoid overfitting. For example, for the Qwen2-VL large model: the convolutional layers and the Transformer layers of the front preset number of layers of the visual encoder are frozen; and the Transformer layers of the rear preset number of layers of the visual encoder are fine-tuned. For the GLM-4V large model: the Transformer layers of the front preset number of layers of the text encoder are frozen, and the bottom layer parameters of the cross-modal attention layer are frozen. The Transformer layers of the rear preset number of layers of the text encoder, the top layer parameters of the cross-modal attention layer, and the output layer are fine-tuned, thereby improving the training efficiency of the model and avoiding overfitting of the model.

[0110] In addition, new completed drawings can also be collected regularly, preprocessed and labeled, and then the new completed drawings and the old new completed drawings are used to fine-tune the Qwen2-VL large model and the GLM-4V large model.

[0111] In this embodiment, the model parameters of the multi-modal large model are optimized by using drawings of different types, styles and qualities, so that the multi-modal large model adapts to various complex drawing recognition scenarios, and the accuracy of drawing recognition is effectively improved.

[0112] In one possible implementation, referring to Figure 6 , Figure 6 is a flowchart provided by the drawing recognition method embodiment five of the present application; the adaptive learning process of the fusion strategy includes the following steps:

[0113] In step S51, the feature vectors of the to-be-trained images and the to-be-trained text of each drawing sample are fused by using different fusion strategies to obtain to-be-trained fusion feature vectors corresponding to different fusion strategies.

[0114] In step S52, the to-be-trained fusion feature vectors of each fusion strategy are identified to obtain target recognition results.

[0115] In step S53, the fusion strategies are optimized according to the sample labels of the drawing samples and the target recognition results corresponding to different fusion strategies.

[0116] In this embodiment, after the multi-model large model is used to extract features of each drawing sample to obtain the to-be-trained image feature vectors and the to-be-trained text feature vectors of each drawing sample, the following operations are performed for each drawing sample:

[0117] A plurality of different fusion strategies are selected for experiments, such as weighted fusion and attention fusion. The to-be-trained image feature vectors and the to-be-trained text feature vectors of the drawing samples are fused by using different fusion strategies to obtain to-be-trained fusion feature vectors corresponding to different fusion strategies. Further, the to-be-trained fusion feature vectors of each fusion strategy are sent to a decision layer to classify, identify, and the like, the to-be-trained fusion feature vectors of each fusion strategy, to obtain target recognition results of each fusion strategy.

[0118] Further, the model performance indicators corresponding to different fusion strategies are calculated according to the sample labels of the drawing samples and the target recognition results corresponding to different fusion strategies, wherein the model performance indicators include quantitative indicators such as accuracy and recall rate, and then a fusion strategy with better model performance indicators is selected. In other embodiments, the hyperparameters in the fusion strategy, such as weights and parameters of the attention mechanism, can also be adjusted.

[0119] Further, the adjustment of the fusion strategy is continued according to the drawing samples that have not participated in the optimization until a preset ending condition is reached, the preset ending condition includes that the number of optimizations reaches a preset number, the model performance indicators reach a preset indicator threshold, and the like, and then a fusion strategy with optimal model performance indicators is selected as the final fusion strategy. In addition, when the model performance indicators are lower than the preset indicator threshold, the model parameters of the Qwen2-VL large model and the model parameters of the GLM-4V large model can also be further optimized.

[0120] In this embodiment, the fusion strategy is optimized by using different types, styles, and qualities of drawings, and the optimized fusion strategy can be used for deep fusion of multi-modal feature vectors in the future, which can comprehensively capture information in the drawings and effectively improve the accuracy of drawing recognition.

[0121] In an implementable embodiment, before the feature extraction of each drawing sample by the multi-model large model, the method further comprises:

[0122] Step S61, identifying the local feature information and the global structural feature information of each drawing sample by the feature enhancement module;

[0123] Step S62, combining the local feature information and the global structural feature information of each drawing sample, and extracting the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample by the multi-model large model.

[0124] It should be noted that, considering that there are a large number of local features in the drawing, for example, the direction of the connection line, etc., in the present embodiment, the feature enhancement module is arranged to enhance the feature extraction capability of the model for the drawing. Optionally, the feature enhancement module includes a convolutional neural network, by increasing the number of convolutional layers and pooling layers in the convolutional neural network (CNN), the model can more accurately capture the local features of the drawing, and in other embodiments, the feature enhancement module can also be a depth separable convolution module. In addition, for the global structural information in the drawing, for example, the overall layout of the drawing, the relative position relationship between the modules, etc., the feature enhancement module introduces an attention mechanism structure, so that the model can better understand the global semantic features of the drawing.

[0125] It should be noted that, by adding the feature enhancement module outside the multi-modal large model, for example, adding a convolutional neural network in the front end of the large model, or inserting an attention mechanism module in the middle layer of the multi-modal large model, the original model structure of the Qwen2-VL large model and the GLM-4V large model does not need to be modified, the parameters of the feature enhancement module are trained separately, and will not cover or modify the input and output logic of the original layer of the multi-modal large model.

[0126] In the present embodiment, the local feature information and the global structural feature information of each drawing sample are identified by the feature enhancement module; so that the multi-model large model can learn the local features and the global semantic features of the drawing, and then the target image feature vector and the target text feature vector of the drawing sample are comprehensively extracted by the multi-model large model in combination with the local feature information and the global structural feature information of each drawing sample.

[0127] By using the feature enhancement module to more accurately capture the local features and the global semantic features of the drawing, the subsequent model can accurately identify the image feature information and the text feature information in the drawing according to the local features and the global semantic features of the drawing.

[0128] In an implementable embodiment, the multi-modal large model comprises a first target large model and a second target large model, the first target large model is used to extract a target image feature vector of the to-be-identified drawing; and the second target large model is used to extract a target text feature vector of the to-be-identified drawing.

[0129] It should be noted that Qwen2-VL is a visual-linguistic multi-modal model, and image feature extraction focuses on capturing the spatial structure, shape features and visual details of the graphic elements in the to-be-identified drawing.

[0130] In an implementable embodiment, the to-be-identified drawing is segmented into non-overlapping blocks (for example, 2048x2048 images generate 128x128=16384 patches) by using a sliding window, each patch corresponds to a local area (such as a section of pipeline or a door and window symbol) in the drawing. Further, each patch is converted into a vector of a preset dimension by a learnable linear projection matrix to form an initial visual token. Further, the initial visual token is spatially feature-encoded, specifically: a relative position code is added to each visual token, the code includes information such as the coordinates of the initial visual patch in the image and the distance information from other initial visual patches, to ensure that the model understands the spatial layout of the graphic elements, for example, a column is on the left side of a wall, a pipeline crosses a floor. Further, the visual token with position information is input into a Transformer encoder, optionally, global correlation can be captured through a self-attention mechanism, for example, low-level attention focuses on local features (such as the continuity of lines and the shape of symbols); high-level attention integrates global structure (such as room layout and circuit system topology). Finally, a target image feature vector including global semantics is output, and the target image feature vector covers the visual features of all graphic elements in the drawing.

[0131] It should be noted that GLM-4V is a language-dominated multi-modal model, and its text feature extraction focuses on analyzing the semantic meaning and context association of the text information in the drawing.

[0132] In a feasible embodiment, the drawing to be identified is subjected to text segmentation and Tokenization, for example, the text is divided into semantic units, and the segmentation result is converted into a target Token recognizable by the model. Optionally, a feature mark is added at the head of the sequence of the target Token, and an end-of-sequence mark is added at the tail of the target Token, for example, the feature mark is [CLS], and the end-of-sequence mark is [SEP]. Further, the semantic features of each target Token are encoded, and optionally, each Token is converted into a vector of a preset dimension, and position encoding is added to obtain an embedding vector. Further, the embedding vector is input into an encoder, the semantic association is captured through a self-attention mechanism, and finally a target text feature vector including global semantics is output, and the target text feature vector covers the semantic meaning of all text information.

[0133] The embodiment can deeply extract the graphic structure, spatial relationship, visual symbol, etc. of the drawing by performing image feature extraction on the drawing to be identified through the Qwen2-VL large model. In addition, the embodiment can deeply analyze the text information in the completed drawing by performing text feature extraction on the drawing to be identified through the GLM-4V large model, and perform semantic association analysis in combination with the drawing content, so as to accurately interpret the engineering meaning expressed by various symbols, annotations and textual descriptions in the drawing, and effectively improve the accuracy of subsequent drawing recognition.

[0134] In a feasible implementation manner, the target drawing is subjected to image feature extraction and text feature extraction, and the image feature extraction and the text feature extraction are performed through the Qwen2-VL large model and the GLM-4V large model respectively. Figure 7 , Figure 7 The embodiment six of the drawing recognition method provided in the present application is shown in the flowchart; the target fusion feature vector is recognized to obtain a drawing recognition result, including:

[0135] Step S71, the target fusion feature vector is classified to obtain a classification result;

[0136] Step S72, the target fusion feature vector is subjected to content recognition to obtain a content recognition result;

[0137] Step S73, the drawing recognition result is generated according to the classification result and the content recognition result.

[0138] In this embodiment, the target fusion feature vector is input to the decision layer for classification, recognition and the like. Optionally, a support vector machine (SVM), a decision tree or the like is used to classify the target fusion feature vector, so as to classify the building components in the drawing and obtain a classification result. Optionally, an optical character recognition (OCR) technology is used in combination with a language model to recognize the text and extract key information such as size values and room names, so as to obtain a content recognition result. Further, the drawing recognition result is generated according to the classification result and the content recognition result. The drawing recognition result includes a layout description of the architectural plan, including the distribution of functional areas such as walls, doors, windows and rooms, a size annotation list, size parameters of each building component and the like.

[0139] It should be noted that the execution order of steps S71 and S72 is not limited in this embodiment, and steps S71 and S72 can be executed in parallel in order to improve the efficiency of recognition.

[0140] In this embodiment, the target fusion feature vector is classified, recognized and the like, the model can more comprehensively capture the information in the drawing, improve the accuracy of drawing recognition, and convert the target fusion feature vector into drawing information that can be directly understood and used by engineers, thereby providing core data support for subsequent drawing review, design optimization and the like.

[0141] In a feasible implementation, before the step of extracting the multi-modal feature vector from the to-be-recognized drawing by using the multi-modal large model, the method further includes:

[0142] Step S81: analyzing the to-be-recognized drawing to obtain overall layout and element information of the to-be-recognized drawing;

[0143] Step S81: determining whether the to-be-recognized drawing belongs to a preset drawing type according to the overall layout and element information of the to-be-recognized drawing.

[0144] Step S81: if yes, executing the step of extracting the multi-modal feature vector from the to-be-recognized drawing by using the multi-modal large model.

[0145] In this embodiment, the overall layout of the to-be-recognized drawing is analyzed, for example, a pre-trained model (such as YOLO or Faster R-CNN) is used to detect key layouts in the drawing, such as the boundary of a wall and the contour of a room. In addition, element information such as text content in the to-be-recognized drawing is extracted, such as room names, equipment names and size annotations. A pre-trained symbol recognition model can also be used, for example, a CNN model is used to recognize symbols in the drawing, such as pipe symbols.

[0146] Further, according to the overall layout and element information of the to-be-recognized drawing, it is judged whether the to-be-recognized drawing belongs to a preset drawing type. It should be noted that the preset drawing type is set according to actual needs. For example, the embodiment is to recognize the as-built drawing in the field of architecture, and a feature library is established for this type of drawing in advance, which includes the typical layout and element information of this type of drawing. These features can be text content, symbols, layout structure, etc. According to the overall layout and element information of the to-be-recognized drawing, it is judged whether the to-be-recognized drawing is an as-built drawing in the field of architecture. If it is an as-built drawing in the field of architecture, the model parameters of the multi-modal large model and the fusion strategy are called to execute step S12: performing feature extraction on the to-be-recognized drawing by using the multi-modal large model to obtain a multi-modal feature vector.

[0147] In other possible implementations, different drawing types can also be defined, such as architectural drawings, electrical drawings, water supply and drainage drawings, and heating and ventilation drawings. Each type of drawing has its specific layout and element features. A feature library is established for each type of drawing, which contains the typical layout and element information of this type of drawing. These features can be text content, symbols, layout structure, etc. The overall layout and element information of the to-be-recognized drawing are matched with the feature libraries of various drawing types. Optionally, a similarity algorithm (such as cosine similarity or Euclidean distance) is used to calculate the similarity, and then the drawing type to which the to-be-recognized drawing belongs is determined according to the similarity. According to the drawing type to which the to-be-recognized drawing belongs, the model parameters and the fusion strategy of the multi-modal large model corresponding to the drawing type are loaded. The model parameters and the fusion strategy are optimized according to the drawings of this type of drawing, which can better process drawings of a specific type. Subsequently, the to-be-recognized drawing can be recognized by using the multi-modal large model and the fusion strategy, effectively improving the accuracy of drawing recognition.

[0148] It should be noted that the examples in the drawings are only used to understand the present application and do not constitute a limitation on the drawing recognition method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0149] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0150] The present application also provides a drawing recognition device, which is described in detail below. Figure 8 , Figure 8 A module structure schematic diagram of the drawing recognition device of the embodiment of the present application is shown in the figure; the drawing recognition device comprises:

[0151] The acquisition module 91 is configured to acquire a to-be-recognized drawing.

[0152] The feature extraction module 92 is configured to perform feature extraction on the to-be-identified drawing by using the multi-modal large model, to obtain a multi-modal feature vector.

[0153] The feature fusion module 93 is configured to perform feature fusion processing on the multi-modal feature vector according to a fusion strategy, to obtain a target fusion feature vector.

[0154] The recognition module 94 is configured to perform recognition on the target fusion feature vector, to obtain a drawing recognition result.

[0155] The model parameters of the multi-modal large model and / or the fusion strategy are obtained by adaptive learning and optimization on different drawings.

[0156] The drawing recognition device provided in the present application adopts the drawing recognition method in the above embodiments, and can solve the technical problems in the background art. Compared with the prior art, the drawing recognition device provided in the present application has the same beneficial effects as the drawing recognition method provided in the above embodiments, and other technical features in the drawing recognition device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0157] The present application provides an electronic device, which comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the drawing recognition method in the above embodiment one.

[0158] Reference will be made to the following Figure 9 which shows a structural schematic diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 9 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0159] As Figure 9As shown, the electronic device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage device 1003 into a random access memory 1004. Various programs and data required for operation of the electronic device are also stored in the random access memory 1004. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or by wire with other devices to exchange data. Although the electronic device having various systems is shown in the figure, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0160] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0161] The electronic device provided by the present application adopts the drawing recognition method in the above-mentioned embodiments, and can solve the technical problems in the background art. Compared with the prior art, the electronic device provided by the present application has the same beneficial effects as the drawing recognition method provided by the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0162] It should be understood that parts of the present disclosure can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0163] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0164] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon for executing the drawing recognition method in the above embodiment.

[0165] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination thereof.

[0166] The above computer readable storage medium can be contained in an electronic device, or can exist separately without being assembled into an electronic device.

[0167] The above computer readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to:

[0168] Obtain a drawing to be recognized;

[0169] Extract features from the drawing to be recognized using a multi-modal large model to obtain a multi-modal feature vector;

[0170] According to a fusion strategy, the multi-modal feature vector is subjected to feature fusion processing to obtain a target fusion feature vector;

[0171] perform recognition on the target fusion feature vector to obtain a drawing recognition result;

[0172] The model parameters of the multi-modal large model and / or the fusion strategy are obtained through adaptive learning and optimization on different drawings.

[0173] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0174] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations for systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0175] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0176] The readable storage medium provided by the application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned drawing recognition method, and can solve the technical problems in the background art. Compared with the prior art, the computer readable storage medium provided by the application has the same beneficial effects as the drawing recognition method provided by the above-mentioned embodiments, which will not be repeated here.

[0177] The embodiment of the application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the drawing recognition method as described above.

[0178] The computer program product provided by the application can solve the technical problems in the background art. Compared with the prior art, the computer program product provided by the embodiment of the application has the same beneficial effects as the drawing recognition method provided by the above-mentioned embodiments, which will not be repeated here.

[0179] The above-mentioned is only part of the embodiments of the application, and does not limit the patent scope of the application, and any equivalent structural transformation, direct / indirect application in other related technical fields made by using the content of the application specification and drawings within the technical concept of the application are included in the patent protection scope of the application.

Claims

1. A method of identifying a drawing sheet, characterized by, The method comprises the following steps: acquire a to-be-recognized drawing; extract features of the to-be-recognized drawing by using a multi-modal large model to obtain a multi-modal feature vector; perform feature fusion processing on the multi-modal feature vector according to a fusion strategy to obtain a target fusion feature vector; recognize the target fusion feature vector to obtain a drawing recognition result; the model parameters of the multi-modal large model and / or the fusion strategy are obtained by self-adaptive learning and optimization of different drawings.

2. The graph paper identification method of claim 1, wherein, The multi-modal feature vector comprises a target image feature vector and a target text feature vector. The feature fusion processing on the multi-modal feature vector according to the fusion strategy comprises the following steps: normalize the target image feature vector and the target text feature vector, and map the normalized target image feature vector and the target text feature vector to the same semantic space; perform feature fusion processing on the target image feature vector and the target text feature vector in the same semantic space according to the fusion strategy to obtain the target fusion feature vector.

3. The graph paper identification method of claim 2, wherein The multi-modal large model comprises a first target large model and a second target large model, and the first target large model and the second target large model transmit feature vectors through a preset protocol; the first target large model is used to extract a target image feature vector of the to-be-recognized drawing; and the second target large model is used to extract a target text feature vector of the to-be-recognized drawing.

4. The graph paper identification method of claim 1, wherein The self-adaptive learning process of the model parameters of the multi-modal large model comprises the following steps: acquire drawing samples of different types, styles and qualities; extract features of each drawing sample by using the multi-model large model to obtain a to-be-trained image feature vector and a to-be-trained text feature vector of each drawing sample; iteratively optimize the model parameters of the multi-modal large model according to the sample labels, the to-be-trained image feature vectors and the to-be-trained text feature vectors of each drawing sample.

5. The graph paper identification method of claim 4, wherein The self-adaptive learning process of the fusion strategy comprises the following steps: perform feature fusion processing on the to-be-trained image feature vectors and the to-be-trained text feature vectors of each drawing sample by using different fusion strategies to obtain to-be-trained fusion feature vectors corresponding to different fusion strategies; recognize the to-be-trained fusion feature vectors of each fusion strategy to obtain target recognition results; optimize the fusion strategy according to the sample labels and the target recognition results corresponding to different fusion strategies of each drawing sample.

6. The graph paper identification method of claim 4, wherein, The feature extraction of each drawing sample by using the multi-model large model to obtain the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample comprises the following steps: recognize local feature information and global structure feature information of each drawing sample by using a pre-constructed feature enhancement module; combine the local feature information and the global structure feature information of each drawing sample, and extract the to-be-trained image feature vector and the to-be-trained text feature vector of each drawing sample by using the multi-model large model.

7. The graph paper identification method of claim 1, wherein, The self-adaptive learning process of the model parameters of the multi-modal large model and / or the fusion strategy further comprises the following steps: Obtaining feedback information of the user, wherein the feedback information is information of the user correcting and labeling the drawing recognition result; According to the feedback information and the drawing recognition result, determining the error rate and accuracy rate of the multi-modal large model recognition; According to the error rate and accuracy rate of the multi-modal large model recognition, dynamically adjusting the model parameters of the multi-modal large model and / or the fusion strategy.

8. The graph paper identification method of claim 1, wherein, The drawing recognition result is obtained by identifying the target fusion feature vector, comprising: Classifying the target fusion feature vector to obtain a classification result; Content recognition is performed on the target fusion feature vector to obtain a content recognition result; According to the classification result and the content recognition result, the drawing recognition result is generated.

9. The graph paper identification method of claim 1, wherein, Before the multi-modal large model is used to extract features from the drawing to be recognized to obtain a multi-modal feature vector, the method further comprises: Analyzing the drawing to be recognized to obtain overall layout and element information of the drawing to be recognized; According to the overall layout and element information of the drawing to be recognized, it is judged whether the drawing to be recognized belongs to a preset drawing type; If it belongs, the step of using the multi-modal large model to extract features from the drawing to be recognized to obtain a multi-modal feature vector is executed.

10. A drawing sheet identifying device characterized by comprising: The drawing recognition device comprises: An acquisition module for acquiring a drawing to be recognized; A feature extraction module for extracting features from the drawing to be recognized using a multi-modal large model to obtain a multi-modal feature vector; A feature fusion module for performing feature fusion processing on the multi-modal feature vector according to a fusion strategy to obtain a target fusion feature vector; An identification module for identifying the target fusion feature vector to obtain a drawing recognition result; The model parameters of the multi-modal large model and / or the fusion strategy are obtained by self-adaptive learning and optimization of different drawings.

11. An electronic device, comprising: The electronic device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the drawing recognition method according to any one of claims 1 to 9.

12. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the drawing recognition method according to any one of claims 1 to 9. The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the drawing recognition method according to any one of claims 1 to 9.

Images