Automatic schematic diagram identification method and system based on multi-modal large model
Automatically identifying the schematic diagram through multimodal large model solves the problems of low efficiency and poor accuracy of traditional manual analysis, and achieves fast, accurate and automatic analysis and in-depth understanding of the schematic diagram, reducing the dependence on professionals.
Patent Information
- Application Number
- CN202410887531.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-15
AI Technical Summary
Traditional schematic diagram analysis methods rely on manual analysis, are inefficient and inconsistent, making it difficult to meet the needs of fast and accurate analysis, and computer-aided design software has limited intelligent analysis capabilities for schematic diagrams.
Using the automatic schematic recognition method based on multimodal large model, the schematic image is preprocessed and sent to the multimodal large model with predefined prompt text for automatic identification, and obtain the recognition results to obtain the schematic description information. .
It realizes rapid, accurate and automatic analysis of schematic diagrams, reduces manual participation, improves the accuracy and consistency of the identification process, reduces dependence on professional and technical personnel, and provides in-depth understanding to assist in design and failure analysis.
Smart Images

Figure CN118968538A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and machine learning, and in particular to a schematic automatic recognition method and system based on a multimodal large model. Background Art
[0002] In the field of engineering technology, schematics have always played a vital role as an important tool for expressing and describing the working principles of systems, equipment or devices. Schematics clearly show the connection relationship and workflow between various components in a graphical way, providing technicians with an intuitive and easy-to-understand reference.
[0003] Traditional schematic analysis methods mainly rely on manual analysis by professional technicians. This process not only consumes a lot of time and human resources, has low recognition efficiency, but is also easily affected by differences in personal technical level and experience, resulting in a large number of inconsistencies in the analysis results. With the expansion of project scale and the increase in complexity, traditional manual analysis methods have been unable to meet the needs of fast and accurate schematic analysis.
[0004] Although computer-aided design (CAD) software has improved the efficiency of drawing production to a certain extent, there is no mature solution for the automated analysis and understanding of the generated schematics. CAD software mainly focuses on the drawing and editing functions of drawings, and has limited intelligent analysis and recognition capabilities for schematics. This leads to the need for manual analysis and judgment in the subsequent processing, analysis and application of schematics.
[0005] Since schematics usually contain complex components and various connection methods, traditional image recognition technology often does not work well when processing schematics. It is prone to misidentification or missed recognition, and it is difficult to accurately identify and understand the detailed information. Summary of the invention
[0006] In order to solve the above technical problems, the present invention provides a schematic diagram automatic recognition method and system based on a multimodal large model, which solves the problems of low efficiency, poor accuracy and high dependence on professionals in the prior art of schematic diagram recognition.
[0007] The technical problem solved by the present invention can be achieved by adopting the following technical solutions:
[0008] The first aspect of the present invention is to provide a method for automatically identifying a schematic diagram based on a multimodal large model, comprising:
[0009] Preprocessing the schematic image to obtain a preprocessed schematic image;
[0010] Predefine prompt text, and send the prompt text and the preprocessed schematic image to the multimodal large model; wherein the prompt text is at least used to prompt the composition in the schematic image;
[0011] The recognition result returned after the multimodal large model automatically recognizes the preprocessed schematic image according to the prompt text is received, and the recognition result is analyzed to obtain description information of the schematic image.
[0012] Preferably, the preprocessing of the schematic image comprises:
[0013] Image format standardization; and / or
[0014] Image resizing and color correction; and / or
[0015] Image denoising; and / or
[0016] The image is converted to a string in a first preset format.
[0017] Preferably, the prompt text is also used to prompt:
[0018] The role played by the multimodal macromodel; and / or
[0019] The tasks to be performed by the multimodal large model; and / or
[0020] The content to be output after the multimodal large model performs the task.
[0021] Preferably, the sending the prompt text and the preprocessed schematic image to the multimodal large model comprises:
[0022] Packing the prompt text and the preprocessed schematic image into a character string having a second predetermined format;
[0023] The character string having the second predetermined format is sent to the multimodal macro model.
[0024] Preferably, the sending the character string having the second predetermined format to the multimodal macro model comprises:
[0025] The character string having the second predetermined format is sent to the multimodal macro model through a network request.
[0026] Preferably, the network request is sent via HTTP protocol.
[0027] Preferably, the step of analyzing the recognition result to obtain the description information of the schematic image includes:
[0028] The recognition result is parsed to extract description information of the schematic image from the parsed recognition result; wherein the description information at least includes the components and connection methods in the schematic image.
[0029] Preferably, the multimodal large model is a large language model.
[0030] Preferably, the schematic image is a schematic diagram of a ground source heat pump system.
[0031] The second aspect of the present invention is to provide a schematic automatic recognition system based on a multimodal large model, which is used to implement the schematic automatic recognition method based on a multimodal large model as claimed in any one of claims 1 to 9, comprising:
[0032] An image preprocessing unit, used for preprocessing the schematic image to obtain a preprocessed schematic image;
[0033] A prompt definition unit, used to predefine prompt text;
[0034] a sending unit, connected to the image preprocessing unit, the prompt definition unit and the multimodal large model, respectively, and used to send the prompt text and the preprocessed schematic image to the multimodal large model; wherein the prompt text is at least used to prompt the composition in the schematic image;
[0035] A feedback receiving unit is connected to the multimodal large model, and is used to receive the recognition result returned by the multimodal large model after automatically recognizing the preprocessed schematic image according to the prompt text, and analyze the recognition result to obtain the description information of the schematic image.
[0036] The advantages or beneficial effects of the technical solution of the present invention are:
[0037] The present invention uses a multimodal large model to perform recognition based on predefined prompt texts, and can automatically, quickly and accurately recognize and parse schematic diagrams, reducing the need for human participation, greatly improving the speed of analysis, and ensuring accuracy and consistency in the recognition process; automated analysis and question-and-answering enable non-professionals to understand and analyze schematic diagrams, reducing dependence on professional technicians; at the same time, it can provide an in-depth understanding of schematic diagrams to assist in more complex designs and fault analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a process flow of a schematic diagram automatic recognition method based on a multi-modal large model in a preferred embodiment of the present invention;
[0039] Figure 2 This is a flow chart of S200 in a preferred embodiment of the present invention;
[0040] Figure 3 It is a structural block diagram of a schematic automatic recognition system based on a multi-modal large model in a preferred embodiment of the present invention;
[0041] Figure 4 It is a schematic diagram of the overall process of the automatic recognition method of the schematic diagram based on the multi-modal large model in a preferred embodiment of the present invention;
[0042] Figure 5 A schematic diagram of a prompt text in a preferred embodiment of the present invention;
[0043] Figure 6 This is a schematic diagram of the network structure of a multimodal large model in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0046] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0047] See also Figure 1 In a preferred embodiment of the present invention, based on the above problems existing in the prior art, a schematic automatic recognition method based on a multi-modal large model is provided, comprising:
[0048] S100, preprocessing the schematic image to obtain a preprocessed schematic image;
[0049] First, input the schematic image to be parsed into the system. The schematic image can be directly imported into the system. For systems that support the Application Program Interface (API), data can be imported through the API interface. The imported API file can be a file exported from other projects. The system also supports batch import function.
[0050] Furthermore, the schematic image can be displayed in the form of an image file. An image file refers to a digital image data file stored in a computer system for display or processing. Image files can be divided into two categories according to their data structure and encoding method: bitmap image files (raster images) and vector graphics files.
[0051] A bitmap image file consists of a grid of pixels, each pixel has its own color value, and can be files including but not limited to JPEG, PNG, BMP, TIFF, etc.
[0052] Vector graphics files record the mathematical description of graphics, which can be lines, curves, shapes, etc., and can include but are not limited to SVG, AI, EPS, PDF, DWG (CAD) and other files.
[0053] After receiving the external input schematic image, it is preprocessed. The input image is optimized through the preprocessing step to better adapt to the subsequent model analysis.
[0054] As a preferred embodiment, preprocessing the schematic image includes:
[0055] Image format standardization; and / or
[0056] Image resizing and color correction; and / or
[0057] Image denoising; and / or
[0058] The image is converted to a string in a first preset format.
[0059] Specifically, the first preset format may be a Base64 format.
[0060] Specifically, firstly, the input schematic image is format-standardized, resized, color-corrected and denoised, and the image is converted into a string format suitable for transmission.
[0061] The image format standardization is implemented by a programming language. The image format is standardized by a programming language to meet the input specification of the multimodal large model 4 (such as the BLIP-2 model). As an example and not a limitation, the programming language may include but is not limited to Python, C, C++ or JAVA.
[0062] Image resizing and color correction are implemented by using an image processing library. The image processing library is used to read and perform resizing and color correction. As an example and not limitation, the image processing library can use Python language OpenCV library functions, read the input image through the OpenCV library functions, and also perform resizing and color correction operations through the OpenCV library functions.
[0063] Image denoising is achieved by using a denoising algorithm to optimize image quality. An appropriate algorithm is selected as needed. As an example but not limitation, the denoising algorithm may include but is not limited to Gaussian blur or median filtering.
[0064] The image is converted into a string format through an encoding library. The image is converted into a string format suitable for transmission through the encoding library. As an example but not limitation, the encoding library can use Python's base64 encoding library to convert the image file into a base64 string.
[0065] S200, predefine prompt text, and send the prompt text and the preprocessed schematic image to the multimodal large model 4; wherein the prompt text is at least used to prompt the composition in the schematic image;
[0066] Specifically, by calling the interface of the multimodal large model 4, the image and text content are jointly understood and described.
[0067] In step S200, Figure 2 As shown, including:
[0068] S201, pre-define prompt text.
[0069] As a preferred implementation, the prompt text is used to prompt:
[0070] Composition in schematic image; and
[0071] The role played by the multimodal macromodel 4; and / or
[0072] The tasks to be performed by the multimodal large model 4; and / or
[0073] The content to be output after the multimodal large model 4 performs the task.
[0074] Specifically, first, prepare the prompt text, the structure of which should at least include the components in the schematic image, and in addition to the components in the image, it can also cover role definition, task scope, and the format of output information.
[0075] Preferably, the structure of the prompt text includes but is not limited to the following parts: a role definition statement, a task definition statement, a schematic component definition statement, and an output formatting statement.
[0076] Specifically, the role definition statement language is used to define the roles of the multimodal large model 4;
[0077] The task definition statement is used to define the scope and type of the task;
[0078] Schematic definition statements are used to define the various components of the schematic;
[0079] The output formatting statement is used to define the specific information returned by the multimodal large model 4, including content and format.
[0080] Furthermore, the above-mentioned pre-definition of roles, tasks, graph composition and output content can be determined according to the user's input. For example, an input interface is provided, which includes a role input box, a task input box, a graph composition input box and an output content input box, for the user to input relevant content, and then the system reads the content input by the user and converts it into a corresponding sentence.
[0081] Furthermore, the predefined prompt text input by the user can be polished. The polishing step can be implemented using a deep learning model, which is obtained by collecting a large number of polished text pairs (original text and corresponding polished text) and performing model training, verification and testing.
[0082] Furthermore, after the above sentences are predefined, the above defined sentences can be spliced together to obtain a prompt text.
[0083] As a preferred embodiment, wherein Figure 2 As shown, sending the prompt text and the preprocessed schematic image to the multimodal large model 4 includes:
[0084] S202, packaging the prompt text and the preprocessed schematic image into a character string having a second predetermined format;
[0085] S203 , sending the character string having the second predetermined format to the multimodal large model 4 .
[0086] As a preferred embodiment, sending the character string having the second predetermined format to the multimodal macro model 4 includes:
[0087] The character string having the second predetermined format is sent to the multimodal large model 4 through a network request.
[0088] Specifically, in this embodiment, the preprocessed schematic image and the prompt text are packaged into a character string in a second predetermined format, and then the API interface of the multimodal large model 4 is called and sent to the API interface of the multimodal large model 4 through a network request.
[0089] The multimodal large model 4 uses the prepared prompt text structure to jointly understand and describe the image and text content.
[0090] Specifically, the second predetermined format may be in Json format. During the packaging process, the image and the prompt text are concatenated into a Json string.
[0091] As a preferred implementation, the network request is sent via HTTP protocol. Specifically, HTTP request is used to ensure correct transmission of data.
[0092] S300, receiving the recognition result returned by the multimodal large model 4 after automatically recognizing the preprocessed schematic image according to the prompt text, and analyzing the recognition result to obtain description information of the schematic image.
[0093] Specifically, the text information returned by the BLIP-2 multimodal large model 4 is received and analyzed to extract a detailed description of the content of the input image.
[0094] As a preferred embodiment, the step of analyzing the recognition result to obtain description information of the schematic image includes:
[0095] The recognition result is parsed to extract description information of the schematic image from the parsed recognition result; wherein the description information at least includes the components and connection methods in the schematic image.
[0096] Specifically, the returned text information is parsed to extract detailed descriptions of schematic components and connections. The parsed results are the final results fed back to the user.
[0097] Furthermore, the parsed text information can also be used for design, maintenance, and education and training.
[0098] The schematic automatic recognition method described in the present invention aims to solve the problems of low efficiency, poor accuracy and high dependence on professionals in the prior art of schematic recognition. In this embodiment, the deep learning capability of the multimodal large model 4 is utilized, combined with image recognition and natural language processing technology, to realize automatic recognition and analysis of various components and connection methods in the schematic diagram of the ground source heat pump system, which greatly improves the speed and accuracy of the analysis while reducing the need for manual participation.
[0099] As a preferred implementation, the multimodal large model 4 is a large language model (Large Language Model, LLM).
[0100] As a preferred embodiment, the schematic image is a schematic diagram of a ground source heat pump system.
[0101] As a system for efficient and clean energy utilization, the schematic diagram of the ground source heat pump system is an important data for system design and fault analysis. By taking the schematic diagram of the ground source heat pump system as an input image, after preprocessing, the image and the prepared prompt text are input into the multi-modal large model 4, and the multi-modal large model 4 automatically recognizes the prompt text and feeds back the result. The present invention significantly improves the efficiency and quality of the design, maintenance and education and training of the ground source heat pump system.
[0102] The second aspect of the present invention is to provide a schematic automatic recognition system based on a multimodal large model, which is used to implement the schematic automatic recognition method based on a multimodal large model as claimed in any one of claims 1 to 9, comprising:
[0103] An image preprocessing unit 1 is used to preprocess the schematic image to obtain a preprocessed schematic image;
[0104] A prompt definition unit 2, used for predefining prompt text;
[0105] The sending unit 3 is connected to the image preprocessing unit 1, the prompt definition unit 2 and the multimodal large model 4, respectively, and is used to send the prompt text and the preprocessed schematic image to the multimodal large model 4; wherein the prompt text is at least used to prompt the composition in the schematic image;
[0106] The feedback receiving unit 5 is connected to the multimodal large model 4, and is used to receive the recognition result returned by the multimodal large model 4 after automatically recognizing the preprocessed schematic image according to the prompt text, and analyze the recognition result to obtain the description information of the schematic image.
[0107] Specifically, the system of the present invention greatly reduces the time required for manual analysis of schematic diagrams through automated recognition methods, significantly improving work efficiency; the multimodal large model 4 ensures accuracy and consistency in the recognition process based on deep learning technology, reducing human errors; the automated analysis and question-answering system enables non-professionals to understand and analyze schematic diagrams, reducing dependence on professional technicians; integrated intelligent tools can provide an in-depth understanding of schematic diagrams, assisting in more complex designs and fault analysis.
[0108] Examples are provided below to illustrate and describe the technical solution:
[0109] Example 1
[0110] Application scenario assumption: A company needs to convert the schematic diagram ('. / original.png') of its product into a detailed text description so that non-professionals can also understand the technical details.
[0111] The implementation process is as follows Figure 4 As shown, the following steps are included:
[0112] Step S1: Image preprocessing
[0113] Specifically, this step involves optimizing the schematic ('. / original.png') to better adapt to the subsequent model analysis. Specifically, the input image is first standardized by the Python language to ensure that it meets the input requirements of the BLIP-2 model. Specifically, the input image is read by the Python Open CV library function, and the size adjustment and color correction operations are also performed by the Open CV library function:
[0114] import cv2
[0115] #Read schematic image
[0116] image=cv2.imread('. / original.png')
[0117] # Resize to fit BLIP-2 model input
[0118] resized_image=cv2.resize(image,(desired_width,desired_height))
[0119] #Use color balance adjustment for correction
[0120] corrected_image=perform_color_correction(resized_image)
[0121] Specifically, denoising can be done by appropriate algorithms such as Gaussian blur or median filtering:
[0122] # Apply a denoising algorithm such as Gaussian blur or median filter
[0123] denoised_image=apply_denoising_algorithm(corrected_image)
[0124] Specifically, the image file is converted into a base64 string using Python's base64 library:
[0125] importbase64
[0126] #Convert the processed image into a base64 string for network transmission
[0127] encoded_image=base64.b64encode(denoised_image).decode('utf-8')
[0128] Step S2: Calling the BLIP-2 large model interface
[0129] Specifically, this step involves using Figure 6 The BLIP-2 multimodal large model shown in Figure 1 is used to understand and describe image and text content. Specifically, first prepare Figure 5 The prompt text structure shown in the figure. Specifically, the prompt text structure includes role definition statements, task definition statements, schematic definition statements, and output formatting statements. Specifically, the role definition statement defines the role of the large model, the task definition statement defines the scope and type of the task, the schematic definition statement defines the various components of the schematic, and the output formatting statement defines the content and format of the specific information returned by the large model:
[0130] #Role definition statement
[0131] prompt_role = "You are an HVAC schematic recognition expert,"
[0132] #Task definition statement
[0133] prompt_task="Your task is to identify the components in the schematic and output a detailed description of the schematic."
[0134] #Schematic definition statement
[0135] prompt_original="In the schematic diagram:\n1. A C in a circle represents a compressor\n2. A short horizontal line represents an evaporator\n3. A series of circles represent underground pipes\n4. Parallel lines represent heat exchangers\n5. Arrows represent flow direction\n"
[0136] # Output format statement
[0137] prompt_output="Please output in the following format:\n1. Output the main functions and detailed description of the schematic diagram\n2. Output the construction flow suggestions of the schematic diagram\n3. Output the potential safety hazards involved in the schematic diagram"
[0138] #Prompt text
[0139] prompt_text=prompt_role+prompt_task+prompt_original+prompt_output
[0140] Specifically, the preprocessed image and prompt text are combined into a Json string and sent to the API interface of the BLIP-2 large model:
[0141] #Combining the preprocessed image and prompt text into a JSON string
[0142] data=json.dumps({"image":encoded_image,"prompt":prompt_text})
[0143] Specifically, use HTTP requests to ensure correct data transmission:
[0144] #Sent to the API interface of the BLIP-2 large model
[0145] response=requests.post(api_url,data=data,headers=headers)
[0146] # Check the response
[0147] ifresponse.status_code==200:
[0148] model_output=response.json()
[0149] else:
[0150] print("ErrorcallingthemodelAPI")
[0151] Step S3: Process the text information returned by the large model
[0152] Specifically, this step involves receiving and analyzing text information returned by the BLIP-2 multimodal large model. Specifically, the returned text is parsed to extract a detailed description of the content of the input image.
[0153] #Parse the returned text
[0154] Descriptions=model_output.text.message[-1]
[0155] Through the analysis of the above examples, it can be proved that the proposed technical solution is feasible in processing complex schematics and converting them into detailed text descriptions. This method utilizes the multimodal learning ability of the BLIP-2 large model and can significantly improve the accuracy and detail of the text description.
[0156] Example 2
[0157] This embodiment compares the output time of schematic diagram description text using manual recognition, traditional image recognition and the BLIP-2 multimodal large model, and uses the text perplexity index to evaluate the quality of the output text. The experimental results are shown in Table 1 below.
[0158] Table 1: Experimental comparison data
[0159] method Text output time / second Text perplexity index Manual Identification 350 0.23 Traditional image recognition 172 0.35 BLIP-2 Multimodal Large Model 30 0.88
[0160] The higher the perplexity, the better the quality of the output text.
[0161] It can be seen from Table 1 that the text output time of the schematic diagram automatic recognition method based on the multimodal large model shown in the present invention is the shortest and the quality of the output text is the best.
[0162] In addition, the confusion index of the output text can be experimented with by using the prompt text lacking different parts, and the results are shown in Table 2.
[0163] Table 2 Ablation experimental data of different prompt text structures
[0164]
[0165] As can be seen from Table 2, the lack of role definition statements has the least impact on the quality of the output text, while the lack of schematic definition statements has the greatest impact on the quality of the output text.
[0166] Application Example 1: Simple Schematic Analysis
[0167] Schematic diagram composition: Contains 5 components, such as a compressor, an evaporator, a condenser, an expansion valve and a pump.
[0168] Step S1: Image preprocessing
[0169] Specifically, read the original JPEG image file.
[0170] Specifically, the image size is resized to 256x256 pixels using Open CV’s cv2.resize() function to meet the BLIP-2 model input size requirement.
[0171] Specifically, the cv2.cvtColor() function is applied to perform color correction and convert the image from the BGR color space to RGB.
[0172] Specifically, the cv2.GaussianBlur() function is used to perform a slight Gaussian blur denoising process.
[0173] Specifically, the processed image is encoded into a base64 string using a base64 library for API calls.
[0174] Step S2: Calling the BLIP-2 multimodal large model interface
[0175] Specifically, the entire prompt text is read.
[0176] Specifically, the preprocessed image and prompt text are combined into a JSON string.
[0177] Specifically, an HTTP POST request is sent to the API interface of the BLIP-2 multimodal large model to transmit JSON data.
[0178] Step S3: Process the text information returned by the multimodal large model
[0179] Specifically, the returned text information is obtained through the Python language.
[0180] Specifically, the perplexity of the returned description text is calculated, and the measured perplexity is 0.97, indicating that the generated text is of high quality.
[0181] Application Example 2: Moderate Complexity Schematic Analysis
[0182] Schematic composition: Contains 10 components, such as the above 5 components plus temperature sensor, flow meter, control valve, pressure gauge and an additional heat exchanger.
[0183] Step S1: Image preprocessing
[0184] Specifically, read the original PNG image file.
[0185] Specifically, the image size was resized to 512x512 pixels to obtain more details.
[0186] Specifically, color correction is performed to ensure color accuracy.
[0187] Specifically, cv2.medianBlur() function is applied to perform median filtering denoising.
[0188] Specifically, the encoded image is a base64 string.
[0189] Step S2: Calling the interface of the BLIP-2 multimodal large model
[0190] Specifically, the entire prompt text is read.
[0191] Specifically, the preprocessed image and the prompt text are combined into a JSON format.
[0192] Specifically, JSON data is sent to the API interface of the BLIP-2 multimodal large model.
[0193] Step S3: Process the text information returned by the multimodal large model
[0194] Specifically, the returned text information is obtained through the Python language.
[0195] Specifically, the perplexity of the returned description text is calculated, and the measured perplexity is 0.89, indicating that the generated text is of high quality.
[0196] Application Example 3: High Complexity Schematic Analysis
[0197] Schematic composition: Contains more than 15 electronic components, including sensors, resistors, capacitors, transistors, operational amplifiers, microcontrollers, etc.
[0198] Step S1: Image preprocessing
[0199] Specifically, high-resolution images are read and resized to 1024x1024 pixels to preserve all details.
[0200] Specifically, a thorough color correction is performed.
[0201] Specifically, an appropriate denoising algorithm is applied, such as bilateral filtering.
[0202] Specifically, the clear image is converted to base64 encoding.
[0203] Step S2: Calling the BLIP-2 multimodal large model interface
[0204] Specifically, the entire prompt text is read.
[0205] Specifically, the image and the prompt text are encapsulated as a JSON string.
[0206] Specifically, it is sent to the API interface of the BLIP-2 multimodal large model through the HTTP interface.
[0207] Step S3: Process the text information returned by the multimodal large model
[0208] Specifically, the returned text information is obtained through the Python language.
[0209] Specifically, the perplexity of the returned description text is calculated, and the measured perplexity is 0.81, which means that the quality of the generated text has declined, but it is still usable.
[0210] The advantages or beneficial effects of adopting the above technical solution are: the present invention uses a multimodal large model to perform recognition based on predefined prompt text, which can automatically, quickly and accurately recognize and parse schematic diagrams, reduce the need for human participation, greatly improve the speed of analysis, and ensure accuracy and consistency in the recognition process; automated analysis and question-and-answering enable non-professionals to understand and analyze schematic diagrams, reducing dependence on professional technicians; at the same time, it can provide an in-depth understanding of schematic diagrams to assist in more complex designs and fault analysis.
[0211] The above are only preferred embodiments of the present invention, and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of this specification and illustrations should be included in the protection scope of the present invention.
Claims
1. A schematic diagram automatic recognition method based on a multimodal large model, characterized in that: include: Preprocessing the schematic image to obtain a preprocessed schematic image; Predefine prompt text, and send the prompt text and the preprocessed schematic image to the multimodal large model; wherein the prompt text is at least used to prompt the composition in the schematic image; The recognition result returned after the multimodal large model automatically recognizes the preprocessed schematic image according to the prompt text is received, and the recognition result is analyzed to obtain description information of the schematic image.
2. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1 is characterized in that: The preprocessing of the schematic image comprises: Image format standardization; and / or Image resizing and color correction; and / or Image denoising; and / or The image is converted to a string in a first preset format.
3. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1, characterized in that: The prompt text is also used to prompt: The role played by the multimodal macromodel; and / or The tasks to be performed by the multimodal large model; and / or The content to be output after the multimodal large model performs the task.
4. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1 is characterized in that: The sending the prompt text and the preprocessed schematic image to the multimodal large model includes: Packing the prompt text and the preprocessed schematic image into a character string having a second predetermined format; The character string having the second predetermined format is sent to the multimodal macro model.
5. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 4 is characterized in that: The sending the character string having the second predetermined format to the multimodal large model comprises: The character string having the second predetermined format is sent to the multimodal macro model through a network request.
6. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 5 is characterized in that: The network request is sent via the HTTP protocol.
7. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1 is characterized in that: The step of analyzing the recognition result to obtain the description information of the schematic image includes: The recognition result is parsed to extract description information of the schematic image from the parsed recognition result; wherein the description information at least includes the components and connection methods in the schematic image.
8. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1, characterized in that: The multimodal large model is a large language model.
9. The method for automatic recognition of schematic diagrams based on a multimodal large model according to claim 1, characterized in that: The schematic image is a schematic diagram of a ground source heat pump system.
10. A schematic diagram automatic recognition system based on a multimodal large model, used to implement the schematic diagram automatic recognition method based on a multimodal large model as claimed in any one of claims 1 to 9, characterized in that: include: An image preprocessing unit, used for preprocessing the schematic image to obtain a preprocessed schematic image; A prompt definition unit, used to predefine prompt text; a sending unit, connected to the image preprocessing unit, the prompt definition unit and the multimodal large model, respectively, and used to send the prompt text and the preprocessed schematic image to the multimodal large model; wherein the prompt text is at least used to prompt the composition in the schematic image; A feedback receiving unit is connected to the multimodal large model, and is used to receive the recognition result returned by the multimodal large model after automatically recognizing the preprocessed schematic image according to the prompt text, and analyze the recognition result to obtain the description information of the schematic image.
Citation Information
Patent Citations
Image content analysis method and device, equipment and medium
CN116824278A
Data processing method and device, electronic equipment and storage medium
CN117079299A
Extracting and analyzing information from engineering drawings
US20210089767A1
Cited By
Method for converting picture into EDA symbol of component
CN121686462A
