Industrial flow chart generation method and device, equipment and storage medium

Abstract

The invention provides an industrial flow chart generation method and device, equipment and a storage medium, and the method comprises the steps: carrying out the recognition of a to-be-generated industrial flow chart image, obtaining a target detection frame of industrial equipment, and extracting the related information of the industrial equipment from the target detection frame of the industrial equipment; performing direction identification on the industrial equipment image with the direction in the target detection frame of the industrial equipment, and outputting direction information of the industrial equipment; performing textbox detection and text content recognition processing on the to-be-generated industrial flow chart image to obtain related information of an identification text; flow chart lines are extracted through a line detection algorithm, and related information of the flow chart lines is obtained; and organizing various information according to a data format required by flow chart drawing software, and automatically generating an industrial flow chart after outputting a file in a preset format. Therefore, accurate analysis of equipment, texts and lines in the flow chart can be realized, the workload of engineers is greatly reduced, and the drawing efficiency of the industrial flow chart is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an industrial flow chart generation method, device, equipment and storage medium. Background Art

[0002] The industrial production process is a complex system engineering, which involves many processes and steps. The industrial flow chart can clearly show the processes and steps involved in the production process, key control points and parameters, so as to help engineering and technical personnel better understand the production process, troubleshoot problems in time, and ensure product quality and production efficiency.

[0003] In the field of industrial automation, due to the life of the control system, existing old engineering projects need to be updated and renovated. For example, the version of the control system is updated or a new control system is directly replaced. These renovations require engineers to redraw the industrial flow chart, and manual recognition and drawing of flow charts are mechanical and repetitive, time-consuming, labor-intensive, and inefficient. At present, the common industrial flow chart recognition technology uses machine learning methods to identify identification text, industrial equipment, and connection relationships.

[0004] However, the above method still requires manual drawing of new flow charts, which is cumbersome and prone to errors. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide an industrial flow chart generation method, device, equipment and storage medium.

[0006] In a first aspect, an embodiment of the present application provides an industrial process flow diagram generation method, comprising:

[0007] The industrial flow chart image to be generated is recognized by the trained deep learning target detection model to obtain the target detection frame of the industrial equipment, and the relevant information of the industrial equipment is extracted from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and location information of the industrial equipment;

[0008] The industrial equipment image with direction in the target detection frame of the industrial equipment is intercepted, and the intercepted image is input into the trained deep learning direction recognition model to output the direction information of the industrial equipment;

[0009] Performing text box detection and text content recognition processing on the industrial flow chart image to be generated by a deep learning OCR (Optical Character Recognition) model to obtain relevant information of the identification text, wherein the relevant information of the identification text includes: the position, size, color, content of the identification text, and device-related text information;

[0010] After deleting the industrial equipment and identification text in the industrial flow chart image to be generated, the flow chart lines are extracted by a line detection algorithm to obtain relevant information of the flow chart lines;

[0011] Organize the relevant information of industrial equipment, the direction information of industrial equipment, the relevant information of identification text, and the relevant information of flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file;

[0012] The preset format file is imported into the flowchart drawing software to automatically generate an industrial flowchart.

[0013] Optionally, before identifying the industrial process flow chart image to be generated by using the trained deep learning target detection model, the method further includes:

[0014] Collect historical industrial process flow charts;

[0015] Annotate the collected historical industrial process diagrams to obtain historical image detection data;

[0016] Constructing a data set according to historical image detection data to obtain a first initial data set;

[0017] After preprocessing the first initial data set, the data set is divided into a first training set, a first test set, and a first validation set;

[0018] Build a deep learning object detection model;

[0019] The deep learning target detection model is trained by using the first training set, and the deep learning target detection model in the training process is tested and verified by using the first test set and the first verification set until the test and verification pass to obtain a trained deep learning target detection model.

[0020] Optionally, before inputting the captured image into a trained deep learning direction recognition model and outputting direction information of the industrial equipment, the method further includes:

[0021] Input the collected historical industrial process flow chart into the trained deep learning target detection model to obtain the target detection frame of the historical industrial equipment;

[0022] Intercepting industrial equipment with directions in the target detection frame of the historical industrial equipment to obtain direction data of the historical industrial equipment;

[0023] constructing a data set based on the direction data of historical industrial equipment to obtain a second initial data set;

[0024] After preprocessing the second initial data set, the data set is divided into a second training set, a second test set, and a second validation set;

[0025] Build a deep learning direction recognition model;

[0026] The deep learning direction recognition model is trained by the second training set, and the deep learning direction recognition model in the training process is tested and verified by the second test set and the second verification set until the test and verification pass to obtain a trained deep learning direction recognition model.

[0027] Optionally, the deep learning OCR model is used to perform text box detection and text content recognition processing on the industrial flow chart image to be generated to obtain relevant information of the identification text, including:

[0028] Input the industrial flow chart image to be generated into the deep learning OCR model for text box detection to obtain the position information and size information of the text box, where the position information is the center point coordinates of the detection box and the size information is the width and height of the detection box;

[0029] Capture the text image according to the position information and size information of the text box, and recognize the content of the text box;

[0030] Traversing the pixel points in the text image and counting the image histogram; wherein the color with the largest number of pixels is used as the background color, and the color with the second largest number of pixels is used as the text color;

[0031] According to the relative positions of the text and the industrial equipment in the industrial flow chart image to be generated, the text information inside the industrial equipment is extracted and used as the equipment-associated text information.

[0032] Optionally, extracting the flowchart lines by a line detection algorithm to obtain relevant information of the flowchart lines includes:

[0033] The region growing segmentation method is used to detect the lines in the industrial flow chart image to be generated, and obtain relevant information of the flow chart lines; wherein the relevant information of the flow chart lines includes: type, position, color, and thickness information of the flow chart lines.

[0034] Optionally, the adopting of a region growing segmentation method to detect lines in the industrial flow chart image to be generated includes:

[0035] Select a seed point as the starting point for region growing;

[0036] Traverse the pixels that have not grown around the seed point, and compare the pixel with the seed point. If the absolute value of the color difference between the pixel and the seed point's RGB three channels is less than or equal to the set threshold, add the pixel to the growth area and mark the pixel as grown;

[0037] The lines are selected by limiting the height and width of the growing area; the color with the most pixels in the growing area is used as the color of the line;

[0038] If the height of the growth area is greater than the width, it is judged as a vertical line; wherein the thickness of the vertical line is the width;

[0039] If the height of the growth area is not greater than the width, it is judged as a horizontal line; wherein the thickness of the horizontal line is the height.

[0040] Optionally, the information related to the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines are organized according to the data format required by the flowchart drawing software, and the preset format file is output, including:

[0041] According to the category, size, position, and direction information of the industrial equipment, the position, size, color, content, equipment-related text information of the identification text, and the type, position, color, and thickness information of the connecting lines, the data are organized in the data format required by the flowchart drawing software and output as a preset format file; wherein the preset format file includes: any one of: a csv file, an xml file, and a json file.

[0042] In a second aspect, an embodiment of the present application provides an industrial process flow chart generating device, comprising:

[0043] The industrial equipment target detection module is used to identify the industrial flow chart image to be generated through the trained deep learning target detection model, obtain the target detection frame of the industrial equipment, and extract the relevant information of the industrial equipment from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and location information of the industrial equipment;

[0044] The industrial equipment direction recognition module is used to intercept the industrial equipment image with direction in the target detection frame of the industrial equipment, input the intercepted image into the trained deep learning direction recognition model, and output the direction information of the industrial equipment;

[0045] A logo text recognition module is used to perform text box detection and text content recognition processing on the industrial flow chart image to be generated through a deep learning OCR model to obtain relevant information of the logo text, wherein the relevant information of the logo text includes: the location, size, color, content of the logo text, and device-related text information;

[0046] A connection line detection module is used to extract the flow chart lines through a line detection algorithm after deleting the industrial equipment and identification text in the industrial flow chart image to be generated, and obtain relevant information of the flow chart lines;

[0047] An information organization module is used to organize the relevant information of industrial equipment, the direction information of industrial equipment, the relevant information of identification text, and the relevant information of flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file;

[0048] The automatic flow chart generation module is used to import the preset format file into the flow chart drawing software to automatically generate an industrial flow chart.

[0049] In a third aspect, an embodiment of the present application provides an industrial flow chart generating device, including: a processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to:

[0050] Execute the steps of the industrial process flow chart generation method as described in any one of the first aspects.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the industrial flow chart generation method as described in any one of the first aspects.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] In the present application, the industrial flowchart image to be generated is identified by a trained deep learning target detection model to obtain a target detection frame of the industrial equipment, and the relevant information of the industrial equipment is extracted from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and position information of the industrial equipment; the industrial equipment image with direction in the target detection frame of the industrial equipment is intercepted, and the intercepted image is input into the trained deep learning direction recognition model to output the direction information of the industrial equipment; the industrial flowchart image to be generated is subjected to text box detection and text content recognition processing by a deep learning OCR model to obtain relevant information of the identification text, wherein the relevant information of the identification text includes: the position, size, color, content of the identification text, and equipment-related text information; after deleting the industrial equipment and identification text in the industrial flowchart image to be generated, the flowchart lines are extracted by a line detection algorithm to obtain relevant information of the flowchart lines; according to the data format required by the flowchart drawing software, the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines are organized, and a preset format file is output; the preset format file is imported into the flowchart drawing software to automatically generate an industrial flowchart. Thus, the functions of industrial equipment target detection, industrial equipment direction recognition, logo text recognition, connection line detection and flow chart generation are integrated, and the automatic recognition and generation of industrial flow charts are realized, solving the problem of cumbersome drawing work for subsequent engineering personnel. In addition, the embodiment of the present application uses a deep learning network to perform industrial equipment target detection, direction recognition and text recognition, and uses a line detection algorithm to detect connecting lines, so as to achieve accurate analysis of equipment, text and lines in the flow chart, greatly reducing the workload of engineering personnel and improving the efficiency of drawing industrial flow charts. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0055] Figure 1 A schematic diagram of the overall architecture of an industrial process flow chart generation method provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of a process flow of an industrial process flow chart generation method provided in an embodiment of the present application;

[0057] Figure 3 A schematic diagram of another industrial process flow chart generation method provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the structure of another industrial flow chart generating device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there can be a central component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there can be a central component at the same time.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0063] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0064] In order to facilitate understanding of the technical solutions in each implementation of this application, the following brief description of the professional terms appearing in the embodiments of this application is given:

[0065] 1) Industrial flow chart: refers to the abstraction and generalization of specific production processes in the field of industrial automation, which graphically represents the relationship and combination between various production processes, equipment and production lines. Usually, it includes various connections between process inputs, outputs, equipment and workbenches, as well as various elements such as data and materials. Its purpose is to present each link of the industrial production process in a graphical and visual way, so that production and technical personnel can quickly understand the relationship between various processes, equipment and working methods, and improve production efficiency, management quality, process monitoring and exception handling.

[0066] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0067] For example, Figure 1 The overall architecture diagram of an industrial flow chart generation method provided in an embodiment of the present application is as follows: Figure 1 As shown, it can include: an industrial flowchart recognition module and an industrial process generation module. Among them, the industrial flowchart recognition module is composed of four modules: industrial equipment target detection, industrial equipment direction recognition, identification text recognition and connection line detection. The industrial flowchart image to be generated is a screenshot of the industrial configuration software flowchart or a P&ID diagram.

[0068] For example, Figure 2 A schematic diagram of a process flow diagram generation method for an industrial process diagram provided in an embodiment of the present application, such as Figure 2 As shown, the method in this embodiment may include the following steps:

[0069] Step S201: The industrial flow chart image to be generated is recognized by using a trained deep learning target detection model to obtain a target detection frame of the industrial equipment, and relevant information of the industrial equipment is extracted from the target detection frame of the industrial equipment.

[0070] The relevant information of the industrial equipment includes: category, size and location information of the industrial equipment.

[0071] Exemplarily, in step S201, before recognizing the industrial process flow chart image to be generated by using the trained deep learning target detection model, the method further includes:

[0072] Step a1: Collect historical industrial process flow charts.

[0073] In this embodiment, when updating and renovating a historical engineering project, the industrial configuration software flow chart is captured using a computer screenshot tool; and the PDF file of the P&ID diagram is captured using a PDF conversion image tool.

[0074] It should be noted that this embodiment does not limit the method of collecting historical industrial process flow charts, and those skilled in the art can collect images according to actual implementation conditions.

[0075] Step a2: Annotate the collected historical industrial process flow charts to obtain historical image detection data.

[0076] In this embodiment, the industrial equipment on the historical image is labeled on the target detection and labeling tool. The labeled objects include industrial equipment such as pumps, valves, heat exchangers, indicator lights, buttons, inputs, outputs, reactors and compressors. Furthermore, the labeled images are saved in the VOC dataset format to establish a historical image detection dataset. In practical applications, as many historical industrial flow chart images as possible are prepared for subsequent target detection model training to improve the detection accuracy of the model.

[0077] It should be noted that this embodiment does not limit the target detection and labeling tools and data set formats, and the type of labeled industrial equipment can be determined according to actual project requirements.

[0078] Step a3: construct a data set based on historical image detection data to obtain a first initial data set.

[0079] Step a4: After preprocessing, the first initial data set is divided into a first training set, a first test set, and a first validation set.

[0080] In this embodiment, the first initial data set may also be enhanced, for example, by using Mosaic enhancement, random perturbation, and color perturbation, and further divided into a training set, a validation set, and a test set of the model.

[0081] Step a5: Build a deep learning target detection model.

[0082] Step a6: Train the deep learning target detection model through the first training set, and test and verify the deep learning target detection model during the training process through the first test set and the first verification set until the test and verification pass to obtain a trained deep learning target detection model.

[0083] In this embodiment, the deep learning target detection model constructed is trained by the historical image detection data set (i.e., the aforementioned first training set), and the model learning rate and the number of training rounds are adjusted. The first validation set and the first test set are input into the deep learning target detection model to verify whether the deep learning target detection model has been trained.

[0084] It should be noted that this embodiment does not limit the neural network structure and operation algorithm of the deep learning target detection model, which can be selected according to the actual implementation situation.

[0085] In this embodiment, the industrial flow chart image to be generated is input into the trained deep learning target model, the target detection frame of the industrial equipment is output, and the category, position, size and other information of the industrial equipment are extracted based on the target detection frame of the industrial equipment. Among them, the position information is the coordinates of the center point of the target detection frame, and the size information is the width and height of the target detection frame.

[0086] Step S202: Capture an industrial equipment image with a direction in a target detection frame of the industrial equipment, input the captured image into a trained deep learning direction recognition model, and output the direction information of the industrial equipment.

[0087] Exemplarily, before inputting the captured image into a trained deep learning direction recognition model and outputting the direction information of the industrial equipment, the method further includes:

[0088] Step b1: Input the collected historical industrial process flow chart into the trained deep learning target detection model to obtain the target detection frame of the historical industrial equipment.

[0089] In this embodiment, the historical industrial flow chart is input into the trained deep learning target detection model to detect industrial equipment. For industrial equipment with directions such as pumps, valves, inputs, outputs, and flow direction arrows, images are captured according to their target detection frames. In actual implementation, the type of industrial equipment with directions should be determined according to actual project requirements.

[0090] Step b2: intercept the industrial equipment with directions in the target detection frame of the historical industrial equipment to obtain the direction data of the historical industrial equipment.

[0091] In this embodiment, the captured industrial equipment image is annotated with directions, including four directions: up, down, left, and right. The image data set format is one image corresponding to one direction.

[0092] Step b3: construct a data set based on the historical direction data of industrial equipment to obtain a second initial data set.

[0093] Step b4: After preprocessing, the second initial data set is divided into a second training set, a second test set, and a second validation set.

[0094] In this embodiment, the second initial data set is subjected to data enhancement processing, such as color perturbation and contrast enhancement, etc. Since the data label is sensitive to direction, it is prohibited to use geometric transformation enhancement methods such as flipping, rotation and shifting. Further, the pre-processed data set is divided into a second training set, a second test set, and a second validation set.

[0095] Step b5: Build a deep learning direction recognition model.

[0096] Step b6: Train the deep learning direction recognition model through the second training set, and test and verify the deep learning direction recognition model during the training process through the second test set and the second verification set until the test and verification pass to obtain a trained deep learning direction recognition model.

[0097] In this embodiment, a classification model is selected according to the second training set, and the model learning rate and the number of training rounds are adjusted. The second training set, the second validation set and the second test set are input into the direction recognition model for training to obtain a model for subsequent industrial equipment direction recognition.

[0098] It should be noted that this embodiment does not limit the specific deep learning direction recognition model, which can be determined based on actual implementation conditions.

[0099] In this embodiment, the industrial equipment image with direction in the industrial equipment detection box in the industrial flow chart image to be generated is intercepted and input into the trained deep learning direction recognition model, thereby outputting the direction information of the industrial equipment.

[0100] Step S203: Perform text box detection and text content recognition processing on the industrial flow chart image to be generated through a deep learning OCR model to obtain relevant information of the identification text.

[0101] The relevant information of the identification text includes: the position, size, color, content, and associated text information of the identification text.

[0102] Exemplarily, the industrial flowchart image to be generated is input into the deep learning OCR model for text box detection to obtain the position information and size information of the text box, wherein the position information is the coordinates of the center point of the detection box, and the size information is the width and height of the detection box; the text image is captured according to the position information and size information of the text box, and the content of the text box is identified; the pixels in the text image are traversed, and the image histogram is counted; wherein the color with the largest number of pixels is used as the background color, and the color with the second largest number of pixels is used as the text color; according to the relative position of the text and the industrial equipment in the industrial flowchart image to be generated, the text information located inside the industrial equipment is extracted and used as the equipment-associated text information.

[0103] In this embodiment, text box detection and text content recognition are performed based on deep learning OCR technology. A deep learning OCR model is selected, and the industrial flow chart image to be generated is input into the model for text box detection, and the position and size information of the text box is obtained, wherein the position information is the coordinates of the center point of the detection box, and the size information is the width and height of the detection box. Further, the text content in the text box image is identified. The text image is intercepted according to the position and size information of the text detection box, and all pixels of the image are traversed to perform a statistical image histogram, wherein the color with the largest number of pixels is the background color, and the color with the second largest number of pixels is the text color. According to the relative position comparison between the text and the industrial equipment, the text located inside the industrial equipment and associated with it is obtained.

[0104] Step S204: after deleting the industrial equipment and identification text in the industrial flowchart image to be generated, extract the flowchart lines through a line detection algorithm to obtain relevant information of the flowchart lines.

[0105] In this embodiment, the industrial equipment detected in step S201 and the text recognized in step S202 in the industrial process diagram are deleted to reduce interference with line detection.

[0106] Exemplarily, a region growing segmentation method is used to detect lines in the industrial flowchart image to be generated to obtain relevant information of the flowchart lines; wherein the relevant information of the flowchart lines includes: type, position, color, and thickness information of the flowchart lines.

[0107] In this embodiment, a seed point is selected as the starting point of regional growth; the pixels that have not grown around the seed point are traversed, and the pixels are compared with the seed point. If the absolute values ​​of the color differences between the pixel and the RGB three channels of the seed point are all less than or equal to the set threshold, the pixel is added to the growth area, and the pixel is marked as grown; the lines are filtered out by limiting the height and width of the growth area; the color with the most pixels in the growth area is used as the color of the line; if the height of the growth area is greater than the width, it is judged as a vertical line; the thickness of the vertical line is the width; if the height of the growth area is not greater than the width, it is judged as a horizontal line; the thickness of the horizontal line is the height.

[0108] Step S205: Organize the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file.

[0109] In this embodiment, according to the category, size, and location information of the industrial equipment, the location, size, color, content, location, equipment-related text information of the identification text, and the type, location, color and thickness information of the connecting lines are organized in the data format required by the flowchart drawing software and output as a preset format file; wherein the preset format file includes: any one of: a csv file, an xml file, and a json file.

[0110] Step S206: Import the preset format file into the flowchart drawing software to automatically generate an industrial flowchart.

[0111] In this embodiment, the format file outputted in step S205 is imported into the flowchart drawing software to automatically generate and update the industrial configuration software flowchart file used by the control system.

[0112] It should be noted that this embodiment does not limit the preset format file and flowchart drawing software, which can be determined according to actual application requirements.

[0113] In this embodiment, the industrial flow chart image to be generated is identified by a trained deep learning target detection model to obtain a target detection frame of the industrial equipment, and the relevant information of the industrial equipment is extracted from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and position information of the industrial equipment; the industrial equipment image with direction in the target detection frame of the industrial equipment is intercepted, and the intercepted image is input into the trained deep learning direction recognition model to output the direction information of the industrial equipment; the industrial flow chart image to be generated is subjected to text box detection and text content recognition processing by a deep learning OCR model to obtain relevant information of the identification text, wherein the relevant information of the identification text includes: the position, size, color, content of the identification text, and equipment-related text information; after deleting the industrial equipment and identification text in the industrial flow chart image to be generated, the flowchart lines are extracted by a line detection algorithm to obtain relevant information of the flowchart lines; according to the data format required by the flowchart drawing software, the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines are organized, and a preset format file is output; the preset format file is imported into the flowchart drawing software to automatically generate an industrial flow chart. Thus, the functions of industrial equipment target detection, industrial equipment direction recognition, logo text recognition, connection line detection and flow chart generation are integrated, and the automatic recognition and generation of industrial flow charts are realized, solving the problem of cumbersome drawing work for subsequent engineering personnel. In addition, the embodiment of the present application uses a deep learning network to perform industrial equipment target detection, direction recognition and text recognition, and uses a line detection algorithm to detect connecting lines, so as to achieve accurate analysis of equipment, text and lines in the flow chart, greatly reducing the workload of engineering personnel and improving the efficiency of drawing industrial flow charts.

[0114] For example, Figure 3 A schematic diagram of another industrial process flow chart generation method provided in an embodiment of the present application, such as Figure 3As shown, the following steps may be included: inputting the industrial flow chart image to be generated into the deep learning target detection model for industrial equipment target detection to obtain the target detection frame of the industrial equipment; intercepting the industrial equipment image with direction in the target detection frame of the industrial equipment, and inputting the intercepted image into the deep learning direction recognition model to output the direction information of the industrial equipment. After completing the industrial equipment target detection and industrial equipment direction recognition, the industrial flow chart image to be generated is subjected to text box detection and text content recognition processing by the deep learning OCR model to obtain the relevant information of the identification text, such as: text color acquisition, equipment associated text acquisition. After deleting the industrial equipment and identification text in the industrial flow chart image to be generated, the flowchart lines are extracted by the line detection algorithm to obtain the relevant information of the flowchart lines. According to the data format required by the flowchart drawing software, the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines are organized, and a preset format file is output; the preset format file is imported into the flowchart drawing software to automatically generate a new industrial flow chart file.

[0115] In this embodiment, artificial intelligence (AI) technology is used to identify industrial equipment, identification text and connecting lines in industrial flow charts, and organized into the data format required by the flow chart drawing software, and further imported into the flow chart drawing software to automatically generate a new flow chart, thereby greatly reducing the workload of manual drawing and reducing the error rate. By combining the industrial equipment target detection method of deep learning detection technology, the industrial equipment direction recognition method of deep learning recognition technology, the identification text recognition method of deep learning OCR technology, and the connecting line detection method of the line detection algorithm, it is possible to prepare to identify and analyze complex industrial flow charts, accurately extract industrial flow chart elements, and accurately generate new flow charts, greatly improving work efficiency.

[0116] For example, Figure 4 A schematic diagram of the structure of another industrial flow chart generating device provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device in this embodiment may include: an industrial equipment target detection module 401, an industrial equipment direction recognition module 402, a logo text recognition module 403, a connection line detection module 404, an information organization module 405, and a flow chart automatic generation module 406, wherein:

[0117] The industrial equipment target detection module 401 is used to identify the industrial flow chart image to be generated through the trained deep learning target detection model, obtain the target detection frame of the industrial equipment, and extract the relevant information of the industrial equipment from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and location information of the industrial equipment;

[0118] The industrial equipment direction recognition module 402 is used to intercept the industrial equipment image with direction in the target detection frame of the industrial equipment, input the intercepted image into the trained deep learning direction recognition model, and output the direction information of the industrial equipment;

[0119] The identification text recognition module 403 is used to perform text box detection and text content recognition processing on the industrial flow chart image to be generated through a deep learning OCR model to obtain relevant information of the identification text, wherein the relevant information of the identification text includes: the position, size, color, content, and associated text information of the identification text;

[0120] A connection line detection module 404 is used to extract the flow chart lines by a line detection algorithm after deleting the industrial equipment and identification text in the industrial flow chart image to be generated, and obtain relevant information of the flow chart lines;

[0121] The information organization module 405 is used to organize the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file;

[0122] The flowchart automatic generation module 406 is used to import the preset format file into the flowchart drawing software to automatically generate an industrial flowchart.

[0123] Exemplarily, the above-mentioned device may also include: a first model building and training module 407, which is used to collect historical industrial process flow charts; annotate the collected historical industrial process flow charts to obtain historical image detection data; build a data set based on the historical image detection data to obtain a first initial data set; pre-process the first initial data set and divide it into a first training set, a first test set, and a first verification set; build a deep learning target detection model; train the deep learning target detection model through the first training set, and test and verify the deep learning target detection model during the training process through the first test set and the first verification set, until the test and verification are passed, and a trained deep learning target detection model is obtained.

[0124] Exemplarily, the above-mentioned device may also include: a second model construction and training module 408, which is used to input the collected historical industrial process flow chart into the trained deep learning target detection model to obtain the target detection frame of the historical industrial equipment; intercept the industrial equipment with direction in the target detection frame of the historical industrial equipment to obtain the direction data of the historical industrial equipment; construct a data set according to the direction data of the historical industrial equipment to obtain a second initial data set; pre-process the second initial data set and divide it into a second training set, a second test set, and a second verification set; construct a deep learning direction recognition model; train the deep learning direction recognition model through the second training set, and test and verify the deep learning direction recognition model during the training process through the second test set and the second verification set until the test and verification pass to obtain a trained deep learning direction recognition model.

[0125] Exemplarily, the identification text recognition module 403 is specifically used to: input the industrial flowchart image to be generated into the deep learning OCR model to perform text box detection, and obtain the position information and size information of the text box, wherein the position information is the coordinates of the center point of the detection box, and the size information is the width and height of the detection box; intercept the text image according to the position information and size information of the text box; traverse the pixel points in the text image, and count the image histogram; wherein the color with the largest number of pixels is used as the background color, and the color with the second largest number of pixels is used as the text color; according to the relative position of the text and the industrial equipment in the industrial flowchart image to be generated, extract the text information located inside the industrial equipment and use it as the equipment-associated text information.

[0126] Exemplarily, the connection line detection module 404 is specifically used to: use the region growing segmentation method to detect the lines in the industrial flowchart image to be generated, and obtain relevant information of the flowchart lines; wherein the relevant information of the flowchart lines includes: the type, position, color, and thickness information of the flowchart lines.

[0127] Exemplarily, the region growing segmentation method is used to detect lines in the industrial flow chart image to be generated, including: selecting a seed point as the starting point of region growth; traversing the pixels that have not grown around the seed point, and comparing the pixels with the seed point, if the absolute value of the color difference between the pixel and the seed point's RGB three channels is less than or equal to a set threshold, then adding the pixel to the growth region, and marking the pixel as grown; filtering out lines by limiting the height and width of the growth region; wherein the color with the most pixels in the growth region is used as the color of the line; if the height of the growth region is greater than the width, it is judged to be a vertical line; wherein the thickness of the vertical line is the width; if the height of the growth region is not greater than the width, it is judged to be a horizontal line; wherein the thickness of the horizontal line is the height.

[0128] Exemplarily, the information organization module 405 is specifically used to: organize the data in the data format required by the flowchart drawing software according to the category, size, position, and direction information of the industrial equipment, the position, size, color, content, position, equipment-related text information of the identification text, and the type, position, color, and thickness information of the connecting lines, and output them as a preset format file; wherein the preset format file includes: any one of: a csv file, an xml file, and a json file.

[0129] An embodiment of the present application provides an industrial flowchart generation device (system), including: a processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to: execute the steps of the industrial flowchart generation method in the above embodiment.

[0130] It should be noted that those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "platform" herein.

[0131] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods. Among them, the computer-readable medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also exist in a communication device as discrete components.

[0132] The present application also provides a program product, which includes a computer program. The computer program is stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server implements any method of the above-mentioned embodiments of the present invention.

[0133] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for generating an industrial flow chart, characterized in that: include: The industrial flow chart image to be generated is recognized by a trained deep learning target detection model to obtain a target detection frame of the industrial equipment, and relevant information of the industrial equipment is extracted from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: category, size and location information of the industrial equipment; The industrial equipment image with direction in the target detection frame of the industrial equipment is intercepted, and the intercepted image is input into the trained deep learning direction recognition model to output the direction information of the industrial equipment; Performing text box detection and text content recognition processing on the industrial flow chart image to be generated through a deep learning OCR model to obtain relevant information of the identification text, wherein the relevant information of the identification text includes: the position, size, color, content of the identification text, and device-related text information; After deleting the industrial equipment and identification text in the industrial flow chart image to be generated, the flow chart lines are extracted by a line detection algorithm to obtain relevant information of the flow chart lines; Organize the relevant information of industrial equipment, the direction information of industrial equipment, the relevant information of identification text, and the relevant information of flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file; The preset format file is imported into the flowchart drawing software to automatically generate an industrial flowchart.

2. The industrial process flow diagram generation method according to claim 1, characterized in that: Before identifying the to-be-generated industrial process flow chart image by using the trained deep learning target detection model, the method further includes: Collect historical industrial process flow charts; Annotate the collected historical industrial process diagrams to obtain historical image detection data; Constructing a data set according to historical image detection data to obtain a first initial data set; After preprocessing the first initial data set, the data set is divided into a first training set, a first test set, and a first validation set; Build a deep learning object detection model; The deep learning target detection model is trained by using the first training set, and the deep learning target detection model in the training process is tested and verified by using the first test set and the first verification set until the test and verification pass to obtain a trained deep learning target detection model.

3. The industrial process flow diagram generation method according to claim 2, characterized in that: Before inputting the captured image into the trained deep learning direction recognition model and outputting the direction information of the industrial equipment, the method further includes: Input the collected historical industrial process flow chart into the trained deep learning target detection model to obtain the target detection frame of the historical industrial equipment; Intercepting industrial equipment with directions in the target detection frame of the historical industrial equipment to obtain direction data of the historical industrial equipment; constructing a data set based on the direction data of historical industrial equipment to obtain a second initial data set; After preprocessing the second initial data set, the data set is divided into a second training set, a second test set, and a second validation set; Build a deep learning direction recognition model; The deep learning direction recognition model is trained by the second training set, and the deep learning direction recognition model in the training process is tested and verified by the second test set and the second verification set until the test and verification pass to obtain a trained deep learning direction recognition model.

4. The industrial process flow diagram generation method according to claim 1, characterized in that: The deep learning OCR model is used to perform text box detection and text content recognition processing on the industrial flow chart image to be generated, and relevant information of the identification text is obtained, including: Input the industrial flow chart image to be generated into the deep learning OCR model for text box detection to obtain the position information and size information of the text box, where the position information is the center point coordinates of the detection box and the size information is the width and height of the detection box; Capture the text image according to the position information and size information of the text box, and recognize the content of the text box; Traversing the pixel points in the text image and counting the image histogram; wherein the color with the largest number of pixels is used as the background color, and the color with the second largest number of pixels is used as the text color; According to the relative positions of the text and the industrial equipment in the industrial flow chart image to be generated, the text information inside the industrial equipment is extracted and used as the equipment-associated text information.

5. The industrial process flow diagram generation method according to claim 1, characterized in that: The flowchart lines are extracted by a line detection algorithm to obtain relevant information of the flowchart lines, including: The region growing segmentation method is used to detect the lines in the industrial flow chart image to be generated, and obtain relevant information of the flow chart lines; wherein the relevant information of the flow chart lines includes: type, position, color, and thickness information of the flow chart lines.

6. The industrial process flow diagram generation method according to claim 5, characterized in that: The method of using the region growing segmentation method to detect lines in the industrial flow chart image to be generated includes: Select a seed point as the starting point for region growing; Traverse the pixels that have not grown around the seed point, and compare the pixel with the seed point. If the absolute value of the color difference between the pixel and the seed point's RGB three channels is less than or equal to the set threshold, add the pixel to the growth area and mark the pixel as grown; The lines are selected by limiting the height and width of the growing area; the color with the most pixels in the growing area is used as the color of the line; If the height of the growth area is greater than the width, it is judged as a vertical line; wherein the thickness of the vertical line is the width; If the height of the growth area is not greater than the width, it is judged as a horizontal line; wherein the thickness of the horizontal line is the height.

7. The method for generating an industrial flow chart according to any one of claims 1 to 6, characterized in that: The method of organizing the relevant information of the industrial equipment, the direction information of the industrial equipment, the relevant information of the identification text, and the relevant information of the flowchart lines according to the data format required by the flowchart drawing software, and outputting a preset format file includes: According to the category, size, position, and direction information of the industrial equipment, the position, size, color, content, position, equipment-related text information of the identification text, and the type, position, color, and thickness information of the connecting lines, the data are organized in the data format required by the flowchart drawing software and output as a preset format file; wherein the preset format file includes: any one of: a csv file, an xml file, and a json file.

8. An industrial flow chart generating device, characterized in that: include: The industrial equipment target detection module is used to identify the industrial flow chart image to be generated through the trained deep learning target detection model, obtain the target detection frame of the industrial equipment, and extract the relevant information of the industrial equipment from the target detection frame of the industrial equipment; wherein the relevant information of the industrial equipment includes: the category, size and location information of the industrial equipment; The industrial equipment direction recognition module is used to intercept the industrial equipment image with direction in the target detection frame of the industrial equipment, input the intercepted image into the trained deep learning direction recognition model, and output the direction information of the industrial equipment; A logo text recognition module is used to perform text box detection and text content recognition processing on the industrial flow chart image to be generated through a deep learning OCR model to obtain relevant information of the logo text, wherein the relevant information of the logo text includes: the location, size, color, content of the logo text, and device-related text information; A connection line detection module is used to extract the flow chart lines through a line detection algorithm after deleting the industrial equipment and identification text in the industrial flow chart image to be generated, and obtain relevant information of the flow chart lines; An information organization module is used to organize the relevant information of industrial equipment, the direction information of industrial equipment, the relevant information of identification text, and the relevant information of flowchart lines according to the data format required by the flowchart drawing software, and output a preset format file; The automatic flow chart generation module is used to import the preset format file into the flow chart drawing software to automatically generate an industrial flow chart.

9. An industrial flow chart generating device, characterized in that: include: A processor and a memory, wherein the memory stores executable program instructions, and when the processor calls the program instructions in the memory, the processor is used to: Execute the steps of the industrial process diagram generation method described in any one of claims 1 to 7.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the industrial process flow chart generation method according to any one of claims 1 to 7 are implemented.

Images