Image recognition and intelligent conversion method, system and computer-readable medium in hand-drawn scenes

Abstract

This invention discloses a method, system, and computer-readable medium for image recognition and intelligent conversion in hand-drawn scenarios, comprising the following steps: Real-time capture and scanning of the current hand-drawn sketch using a camera, or direct flowchart drawing using software, to collect the current hand-drawn flowchart image information and input it into a computer, achieving real-time acquisition and transmission of the hand-drawn flowchart; acquiring the hand-drawn flowchart image information, and through steps of locating shape positions and identifying shape categories, ultimately outputting the position coordinate frames of each predicted shape, shape type, arrow key point positions, and the shape data to which the arrow belongs; intelligently displaying the image using PPT / Visio software, outputting an editable graphic for the specified software; and incorporating an OCR module to automatically generate text, ultimately outputting a complete editable graphic. This invention solves the problems of low accuracy, efficiency, and limited availability in existing hand-drawn flowchart conversion technologies.

Description

Technical Field

[0001] This invention belongs to the field of image recognition and human-computer interaction technology, and relates to the recognition of hand-drawn flowcharts and the generation of computer-editable standard formats. Specifically, it relates to image recognition and intelligent conversion methods, systems and computer-readable media in hand-drawn scenarios. Background Technology

[0002] The rapid development of computer and imaging technologies has provided convenient conditions for drawing graphics and human-computer interaction. However, in daily work and life, people still tend to use whiteboards, blackboards, paper, or portable electronic devices to draw flowcharts in real time, and then manually recognize and operate them to convert the hand-drawn sketches into editable graphics. This traditional manual conversion method for human-computer interaction is inefficient. The conversion requires users to be highly familiar with the drawing software itself, including its layout, the names and positions of various shapes, and the functions and adjustment methods of the software and its shapes. Users also need to manually drag and drop shapes with the mouse and memorize numerous keyboard shortcuts for quick operations, severely reducing drawing efficiency and preventing them from spending more time on the quality of the flowcharts themselves. This fails to meet the needs of human-computer interaction. Therefore, exploring natural, efficient, and intelligent human-computer interaction methods has become one of the important topics in the field of computer research.

[0003] In the field of human-computer interaction, end-to-end functionality is the most convenient and widely recognized channel for human-computer interaction. Users only need to input their initial requirements, and the end-to-end system can output the final desired paradigm in real time. For example, by directly inputting an original hand-drawn flowchart image, the system can output an editable graphic of the specified software. Users only need to make minor, precise modifications to the final output based on their additional needs. Therefore, how to implement an end-to-end system that directly generates editable graphics from original hand-drawn flowchart images to significantly improve the efficiency of human-computer interaction in software drawing is a pressing technical problem that needs to be solved.

[0004] However, current research on converting hand-drawn flowcharts into software-editable graphics is still largely lacking. Most methods only focus on the overall recognition of the hand-drawn flowchart, with some methods only locating and recognizing basic shapes while ignoring the connections between them. Other methods, while considering the location and recognition of basic shapes and their connections, require users to use specific connection methods. Therefore, using these methods necessitates understanding the flowchart shapes and connecting lines defined by the method itself, further limiting user drawing efficiency. Although some flowchart location and recognition methods with fewer restrictions exist, they suffer from small training sets and limited coverage. Most publicly available training sets contain only 500-800 flowcharts, and most flowcharts contain no more than 10 shapes, lacking coverage of common medium-to-large-sized flowcharts in daily work and life. Furthermore, they fail to effectively integrate with commonly used drawing software. For common drawing software, integration is hampered by a lack of lightweight and practical development tools like Visio, and existing development tools have high barriers to entry and are not directly compatible with the output of flowchart recognition methods, requiring significant time and effort for integration, such as PowerPoint. Therefore, current research on hand-drawn flowchart conversion technology has low accuracy, efficiency, and available options. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to provide an image recognition and intelligent conversion method, system, and computer-readable medium for hand-drawn scenarios. It effectively solves various problems in converting hand-drawn flowcharts into standard editable graphics in various scenarios, including departmental meetings, group discussions, and study notes. Furthermore, it achieves a balance between conversion speed and accuracy by providing different versions of network architecture. Especially in daily meeting recording scenarios, it makes human-computer interaction more convenient. By uploading original flowchart images, it achieves true end-to-end hand-drawn image conversion and integrates external OCR resource packages for handwritten font recognition. Users can make small-scale fine adjustments according to their specific needs. Through automatic conversion and minimal human intervention, it achieves fast, efficient, and accurate conversion of various hand-drawn flowcharts, solving the problems of low accuracy, efficiency, and limited availability in existing hand-drawn flowchart conversion technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for image recognition and intelligent conversion in hand-drawn scenes includes the following steps;

[0008] Step 1: Hand-drawn flowchart acquisition: Capture the current hand-drawn flowchart in real time by taking pictures with a camera or by drawing the flowchart directly with software. Collect the hand-drawn flowchart image information at the current moment and input it into the computer to realize the real-time acquisition and transmission of hand-drawn flowcharts.

[0009] Step 2: Obtain the hand-drawn process image information obtained in Step 1. Through the steps of locating the shape position and identifying the shape category, finally output the position coordinate box, shape type, arrow key point position and the shape data to which the arrow belongs for each predicted shape;

[0010] Step 3: Obtain the position coordinate frame, shape type, arrow key point position, and shape data of the predicted shape from Step 2, and intelligently display them through computer graphics software to output an editable graphic for the specified software;

[0011] Step 4: The OCR module is integrated with the OCR function package through a reserved external interface to recognize hand-drawn flowchart text. It also interfaces with the software to automatically generate text from the editable graphics obtained in Step 3, and finally outputs a complete editable graphic.

[0012] Step 2 involves locating the shape position and identifying the shape category, including shape coordinate detection, shape type identification, arrow feature point localization, arrow pointing object estimation, and setting a confidence threshold.

[0013] Shape coordinate detection: The original hand-drawn process image information is input into a deep cascaded neural network model for feature extraction, and then the shape candidate box is filtered to output the position coordinate box of each shape;

[0014] The deep cascaded neural network model includes an FPN network (mainly composed of CNN networks) for extracting global image features and an RPN network (mainly composed of CNN networks) for generating candidate boxes.

[0015] Shape type recognition: The position coordinate box of each shape is input into a parallel neural network model for feature extraction, then shape classification is performed, and the shape box of the final position is output by combining the shape position coordinate box;

[0016] The parallel neural network model comprises three parallel networks for shape coordinate frame regression, shape type prediction, and keypoint regression, all of which are composed of fully connected networks. The shape position coordinate frame is composed of a quadruple of (x1, y1, x2, y2) representing the coordinates of the upper left and lower right corners of the coordinate frame, while the final shape frame is composed of a triple of (box, class, score), where box represents the aforementioned shape position coordinate frame, class represents the type of the shape, and score represents the probability that the shape belongs to type class.

[0017] Arrow feature point localization: The arrow is transformed into a causal relationship connecting shape objects, and the corresponding causal relationship is represented by the start and end feature points. The feature points of the arrow shape box at the final position are detected by the constraint model, and the two-dimensional feature key points of the arrow in the arrow shape box at the final position are marked to realize the arrow feature point localization.

[0018] The connected shape objects include all shapes predicted by shape coordinate detection and shape type recognition; the causal relationship refers to the correspondence between the predicted arrow key points, with the key point closer to the arrow start (initial feature point) as the cause and the key point closer to the arrow end (final feature point) as the effect; the constraint model mainly includes bounding box constraints, that is, by comparing the key points with the corresponding arrow coordinate boxes, the key points are restricted to be located within the arrow coordinate boxes.

[0019] Arrow pointing object estimation: For each feature point in the output arrow shapebox and its surrounding shape, based on the shape position, key point position, and the geometric relationship between the two in the pixel coordinate system, the intelligent algorithm is used to estimate the belonging relationship of the arrow key points;

[0020] Among them, the attribution relationship of the arrow key points refers to the specific predicted shape corresponding to the key points; the intelligent algorithm estimation mainly includes shape determination and shortest distance: shape determination determines the vertex coordinates of the shape by the symmetry between the predicted rectangular coordinate frame and the predicted type shape, thereby determining the edges; the shortest distance determines the specific shape corresponding to the key point by calculating the shortest distance from the arrow key points to the edges of each shape.

[0021] Set a confidence threshold: Compare the coordinates and confidence of each predicted shape on the overall image, and combine the confidence of the object the arrow points to. Select the recognition results with reasonable shape position, clear object pointing to by the arrow, and confidence higher than the set threshold as the final shape position coordinate box, shape type, arrow key point position and the shape data to which the arrow belongs.

[0022] The intelligent display of computer graphics software in step 3 includes deep neural network and software interface implementation, causal conversion of arrow key points, and intelligent layout of the overall outline.

[0023] Deep Neural Network and Software Interface Implementation: Combining the specific document interface input requirements of computer graphics software, extract the raw output of the neural network and transform it into the input form specified by the software interface. Write interface functions to automatically generate editable graphics based on the predicted shape.

[0024] The original output of the deep neural network refers to the output of the aforementioned parallel neural network, including the shape coordinate frame, shape type, and arrow key points.

[0025] Arrow key point causal transformation: Estimate the objects pointed to by the arrows, and then infer the connection relationship between the objects. The starting point is the cause and the ending point is the effect. Write a function to realize the relationship connection between shape objects.

[0026] Intelligent layout of overall outline: For the generated original editable graphics, the absolute and relative positions of each shape are compared, and a heuristic clustering algorithm is used to achieve horizontal and vertical alignment; the shape size is standardized and consistent to complete the automatic intelligent layout, while providing the software's built-in intelligent layout algorithm for selection;

[0027] The absolute position of a shape is its coordinate system; the relative position of a shape is its orientation within the whole (e.g., the top left corner).

[0028] The computer drawing software is PowerPoint or Visio.

[0029] The hand-drawn process image information in step 1 includes the original RGB image captured by the camera, the pre-processed image after binarization by scanning software, or the three-dimensional matrix of the image directly drawn on the electronic device and read in by a programming language (Python's OpenCV toolkit).

[0030] The shape coordinate detection in step 2 specifically involves:

[0031] The collected hand-drawn flowchart image is processed by a shape / background binary classifier to obtain the coordinate boxes of each possible shape. The coordinate boxes of each possible shape are then input into the first sub-network structure of a deep cascaded neural network model for feature extraction, outputting numerous candidate boxes. The bounding box regression method is used to merge overlapping candidate boxes to obtain corrected candidate boxes. The corrected candidate boxes are divided into two parts: the first part is used to determine whether there is a shape in the image within the current corrected candidate box, and the second part represents the offset of the shape position within the current corrected candidate box relative to the shape box position of the real target.

[0032] The corrected candidate boxes obtained through the first sub-network are used as input, and the coarsely predicted candidate boxes are output through the second sub-network structure in the deep cascaded neural network model. The overlapping coarsely predicted candidate boxes are merged using the bounding box regression method to obtain the second corrected candidate boxes.

[0033] The candidate box obtained after the second correction through the second sub-network structure is used as input, and the final shape and position coordinate box is output through the third sub-network structure in the deep cascaded neural network model.

[0034] The shape type identification in step 2 specifically involves:

[0035] The shape location coordinate box obtained through the third sub-network structure is input into the deep neural network model for feature extraction. The shape multi-classifier outputs the probability of each shape belonging to the location box. The probabilities of each shape are then normalized through a softmax layer, as shown in the following formula:

[0036]

[0037] In the formula: z i z j p is the raw output of the classifier. i Here, N represents the normalized probability, and N is the total number of shape categories.

[0038] The cross-entropy loss function of the shape / background binary classifier The expression is as follows:

[0039]

[0040]

[0041] In the formula: The actual label for the shape / background, p i The probability of the shape;

[0042] The bounding box regression method uses non-maximum suppression to merge overlapping candidate boxes, resulting in corrected candidate boxes. The bounding box regression method calculates the background coordinates of the candidate boxes. Compared to the actual background coordinates Calculate the regression loss function based on the Euclidean distance between them. The expression is as follows:

[0043]

[0044]

[0045] In the formula: The L2 loss function represents the Euclidean distance. This represents the position coordinates of the i-th actual shapebox. R represents the position coordinates of the i-th predicted shapebox. 4 Represents the four-dimensional real number space;

[0046] The cross-entropy loss function of the shape multi-classifier The expression is as follows:

[0047]

[0048]

[0049] In the formula: This refers to the actual shape label below the shape frame. This represents the predicted probability.

[0050] The estimation of the object pointed to by the arrow in step 2 is specifically as follows:

[0051] Based on the shape type of the shape box in the final position output by shape coordinate detection, it is identified as an arrow / line / double arrow position coordinate box (collectively referred to as an arrow box). Feature extraction and regression are performed independently on the key points of each arrow box:

[0052] These are the two actual two-dimensional key points within the arrow frame: the start point and the end point. To predict two corresponding predicted 2D keypoints within the arrow box, the arrow keypoint linear regression method calculates the regression loss function by measuring the Euclidean distance between the actual 2D keypoints in the actual arrow box and their corresponding predicted 2D keypoints in the predicted arrow box.

[0053]

[0054]

[0055] In the formula: For the i-th actual 2D key point in the actual arrow box, This refers to the predicted two-dimensional key point in the corresponding i-th prediction arrow box.

[0056] The process involves determining the final position of the key points within the arrow frame detected by the arrow key point detection, and comparing them with the surrounding shape. Based on the geometric relationship between the shape position, the key point position, and the two in the pixel coordinate system, an intelligent algorithm is used to estimate the attribution relationship of the arrow key points. Specifically:

[0057] The distance measurement between keypoints and neighboring shape objects is abstracted as the Euclidean distance from a point to a line. For a given polygon and the arrow keypoint, the shortest distance is selected as the distance from the keypoint to the polygon using the formula for the distance from a point to a line. The expression is as follows:

[0058]

[0059]

[0060] In the formula: (x0, y0) are the two-dimensional coordinates of the key point of the arrow. Let the coordinates be the coordinates of the two endpoints of the i-th side of the polygon;

[0061] For all predicted shape objects and arrow keypoints, the distance is calculated, and the polygon with the smallest distance is selected as the object to which the arrow keypoint belongs. The expression is as follows:

[0062]

[0063] In the formula: d i The distance from the arrow keypoint to the i-th shape among all predicted shape objects.

[0064] In step 3, the drawing software PowerPoint / Visio, combined with the software's specific document interface input requirements, extracts and transforms the raw output of the neural network, writes an interface function to automatically generate predicted shapes, and simultaneously estimates objects by pointing to them with arrows, thereby inferring the connection relationships between objects. The starting point is the cause, and the ending point is the effect. The function that implements the connection between shape objects is as follows:

[0065] In the integration with PowerPoint software, the existing PowerPoint development tool, the pptx extension package of the Python language, is used to directly manipulate the PowerPoint software and perform a series of operations such as adding shapes, connecting objects, and setting text.

[0066] In the process of interfacing with Visio software, the win32com extension package of the Python language is used to launch Visio software through the Windows operating system and perform a series of operations such as adding shapes, connecting objects, and setting text.

[0067] To meet the interface requirements of PPTX and Win32COM, the shape and position in the original output of the neural network are transformed into a center point + range pattern. An example of a rectangle is shown below:

[0068] (x0, y0, y1, y2) → (x c y c (H, W)

[0069]

[0070] In the formula: (x0, y0, y1, y2) are the original coordinate outputs of the neural network, (x c ,y c (H,W) is the center point of the rectangle, and (H,W) is the length and width of the rectangle.

[0071] The arrow keypoints in the original output of the neural network are converted into connection attribute selections of the shape objects corresponding to the arrow objects based on the estimated shape objects. An example of a rectangle is shown below:

[0072] (x hegin y begin ) / (x end y end )→(center, down, right, up, left)

[0073] In the formula: (x begin ,y begin ) / (x end ,y end () represents the starting / ending point of the arrow key point, and (center, down, right, up, left) are optional interface points of the rectangle object;

[0074] Meanwhile, in Visio operations, the above interface calls are simplified by encapsulating them into direct function calls.

[0075] The intelligent layout of the overall outline in step 3 specifically involves:

[0076] First clustering: Using the length and width of editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to treat graphics whose length and width feature distances are less than the thresholds as having the same size, obtaining the number of clusters K and the cluster centers m. i Next, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the k-means clustering algorithm, we obtain a set of K editable graphics after clustering. The average of the features is taken as the cluster center. The size of all editable graphics in the same cluster is set to the size of the cluster center, as shown in the following expression:

[0077]

[0078]

[0079] Where: m i Let C be the cluster center of the i-th class of data. i Let i be the data set of the i-th class;

[0080] Secondary clustering: Using the top-left x and y coordinates of the editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to consider coordinates with feature distances less than the threshold as the same horizontal / vertical line. The number of clusters K and the cluster centers m are obtained. i Next, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the kmean clustering algorithm, we obtain a set of K editable graphics after clustering. We set the alignment value of all editable graphics in the same cluster to the alignment value of the cluster center to achieve automatic alignment. The expression is the same as the above formula.

[0081] It also provides Visio's built-in intelligent layout algorithm, which, based on the input requirements of the intelligent layout interface, further transforms the selection of connection attributes for shape objects in the causal transformation of arrow key points into automatic connection attribute selection. An example of a rectangle is shown below:

[0082]

[0083] In the formula: (center, down, right, up, left) are optional interface points of the rectangle object. Optional interface points for automatic connection attributes.

[0084] The OCR module in step 4 is specifically configured as follows:

[0085] We selected the Python version of PaddleOCR, the natural language processing package of PaddlePaddle under Baidu PaddlePaddle, to extract the position coordinates of the shape type identified as text from the final position of the shape coordinate detection output. We then used PaddleOCR for recognition and simultaneously performed overlap detection with the predicted bounding boxes of other shapes. The expression is as follows:

[0086]

[0087] In the formula: S1 is the prediction box of the given shape, and S2 is the prediction box of the text;

[0088] For J IoU Text prediction boxes exceeding a given threshold are identified as embedded text within the corresponding shape prediction box; for J IoU Text prediction boxes smaller than a given threshold are identified as free text. Simultaneously, the software interfaces with PowerPoint / Visio to embed the built-in text of the shape into the corresponding shape, and generates a text box at the free text prediction location and embeds the corresponding text.

[0089] Another objective of this application is to provide an image recognition and intelligent conversion system for hand-drawn scenes, comprising:

[0090] Memory is used to store instructions that can be executed by the processor;

[0091] A processor for executing the instructions to implement the method described above.

[0092] Another objective of this application is to provide a computer-readable medium storing computer program code that, when executed by a processor, implements the method described above.

[0093] The beneficial effects of this invention are:

[0094] This invention proposes an image recognition and intelligent conversion method, system, and computer-readable medium for hand-drawn scenes. It comprehensively considers the position and category of shapes, and the interrelationships formed by specified arrows, which can realize the causal conversion of key arrow points, determine the object pointed to by the arrows, and develop an ultra-lightweight Python version of Visio software development tool, providing a solution for converting current end-to-end hand-drawn flowcharts into editable graphics for specific software.

[0095] The functions of each module in the image recognition and intelligent conversion system for hand-drawn scenes of this invention are independent. The processing module is embedded in the computer only in the form of a software package, without the need for additional underlying software or program support. The system can be quickly adapted to the computer without the need for special equipment support. The execution software has strong functionality, short and concise code, and is convenient for various types of security audits. Attached Figure Description

[0096] Figure 1 This is a flowchart illustrating the workflow of the image recognition and intelligent conversion method for hand-drawn scenes according to the present invention.

[0097] Figure 2 This is an example of a hand-drawn flowchart used in this invention.

[0098] Figure 3 This is an example diagram illustrating the shape coordinate detection and shape type recognition effects in this invention.

[0099] Figure 4 This is an example diagram illustrating the positioning effect of arrow feature points in this invention.

[0100] Figure 5 This is a schematic diagram of arrow object estimation in this invention.

[0101] Figure 6 This is a schematic diagram of the cause-effect transformation of the arrow key points in this invention.

[0102] Figure 7 This is an example diagram illustrating the intelligent layout effect of the overall outline in this invention.

[0103] Figure 8 This is an example image showing the text recognition effect of the OCR module in this invention.

[0104] Figure 9 This is a system architecture diagram of the image recognition and intelligent conversion system in the hand-drawn scene of the present invention.

[0105] Figure 10 This is an example diagram illustrating the effect of a preferred embodiment of the present invention. Detailed Implementation

[0106] The present invention will now be described in further detail with reference to the accompanying drawings.

[0107] like Figure 1As shown, the image recognition and intelligent conversion method for hand-drawn scenes of the present invention includes the following steps:

[0108] Hand-drawn sketches collected:

[0109] The system captures real-time images of the current hand-drawn sketch using a camera or by directly drawing the flowchart using software. This image information includes the original RGB image captured by the camera, a pre-processed image after binarization by scanning software, or an image directly generated on an electronic device. Figure 2 As shown;

[0110] The process of locating shape positions and identifying shape categories mainly includes shape coordinate detection, shape type recognition, arrow feature point localization, arrow pointing object estimation, and setting confidence thresholds.

[0111] Shape coordinate detection:

[0112] The collected original hand-drawn process image information is input into a deep cascaded neural network model for feature extraction, and then the shape candidate box is filtered to output the position coordinate box of each shape.

[0113] The collected hand-drawn flowchart image is processed by a shape / background binary classifier to obtain the coordinate boxes of each possible shape. The coordinate boxes of each possible shape are then input into the first sub-network structure of the deep cascaded neural network model for feature extraction, resulting in numerous candidate boxes. The bounding box regression method is used to merge overlapping candidate boxes to obtain corrected candidate boxes. The corrected candidate boxes are divided into two parts: the first part is used to determine whether there is a shape in the image within the current corrected candidate box (the first part output vector size is 1×1×2 in this embodiment); the second part represents the offset of the shape position within the current corrected candidate box relative to the shape box position of the real target (the second part output vector size is 1×1×4 in this embodiment).

[0114] The shape / background binary classifier segments the bounding boxes that may contain shapes in the collected hand-drawn flowchart images, such as... Figure 3 As shown;

[0115] Among them, the cross-entropy loss function of the shape / background binary classifier The expression is as follows:

[0116]

[0117]

[0118] In the formula: The actual label for the shape / background, p i The probability of the shape;

[0119] The bounding box regression method uses non-maximum suppression to merge overlapping candidate boxes, resulting in corrected candidate boxes. The bounding box regression method calculates the background coordinates of the candidate boxes. Compared to the actual background coordinates Calculate the regression loss function based on the Euclidean distance between them. The expression is as follows:

[0120]

[0121]

[0122] Where: The L2 loss function represents the Euclidean distance. This represents the position coordinates of the i-th actual shapebox. R represents the position coordinates of the i-th predicted shapebox. 4 It represents the four-dimensional real space.

[0123] Where y is a quadruple, including the x-coordinate of the top-left corner of the candidate box, the y-coordinate of the top-left corner of the candidate box, the x-coordinate of the bottom-right corner of the candidate box, and the y-coordinate of the bottom-right corner of the candidate box. Table 1 shows some of the predicted shape position coordinates:

[0124] The first sub-network is a multi-layer convolutional network, and its model architecture is shown in Table 1:

[0125] Table 1 Model structure of the first sub-network

[0126] Name Kernel size Stride Padding Conv1 1x1x256 1x1 1x1 Conv2 3x3x256 1x1 1x1 Conv3 1x1x512 1x1 1x1 Conv4 3x3x256 1x1 1x1 Conv5 1x1x1024 1x1 \ Conv6 3x3x256 1x1 1x1 Conv7 1x1x2048 1x1 \ Conv8 3x3x256 1x1 1x1

[0127] Shape type recognition:

[0128] The corrected candidate boxes obtained through the first sub-network are used as input, and the coarsely predicted candidate boxes are output through the second sub-network structure in the deep cascaded neural network model. The overlapping coarsely predicted candidate boxes are merged using the bounding box regression method to obtain the second corrected candidate boxes.

[0129] The candidate box obtained after the second correction through the second sub-network structure is used as input, and the final shape and position coordinate box is output through the third sub-network structure in the deep cascaded neural network model.

[0130] The shape location bounding boxes are input into a deep neural network model for feature extraction, followed by shape type recognition. The output is the shape type within the location bounding box of each shape, such as... Figure 3 As shown;

[0131] The shape location bounding box obtained through the third sub-network structure is input into the deep neural network model for feature extraction. The shape multi-classifier outputs the probability of each shape belonging to the bounding box. The probabilities of each shape are then normalized through a softmax layer.

[0132] The softmax layer expression is as follows:

[0133]

[0134] In the formula: z i z j p is the raw output of the classifier. i Here, N represents the normalized probability, and N is the total number of shape categories.

[0135] Where z is a unary tuple, which is the network's original output for the candidate box;

[0136] Among them, the cross-entropy loss function of the shape multi-classifier The expression is as follows:

[0137]

[0138]

[0139] Where: This refers to the actual shape label below the shape frame. The probability is the predicted value, and N is the total number of shape categories.

[0140] Where y is an N-tuple, including the probability that the predicted shape in the candidate box belongs to each shape. Table 2 shows the prediction types and probabilities of some shape boxes:

[0141] The second and third sub-networks are multi-layer convolutional + fully connected networks:

[0142] The second sub-network is the ResNet50 network.

[0143] The architecture of the third sub-network model is shown in Table 2:

[0144] Table 2 Model structure of the third sub-network

[0145] Name Kernel size Stride Padding Conv1 3x3x256 1x1 1x1 ReLU \ \ \ Conv2 1x1x3 1x1 \ Conv3 1x1x12 1x1 \ Cls_score 1024x13 \ \ Bbox_pred 1024x48 \ \

[0146] Arrow feature point localization:

[0147] The shape type of the final position bounding box output by shape coordinate detection is identified as an arrow / line / double arrow position coordinate box (collectively referred to as an arrow box). Feature extraction and regression are performed independently on the key points of each arrow box, specifically as follows:

[0148] These are the two actual two-dimensional key points within the arrow frame: the start point and the end point. To predict two corresponding predicted 2D keypoints within the arrow box, the arrow keypoint linear regression method calculates the Euclidean distance between the actual 2D keypoints in the real arrow box and their corresponding predicted 2D keypoints in the predicted arrow box. Figure 4 As shown. Calculate the regression loss function.

[0149]

[0150]

[0151] In the formula: For the i-th actual 2D key point in the actual arrow box, This refers to the predicted two-dimensional key point in the corresponding i-th prediction arrow box.

[0152] The arrow keypoint detection network is a multi-layer convolutional network, and its model architecture is shown in Table 3.

[0153] Table 3. Model structure of the arrow keypoint detection network

[0154] Name Kernel size Stride Padding Conv1 3x3x512 1x1 1x1 Conv2 3x3x512 1x1 1x1 Conv3 3x3x512 1x1 1x1 Conv4 3x3x512 1x1 1x1 Conv5 3x3x512 1x1 1x1 Conv6 3x3x512 1x1 1x1 Conv7 3x3x512 1x1 1x1 Conv8 3x3x512 1x1 1x1 ReLU \ \ \ ConvTranspose 4x4x2 2x2 1x1

[0155] The arrow points to object estimation:

[0156] Based on the final positions of the key points in the arrow frame detected by the arrow keypoint detection, and their surrounding shapes, an intelligent algorithm is used to estimate the attribution relationship of the arrow key points according to the geometric relationship between the shape position, keypoint position, and the two in the pixel coordinate system. Figure 5 As shown;

[0157] The distance measurement between keypoints and neighboring shape objects is abstracted as the Euclidean distance from a point to a line. For a given polygon and the arrow keypoint, the shortest distance is selected as the distance from the keypoint to the polygon using the formula for the distance from a point to a line, as shown in the following formula:

[0158]

[0159]

[0160] In the formula: (x0, y0) are the two-dimensional coordinates of the key point of the arrow. Let the coordinates be the coordinates of the two endpoints of the i-th side of the polygon;

[0161] For all predicted shape objects and arrow keypoints, the distance is calculated, and the polygon with the smallest distance is selected as the object to which the arrow keypoint belongs. Figure 6 As shown, the calculation formula is:

[0162]

[0163] In the formula: d i The distance from the arrow keypoint to the i-th shape among all predicted shape objects.

[0164] Set the confidence threshold:

[0165] The above-mentioned shape coordinate detection, shape type recognition, arrow feature point localization, and arrow pointing object estimation are performed on the input hand-drawn flowchart image. The probability of each predicted shape under each predicted shape box is output. The calculation results with reasonable predicted shape box position and high predicted shape probability are selected as the final shape box position coordinates and shape box type output results.

[0166] Intelligent presentation using PPT / Visio software mainly includes deep neural network and software interface implementation, causal transformation of arrow key points, and intelligent layout of overall outlines.

[0167] Deep Neural Networks and Software Interface Implementation:

[0168] Select the drawing software Visio, combine the software's specific document interface input requirements, extract and transform the raw output of the neural network, and write interface functions to automatically generate the predicted shape;

[0169] Since a suitable Visio development tool is unavailable, this invention uses the win32com extension package of the Python language to launch the Visio software at the Windows operating system level and perform a series of operations such as adding shapes, connecting objects, and setting text. Based on the available interfaces of win32com, the shape positions in the original output of the neural network are transformed into a center point + range pattern. An example of a rectangle is shown below:

[0170] (x0, y0, y1, y2) → (x c y c (H, W)

[0171]

[0172] In the formula: (x0, y0, y1, y2) are the original coordinate outputs of the neural network, (x c ,y c (H,W) is the center point of the rectangle, and (H,W) is the length and width of the rectangle.

[0173] Arrow key point causal transformation:

[0174] By estimating the objects pointed to by arrows, the connection relationships between the objects can be inferred, with the starting point as the cause and the ending point as the effect. Write a function to implement the connection relationship between shape objects.

[0175] The arrow keypoints in the original output of the neural network are converted into connection attribute selections of the shape objects corresponding to the arrow objects based on the estimated shape objects. An example of a rectangle is shown below:

[0176] (x begin y begin ) / (x end y end )→(Center, down, right, up, left)

[0177] In the formula: (x begin ,y begin ) / (x end ,y end () represents the starting / ending point of the arrow key point, and (center, down, right, up, left) represents the optional interface points of the rectangle object.

[0178] Furthermore, in the pixel coordinate system, we can obtain the transformation calculation model for the connection relationship between the estimated shape object corresponding to the arrow object and the four sides of the rectangle to which the arrow key point belongs:

[0179]

[0180] index∈(center,down,right,up,left)

[0181] (x0,y0)∈((x begin ,y begin ),(x end ,y end ))

[0182] In the formula: (x0, y0) are the two-dimensional coordinates of the key point of the arrow. Let be the coordinates of the two endpoints of the i-th side of the rectangle.

[0183] Intelligent layout of overall outline:

[0184] Furthermore, for the generated original editable graphics, the absolute and relative positions of each shape are compared, and a heuristic clustering algorithm is used to achieve horizontal and vertical alignment; shape size standardization and consistency are also performed to complete automatic intelligent layout. The software also provides a built-in intelligent layout algorithm for selection, specifically:

[0185] First clustering: Using the length and width of editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to treat graphics whose length and width feature distances are less than the thresholds as having the same size, obtaining the number of clusters K and the cluster centers m. iNext, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the k-means clustering algorithm, we obtain a set of K editable graphics after clustering. The average of the features is taken as the cluster center. The size of all editable graphics in the same cluster is set to the size of the cluster center, as shown in the following expression:

[0186]

[0187]

[0188] Where: m i Let C be the cluster center of the i-th class of data. i Let i be the data set of the i-th class.

[0189] Secondary clustering: Using the top-left x and y coordinates of the editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to consider coordinates with feature distances less than the threshold as the same horizontal / vertical line. The number of clusters K and the cluster centers m are obtained. i Next, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the k-means clustering algorithm, a set of K editable graphics is obtained after clustering. The alignment value of all editable graphics in the same cluster is set to the alignment value of the cluster center, achieving automatic alignment. Figure 7 As shown, the expression is the same as the one above.

[0190] In practical applications, the two thresholds for the two Canopy algorithms are given by the general model inside the algorithm, as shown in Table 4:

[0191] Table 4. Thresholds for the General Canopy Algorithm

[0192]

[0193]

[0194] In the formula: (length i ,width i (i) represents the length and width of the i-th shape, and (length, width) represents the length and width of all shapes.

[0195] It also provides Visio's built-in intelligent layout algorithm, which, based on the input requirements of the intelligent layout interface, further transforms the selection of connection attributes for shape objects in the causal transformation of arrow key points into automatic connection attribute selection. An example of a rectangle is shown below:

[0196]

[0197] In the formula: (center, down, right, up, left) are optional interface points of the rectangle object. Optional interface points for automatic connection attributes.

[0198] Furthermore, there is a one-to-one correspondence between the shape object of the causal transformation of the arrow key points and the automatic connection property:

[0199]

[0200]

[0201]

[0202]

[0203]

[0204] In the formula: AutoConnectDirAny is any optional interface point for the automatic connection attribute.

[0205] Furthermore, by reserving external interfaces, it achieves integration with existing high-quality OCR function packages for hand-drawn flowchart text recognition, and interfaces with software to achieve automatic text generation, such as... Figure 8 As shown, specifically:

[0206] This invention uses the Python version of PaddleOCR, a natural language processing package under PaddlePaddle, Baidu PaddlePaddle. It extracts the location coordinates of the shape type identified as text from the final position of the shape coordinate detection output shape box, and then uses PaddleOCR for recognition. Compared to the overall OCR of the entire image, the accuracy is significantly improved. Simultaneously, it performs overlap detection with the predicted boxes of other shapes, as shown in the following expression:

[0207]

[0208] In the formula: S1 is the prediction box of the given shape, and S2 is the prediction box of the text.

[0209] Furthermore, since the shape coordinate detection output is always a rectangular candidate box, the specific calculation formula for overlap detection can be obtained as follows:

[0210]

[0211] Where: These are the top-left and bottom-right coordinates of the text box. These are the top-left and bottom-right coordinates of the shape candidate box;

[0212] For J IoU Text prediction boxes exceeding a given threshold are identified as embedded text within the corresponding shape prediction box; for J IoU Text prediction boxes smaller than a given threshold are identified as free text. Simultaneously, it interfaces with PowerPoint / Visio software to embed the built-in text of shapes into corresponding shapes, generating text boxes at the free text prediction locations and embedding the corresponding text.

[0213] like Figures 1-10 As shown in the embodiment of the present invention, the image recognition and intelligent conversion system for hand-drawn scenes includes an acquisition module, a recognition module, a conversion module, and an additional module. The acquisition module consists of a camera, scanning software, or electronic devices, and is connected to the recognition module. The recognition module and the conversion module are control software installed in a computer. One end of the recognition module is connected to the acquisition module, and the other end is connected to the conversion module. One end of the conversion module is connected to the recognition module, and the other end is connected to the drawing software. The additional module is an optional module, connected to the conversion module and the drawing software. The entire image recognition and intelligent conversion system for hand-drawn scenes is powered by the computer and does not require secondary calibration.

[0214] Figure 9 This paper demonstrates the implementation architecture of an image recognition and intelligent conversion system for hand-drawn scenes. The system consists of four modules: an acquisition module, a recognition module, a conversion module, and an additional module. The acquisition module is responsible for acquiring hand-drawn flowchart images, including raw RGB images captured by a camera, pre-processed images after binarization by scanning software, or images directly drawn on electronic devices. The recognition module performs shape coordinate detection, shape type recognition, arrow feature point localization, and arrow pointing object estimation on the acquired hand-drawn flowchart images to determine the position coordinate frame, shape type, arrow key point position, and the shape to which the arrow belongs for each predicted shape. The conversion module performs causal transformation of arrow key points and intelligent layout of the overall outline on the output of the recognition module, and finally outputs an editable graphic for specified software. The additional module performs text recognition on part of the output of the recognition module (recognized as text), embeds the text recognition results into the output of the conversion module, and finally outputs a complete editable graphic.

[0215] Data acquisition module: such as Figure 2 As shown, hand-drawn flowcharts can be captured by ordinary photographic equipment such as cameras, scanned by scanning software, or drawn directly on electronic devices. Finally, the flowcharts are connected to a computer and a recognition module to achieve real-time acquisition and transmission.

[0216] Recognition Module: As part of the computer software, it includes sub-modules such as shape coordinate detection, shape type recognition, arrow feature point localization, and arrow pointing object estimation. It obtains hand-drawn flowchart image information from the acquisition module, which flows sequentially through each sub-module, and finally outputs data such as the position coordinate frame of each predicted shape, shape type, arrow key point position, and the shape to which the arrow belongs.

[0217] The transformation module, as part of the computer software, includes sub-modules such as arrow key point causal transformation and overall contour intelligent layout. It obtains data such as the position coordinate frame of the predicted shape, shape type, arrow key point position and the shape to which the arrow belongs from the output of the recognition module, and sequentially flows through each sub-module to finally output an editable graphic for the specified software.

[0218] Additional module: As part of the computer software, it is an optional module of the system; it performs text recognition by obtaining the position coordinates of the predicted text box output by the recognition module, embeds the text recognition results into the output of the conversion module, and finally outputs a complete editable graphic.

[0219] The image recognition and intelligent conversion system for hand-drawn scenes described above can be implemented as a computer program, stored on a hard disk, and recorded in a processor for execution to implement the method of the present invention.

[0220] This invention also provides a computer-readable medium storing computer program code, which, when executed by a processor, implements the image recognition and intelligent conversion method for hand-drawn scenes based on deep neural networks as described above.

[0221] When the image recognition and intelligent conversion method for hand-drawn scenes is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described in the embodiments of this invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0222] It should be understood that the above embodiments are merely illustrative. The embodiments described in this invention can be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processing unit can be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described in this invention, or combinations thereof.

[0223] It should be noted that in this application, relational terms such as "first," "second," etc., are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0224] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0225] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for image recognition and intelligent conversion in hand-drawn scenes, characterized in that, Includes the following steps; Step 1: Hand-drawn flowchart acquisition: Capture the current hand-drawn flowchart in real time by taking pictures with a camera or by drawing the flowchart directly with software. Collect the hand-drawn flowchart image information at the current moment and input it into the computer to realize the real-time acquisition and transmission of hand-drawn flowcharts. Step 2: Obtain the hand-drawn process image information obtained in Step 1. Through the steps of locating the shape position and identifying the shape category, finally output the position coordinate box, shape type, arrow key point position and the shape data to which the arrow belongs for each predicted shape; Step 3: Obtain the position coordinate frame, shape type, arrow key point position, and shape data of the predicted shape from Step 2, and intelligently display them through computer graphics software to output an editable graphic for the specified software; Step 4: The OCR module is integrated with the OCR function package through a reserved external interface to recognize the text in the hand-drawn flowchart. It also interfaces with the software to automatically generate text from the editable graphic obtained in Step 3, and finally outputs a complete editable graphic. Step 2 involves locating the shape position and identifying the shape category, including shape coordinate detection, shape type identification, arrow feature point localization, arrow pointing object estimation, and setting a confidence threshold. Shape coordinate detection: The original hand-drawn process image information is input into a deep cascaded neural network model for feature extraction, and then the shape candidate box is filtered to output the position coordinate box of each shape; The deep cascaded neural network model includes an FPN network for extracting global image features and an RPN network for generating candidate boxes. Shape type recognition: The position coordinate boxes of each shape are used to extract features in a parallel neural network model, and then shape classification is performed. The shape position coordinate boxes are combined with the input of the shape position coordinate boxes to output the shape box of the final position. The parallel neural network model comprises three parallel networks for shape coordinate frame regression, shape type prediction, and keypoint regression, all of which are composed of fully connected networks. The shape position coordinate frame is composed of a quadruple of (x1, y1, x2, y2) representing the coordinates of the upper left and lower right corners of the coordinate frame, while the final shape frame is composed of a triple of (box, class, score), where box represents the aforementioned shape position coordinate frame, class represents the type of the shape, and score represents the probability that the shape belongs to type class. Arrow feature point localization: The arrow is transformed into a causal relationship connecting shape objects and represented by the beginning and end feature points. The feature points in the arrow shape box of the final position are detected by the constraint model, and the two-dimensional feature key points of the arrow in the arrow shape box of the final position are marked to realize the arrow feature point localization. The connected shape objects include all shapes predicted by shape coordinate detection and shape type recognition; the causal relationship refers to the correspondence between the predicted arrow key points, with the key point closer to the arrow start being the cause and the key point closer to the arrow end being the effect; the constraint model includes border constraints, that is, by comparing the key points with the corresponding arrow coordinate boxes, the key points are restricted to be located within the arrow coordinate boxes. Arrow pointing object estimation: For each feature point in the output arrow shapebox and its surrounding shape, based on the shape position, key point position, and the geometric relationship between the two in the pixel coordinate system, the intelligent algorithm is used to estimate the belonging relationship of the arrow key points; Among them, the attribution relationship of the arrow key points refers to the specific predicted shape corresponding to the key points; the intelligent algorithm estimation includes shape determination and shortest distance: shape determination determines the vertex coordinates of the shape by the symmetry between the predicted rectangular coordinate frame and the predicted type shape, thereby determining the edges; the shortest distance determines the specific shape corresponding to the key point by calculating the shortest distance from the arrow key points to the edges of each shape. Set a confidence threshold: Compare the coordinates and confidence of each predicted shape on the overall image, and combine the confidence of the object the arrow points to. Select the recognition results with reasonable shape position, clear object pointing to by the arrow, and confidence higher than the set threshold as the final shape position coordinate box, shape type, arrow key point position and the shape data to which the arrow belongs. The intelligent display of computer graphics software in step 3 includes deep neural network and software interface implementation, causal conversion of arrow key points, and intelligent layout of the overall outline. Deep Neural Network and Software Interface Implementation: Combining the specific document interface input requirements of computer graphics software, extract the raw output of the neural network and transform it into the input form specified by the software interface. Write interface functions to automatically generate editable graphics based on the predicted shape. The original output of the deep neural network refers to the output of the aforementioned parallel neural network, including the shape coordinate frame, shape type, and arrow key points. Arrow key point causal transformation: Estimate the objects pointed to by the arrows, and then infer the connection relationship between the objects. The starting point is the cause and the ending point is the effect. Write a function to realize the relationship connection between shape objects. Intelligent layout of overall outline: For the generated original editable graphics, the absolute and relative positions of each shape are compared, and a heuristic clustering algorithm is used to achieve horizontal and vertical alignment; the shape size is standardized and consistent to complete the automatic intelligent layout, while providing the software's built-in intelligent layout algorithm for selection; The absolute position of a shape is its position as indicated by its coordinates; the relative position of a shape is its orientation within the whole.

2. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 1, characterized in that, The hand-drawn process image information in step 1 includes the original RGB image captured by the camera, the pre-processed image after binarization by scanning software, or the three-dimensional matrix of the image directly drawn on the electronic device and read in by a programming language.

3. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 1, characterized in that, The shape coordinate detection in step 2 specifically involves: The collected hand-drawn flowchart image is processed by a shape / background binary classifier to obtain the coordinate boxes of each possible shape. The coordinate boxes of each possible shape are then input into the first sub-network structure of a deep cascaded neural network model for feature extraction, outputting numerous candidate boxes. The bounding box regression method is used to merge overlapping candidate boxes to obtain corrected candidate boxes. The corrected candidate boxes are divided into two parts: the first part is used to determine whether there is a shape in the image within the current corrected candidate box, and the second part represents the offset of the shape position within the current corrected candidate box relative to the shape box position of the real target. The corrected candidate boxes obtained through the first sub-network are used as input, and the coarsely predicted candidate boxes are output through the second sub-network structure in the deep cascaded neural network model. The overlapping coarsely predicted candidate boxes are merged using the bounding box regression method to obtain the second corrected candidate boxes. The candidate box obtained after the second correction through the second sub-network structure is used as input, and the final shape and position coordinate box is output through the third sub-network structure in the deep cascaded neural network model.

4. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 3, characterized in that, The shape type identification in step 2 specifically involves: The shape location coordinate box obtained through the third sub-network structure is input into the deep neural network model for feature extraction. The shape multi-classifier outputs the probability of each shape belonging to the location box. The probabilities of each shape are then normalized through a softmax layer, as shown in the following formula: In the formula: z i z j p is the raw output of the classifier. i The normalized probability is N, where N is the total number of shape categories. The cross-entropy loss function of the shape / background binary classifier The expression is as follows: In the formula: The actual label for the shape / background, p i The probability of the shape; The bounding box regression method uses non-maximum suppression to merge overlapping candidate boxes, resulting in corrected candidate boxes. The bounding box regression method calculates the background coordinates of the candidate boxes. Compared to the actual background coordinates Calculate the regression loss function based on the Euclidean distance between them. The expression is as follows: In the formula: The L2 loss function represents the Euclidean distance. This represents the position coordinates of the i-th actual shapebox. R represents the position coordinates of the i-th predicted shapebox. 4 Represents the four-dimensional real number space; The cross-entropy loss function of the shape multi-classifier The expression is as follows: In the formula: This refers to the actual shape label below the shape frame. This represents the predicted probability.

5. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 1, characterized in that, The estimation of the object pointed to by the arrow in step 2 is specifically as follows: Based on the shape type of the shape box at the final position output by shape coordinate detection, it is identified as an arrow / line / double arrow position coordinate box, collectively referred to as an arrow box. Feature extraction and regression are performed independently on the key points of each arrow box: These are the two actual two-dimensional key points within the arrow frame: the start point and the end point. To predict two corresponding predicted 2D keypoints within the arrow box, the arrow keypoint linear regression method calculates the regression loss function by measuring the Euclidean distance between the actual 2D keypoints in the actual arrow box and their corresponding predicted 2D keypoints in the predicted arrow box. Where: For the i-th actual 2D key point in the actual arrow box, For the predicted two-dimensional key point in the corresponding i-th prediction arrow box; The process involves determining the final position of the key points within the arrow frame detected by the arrow key point detection, and comparing them with the surrounding shape. Based on the geometric relationship between the shape position, the key point position, and the two in the pixel coordinate system, an intelligent algorithm is used to estimate the attribution relationship of the arrow key points. Specifically: The distance measurement between keypoints and neighboring shape objects is abstracted as the Euclidean distance from a point to a line. For a given polygon and the arrow keypoint, the shortest distance is selected as the distance from the keypoint to the polygon using the formula for the distance from a point to a line. The expression is as follows: In the formula: (x0, y0) are the two-dimensional coordinates of the key point of the arrow. Let the coordinates be the coordinates of the two endpoints of the i-th side of the polygon; For all predicted shape objects and arrow keypoints, the distance is calculated, and the polygon with the smallest distance is selected as the object to which the arrow keypoint belongs. The expression is as follows: In the formula: d i The distance from the arrow keypoint to the i-th shape among all predicted shape objects.

6. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 1, characterized in that, In step 3, the drawing software PowerPoint / Visio, combined with the software's specific document interface input requirements, extracts and transforms the raw output of the neural network, writes an interface function to automatically generate predicted shapes, and simultaneously estimates objects by pointing to them with arrows, thereby inferring the connection relationships between objects. The starting point is the cause, and the ending point is the effect. The function that implements the connection between shape objects is as follows: In the integration with PowerPoint software, the existing PowerPoint development tool, the pptx extension package of the Python language, is used to directly manipulate the PowerPoint software and perform a series of operations such as adding shapes, connecting objects, and setting text. In the process of interfacing with Visio software, the win32com extension package of the Python language is used to launch Visio software through the Windows operating system and perform a series of operations such as adding shapes, connecting objects, and setting text. To meet the interface requirements of PPTX and Win32COM, the shape and position in the original output of the neural network are transformed into a center point + range pattern. An example of a rectangle is shown below: (x0,y0,y1,y2)→(x c ,y c ,H,W) In the formula: (x0, y0, y1, y2) are the original coordinate outputs of the neural network, (x c ,y c (H,W) is the center point of the rectangle, and (H,W) is the length and width of the rectangle. The arrow keypoints in the original output of the neural network are converted into connection attribute selections of the shape objects corresponding to the arrow objects based on the estimated shape objects. An example of a rectangle is shown below: (x begin ,y begin ) / (x end ,y end )→(center,down,right,up,left) In the formula: (x begin ,y begin ) / (x end ,y end () represents the starting / ending point of the arrow key point, and (center, down, right, up, left) are optional interface points of the rectangle object; Meanwhile, in Visio operations, the above interface calls are simplified by encapsulating them into direct function calls; The intelligent layout of the overall outline in step 3 specifically involves: First clustering: Using the length and width of editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to treat graphics whose length and width feature distances are less than the thresholds as having the same size, obtaining the number of clusters K and the cluster centers m. i Next, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the k-means clustering algorithm, we obtain a set of K editable graphics after clustering. The average of the features is taken as the cluster center. The size of all editable graphics in the same cluster is set to the size of the cluster center, as shown in the following expression: Where: m i Let C be the cluster center of the i-th class of data. i Let i be the data set of the i-th class; Secondary clustering: Using the top-left x and y coordinates of the editable graphics as features, a Canopy algorithm + k-means clustering mode is used: First, coarse clustering is performed on all editable graphics, setting two thresholds T1 and T2. Then, the Canopy algorithm is used to consider coordinates with feature distances less than the threshold as the same horizontal / vertical line. The number of clusters K and the cluster centers m are obtained. i Next, set the number of clusters in the k-means clustering algorithm to K, and the initial cluster centers to m. i Using the kmean clustering algorithm, we obtain a set of K editable graphics after clustering. We set the alignment value of all editable graphics in the same cluster to the alignment value of the cluster center to achieve automatic alignment. The expression is the same as the above formula. It also provides Visio's built-in intelligent layout algorithm, which, based on the input requirements of the intelligent layout interface, further transforms the selection of connection attributes for shape objects in the causal transformation of arrow key points into automatic connection attribute selection. An example of a rectangle is shown below: In the formula: (center, down, right, up, left) are optional interface points of the rectangle object. Optional interface points for automatic connection attributes.

7. The image recognition and intelligent conversion method for hand-drawn scenes according to claim 1, characterized in that, The OCR module in step 4 is specifically configured as follows: We selected the Python version of PaddleOCR, the natural language processing package of PaddlePaddle under Baidu PaddlePaddle, to extract the position coordinates of the shape type identified as text from the final position of the shape coordinate detection output. We then used PaddleOCR for recognition and simultaneously performed overlap detection with the predicted bounding boxes of other shapes. The expression is as follows: In the formula: S1 is the prediction box of the given shape, and S2 is the prediction box of the text; For J IoU Text prediction boxes exceeding a given threshold are identified as embedded text within the corresponding shape prediction box; for J IoU Text prediction boxes smaller than a given threshold are identified as free text. Simultaneously, the software interfaces with PowerPoint / Visio to embed the built-in text of the shape into the corresponding shape, and generates a text box at the free text prediction location and embeds the corresponding text.

8. An image recognition and intelligent conversion system for hand-drawn scenes, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing the instructions to implement the method as described in any one of claims 1-7.

9. A computer-readable medium, characterized in that, The device stores computer program code that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images