An AI image recognition-based zero-code form automatic generation method

Abstract

The application discloses an AI image recognition-based zero-code form automatic generation method, which comprises the following steps: receiving a sketch or a screenshot provided by a user, performing Gaussian filter noise reduction, simplified Canny algorithm edge detection and Otsu method binarization processing on the sketch or the screenshot; utilizing an object detection model to identify form components in the preprocessed image, and outputting the boundary box coordinates of each component; detecting a connection line by Hough transformation, judging whether the connection line is an arrow, and extracting the potential relationship between the form components; then constructing a dynamic relationship graph according to the extracted potential relationship, utilizing a graph neural network to analyze the logical relationship in the dynamic relationship graph, optimizing a user interaction path through reinforcement learning, and generating an interactive form according to the analyzed logical relationship and the optimized interaction path; the method can automatically generate an interactive form from a sketch or a screenshot provided by a user, reduces the form development threshold, and improves the form development efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent application development, and in particular to a zero-code form automatic generation method based on AI image recognition. BACKGROUND

[0002] With the rapid development of information technology, the demand for form design and development of enterprises and individual users is increasing. Traditional form development methods usually require professional developers to manually code to achieve this. This approach not only has a long development cycle and high cost, but also makes it difficult to quickly respond to changes in user needs. In recent years, zero-code or low-code development platforms have gradually emerged. Such platforms allow users to quickly build forms through a graphical interface by dragging and dropping components, greatly improving development efficiency. However, even with zero-code platforms, users still need to manually select and layout form components, especially when faced with complex forms or a large number of form components, the design efficiency and accuracy of users are still greatly limited. In addition, existing zero-code platforms usually lack the ability to automatically recognize and understand user sketches or existing form screenshots, and cannot automatically generate interactive forms from user-provided design sketches or screenshots. Users still need to do a lot of repetitive work, which seriously affects the efficiency of form design and user experience.

[0003] On the other hand, deep learning technology in artificial intelligence technology has made significant progress in image recognition, object detection, and relationship inference. YOLO series algorithms, graph neural networks (GNN), and reinforcement learning (RL) have matured and are widely used. However, current technical means cannot automatically recognize form components in user-provided sketches or screenshots and accurately infer the logical relationships between components, resulting in the inability to automatically generate interactive forms from user design sketches or screenshots. In addition, existing methods usually rely on manually defined rules for form component relationship inference, lack a deep understanding of complex logical relationships between form components, and are difficult to achieve efficient and accurate automatic form generation. Therefore, how to accurately and efficiently realize the automatic generation of interactive forms from user sketches or screenshots has become a technical problem that needs to be solved in the field. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, abstract and title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed. Therefore, the present application provides an AI image recognition-based zero-code form automatic generation method to solve the problems proposed in the background art.

[0006] To solve the above technical problems, the present application provides the following technical solutions: an AI image recognition-based zero-code form automatic generation method, comprising:

[0007] Receiving a sketch or screenshot provided by a user, performing image preprocessing on the sketch or screenshot, and using an object detection model to identify form components in the preprocessed image, and extracting potential relationships in the form components;

[0008] According to the potential relationships in the form components, obtaining the logical relationships in the form components and constructing a dynamic relationship graph, and using a graph neural network to analyze the logical relationships in the dynamic relationship graph;

[0009] Optimizing the user interaction path through reinforcement learning, and generating an interactive form according to the analyzed logical relationships and the optimized interaction path.

[0010] As a preferred scheme of the AI image recognition-based zero-code form automatic generation method of the present application, wherein: the image preprocessing of the sketch or screenshot comprises:

[0011] The sketch or screenshot is regarded as an image, image noise is suppressed by Gaussian filtering, image edges are detected by a simplified Canny algorithm, and the image is binarized by the Otsu method.

[0012] As a preferred scheme of the AI image recognition-based zero-code form automatic generation method of the present application, wherein: using an object detection model to identify form components in the preprocessed image comprises:

[0013] Taking YOLOv5 as the object detection model, generating a labeled image set of form components through a graphical image annotation tool labelme, taking the preprocessed image as input, identifying the form components in the preprocessed image according to the labeled image set of form components, and outputting the bounding box coordinates of each form component.

[0014] As a preferred scheme of the AI image recognition-based zero-code form automatic generation method of the present application, wherein: extracting potential relationships in the form components comprises:

[0015] Detecting the connection line in the form based on the Hough transform, setting the minimum line length and the maximum gap, and analyzing the pixel distribution near the line segment endpoints on the connection line to determine whether the connection line is an arrow, and if it is an arrow, outputting the starting point, ending point coordinates and arrow direction of the arrow.

[0016] As a preferred scheme of the zero-code form automatic generation method based on AI image recognition, wherein: according to the potential relationship in the form component, the logical relationship in the form component is obtained and a dynamic relationship graph is constructed, including:

[0017] By the text detection method, the text position and the text content of the position are located, and according to the arrow starting point, the arrow ending point and the arrow direction, the logical relationship in the form component is obtained;

[0018] Each form component is defined as a node in the dynamic relationship graph, and the node attribute includes the bounding box coordinates;

[0019] If a connection line is detected, a directed edge is added between the two nodes, if no connection line is detected, but the spatial distance between the two nodes is less than a preset threshold, and the category has a dependency relationship, an undirected edge is added;

[0020] All the formed directed edges or undirected edges are connected.

[0021] As a preferred scheme of the zero-code form automatic generation method based on AI image recognition, wherein: the logical relationship between the nodes is inferred by using a graph neural network, including:

[0022] According to the dynamic relationship graph, a node feature matrix is created;

[0023] The node features are updated through the attention mechanism of the graph neural network, and each edge where the node is located is classified, and a labeled dynamic relationship graph is output.

[0024] As a preferred scheme of the zero-code form automatic generation method based on AI image recognition, wherein: the user interaction path is optimized through reinforcement learning, including:

[0025] The user interaction path is modeled through a Markov decision process, the state in the Markov decision process is represented as the form filling state, the action is represented as the user operation process, and the reward is defined as the efficiency of the user filling the form.

[0026] As a preferred scheme of the zero-code form automatic generation method based on AI image recognition, wherein: further comprising:

[0027] In the Markov decision process, a proximal policy optimization algorithm is used, and the user interaction path is updated in real time according to the state and the action.

[0028] As a preferred scheme of the zero-code form automatic generation method based on AI image recognition, wherein: according to the analyzed logical relationship and the optimized interaction path, an interactive form is generated, including:

[0029] The edge relationship in the labeled dynamic relationship graph is converted into a JavaScript rule and embedded into a static element rendered based on HTML / CSS to generate an interactive form.

[0030] Compared with the prior art, the application has the following advantages:

[0031] 1. The AI image recognition technology is used to automatically identify form components from the user-provided sketch or screenshot and extract their potential relationships and logical relationships, thereby saving the tedious manual configuration steps, and without programming knowledge, the development threshold is greatly reduced, so that non-professional users can quickly complete form design, reduce repetitive work, and improve design efficiency.

[0032] 2. The graph neural network (GNN) technology is used to analyze the potential relationships and logical relationships in the form components based on the dynamic relationship graph, which can deeply understand the dependency and interaction logic between components, overcome the limitations of traditional manual rules, and lay a foundation for forming a functional and interactive form.

[0033] 3. The reinforcement learning (RL) technology is used to intelligently optimize the user interaction path based on the Markov decision process and the proximal policy optimization algorithm, and adjust the filling process according to the design of the form, so that the user operation is more smooth and efficient, effectively improving the convenience of form use and user satisfaction, and solving the problem of insufficient user experience in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings. Among them:

[0035] Figure 1 The overall flowchart of the zero-code form automatic generation method based on AI image recognition described in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0037] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0038] It should also be noted that, as used in the specification and the appended claims, the articles "a," "an," "the" and "said" are intended to mean that there are one or more of the features (i.e., either one feature or more than one feature). The articles "a" (or "an"), "the" and "said" are to be construed as one or more unless otherwise stated.

[0039] The present application is described in detail below with reference to the attached drawings.

[0040] In the description of the present application, it should be noted that the terms "upper and lower, inner and outer" and the like indicate the positional or directional relationship shown in the drawings, and are intended to facilitate the description of the present application and simplify the description, and are not intended to indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be construed as limiting the present application. In addition, the terms "first, second or third" are for descriptive purposes only and cannot be construed as indicating or implying relative importance.

[0041] Unless otherwise defined and limited in the specification, the terms "mounting, connecting, connecting" should be broadly understood, for example: it can be fixedly connected, detachably connected or integrally connected; it can also be mechanically connected, electrically connected or directly connected; it can also be indirectly connected through an intermediate medium; it can also be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] Embodiment 1

[0043] Reference Figure 1 For the first embodiment of the present application, the embodiment provides an AI image recognition based zero-code form automatic generation method, comprising:

[0044] S1, receiving a sketch or screenshot provided by a user, performing image preprocessing on the sketch or screenshot, and using an object detection model to identify form components in the preprocessed image, and extracting potential relationships in the form components;

[0045] Specifically, the user uploads a hand-drawn form sketch (take a photo after drawing on paper) or a screenshot (which can be a page screenshot of a website or a custom screenshot such as a WeChat screenshot), saves the sketch or screenshot as an image file in JPG or PNG format, and places it in the corresponding picture folder.

[0046] It should be explained that, since the original image (image file) placed in the picture folder may have problems such as noise, uneven illumination or blurred edges, directly processing the original image through the object detection model will affect the accuracy of identifying form components, so the original image needs to be preprocessed before recognition.

[0047] Further, the noise of the original image is suppressed by Gaussian filtering, wherein the Gaussian filtering is an image processing filtering technology based on normal distribution function, which mainly functions to smooth the image, thereby reducing noise, and can preserve more image details than simple mean filtering, achieving the purpose of accurately identifying form components, and the specific steps are as follows:

[0048] First, set or select the Gaussian kernel size and standard deviation to generate a Gaussian kernel, i.e. a weight matrix; the Gaussian kernel size is obtained from the size of the Gaussian filter; then, the original image is taken as input, and a two-dimensional convolution is performed between the Gaussian filter and the original image; for each pixel in the original image, the pixel value and the pixel value around the pixel are taken, and weighted summation is performed according to the weight matrix to form a new pixel value; repeat the convolution operation until the entire image is traversed through the pixel points, and the denoised original image is obtained.

[0049] It should be noted that the Gaussian filtering removes the noise in the original image, making the original image clearer.

[0050] Further, the image edge is detected by a simplified Canny algorithm, wherein the Canny edge detection algorithm is an edge detection method for image processing, which can accurately detect the edges in the image to highlight the boundaries of the form components in the image, and the specific steps are as follows:

[0051] Based on the original image processed by the Gaussian filter (after denoising), the direction of a pixel point in the image is taken, which is divided into horizontal (x) component and vertical (y) component, and the gradient value of the pixel point in the horizontal component and the vertical component is calculated by using the sobel operator associated with the image, to obtain the gradient value of the pixel point, which is taken as the edge region extracted from the image.

[0052] It needs to be explained that, since considering that the form border in the image is usually a straight line or a rectangle, it has high contrast, simple structure and obvious boundary, unlike the complex edges in natural images, based on the actual characteristics of the form border, the gradient image is used to extract the form border with simple threshold processing, without further processing by non-maximum suppression and double threshold, which simplifies the image edge detection process of Canny algorithm and improves the efficiency of image processing;

[0053] Further, the image after edge detection is binarized by Otsu method to distinguish the form components from other objects in the image under different light (uniform) conditions, wherein the Otsu method is an automatic threshold selection algorithm for image segmentation, the purpose of which is to select a best threshold to make the gray difference between the foreground and background of the image as large as possible, so as to clearly distinguish the two kinds of pixels, and the specific steps are as follows:

[0054] According to the image after edge detection, the number of pixels of each gray level (0~255) in the image is counted, and the probability of each gray level is calculated. All gray levels are taken as thresholds, the image is divided into foreground and background classes by traversing all gray levels, the inter-class variance of the foreground and background classes is calculated, the maximum threshold of the inter-class variance is found, and the pixel values greater than the maximum threshold in the image are represented as foreground, and the pixel values less than or equal to the maximum threshold are represented as background.

[0055] It needs to be explained that in the image, the form components and the background usually have different gray characteristics (for example, the text box is usually dark, and the background is usually light), and the Otsu method can automatically find the best threshold according to the gray distribution of the image, which saves the problem of manual setting;

[0056] Further, taking YOLOv5 as an object detection model, a form component annotation image set is generated by using a graphical image annotation tool labelme, the preprocessed image is taken as input, the form components (text box, drop-down menu, etc.) in the preprocessed image are recognized according to the form component annotation image set, and the bounding box coordinates of each form component are output.

[0057] It should be noted that the bounding box and the text box have essential differences, the bounding box only represents the position, and it does not contain the input content, while the text box can input content; in addition, each form component has a bounding box, and the component body wrapped by the bounding box is a specific form component.

[0058] It should be noted that labelme is an open source image annotation tool that supports bounding box (bounding box) annotation, and the operation steps are as follows:

[0059] Open labelme, load the form image, draw the bounding box by selecting the "Create Rectangle" tool, then input (label) the name for each drawn bounding box, and save the labeling result to get the corresponding JSON file;

[0060] It should be noted that YOLOv5 can only recognize rectangles, that is, the input image must be a matrix, if you want to use images of other shapes, you can achieve it by padding or cropping; in addition, since YOLOv5 uses TXT files to store labeling information, and labelme generates JSON files, it is necessary to convert the JSON file into the format required by YOLOv5;

[0061] Specifically, the JSON file is converted into a TXT file, so that each image in the labeled image set corresponds to a TXT file, and each line in the TXT file can represent a form component, and by traversing each line in the TXT file using YOLOv5, the bounding box coordinates of each form component in the TXT file are output;

[0062] It should be noted that to detect the connection line in the form and determine whether the connection line is an arrow, Hough Transform can be used, where Hough Transform is a feature extraction technique mainly used for detecting geometric shapes in images, and arrows usually play a guiding and prompting role in user sketches of form design, for example, "basic information → contact information → confirm submission", "fill in the ID number first → upload the certificate photo" and the like;

[0063] Further, based on Hough Transform to detect the connection line (straight line) in the form, by setting the minimum line length and the maximum gap, and analyzing the pixel distribution near the end points of the line segment on the connection line, the potential relationship of the form components is obtained;

[0064] Specifically, the minimum line length is set, only the line segment whose length exceeds the minimum line length will be detected; the maximum gap is set, if the gap in the line segment is less than the maximum gap, it will be connected into a complete line segment (form a connection line), and all complete line segments detected are aggregated into a connection line list, where each connection line is represented by the start point coordinates and the end point coordinates;

[0065] Further, it is determined whether the connection line is an arrow, if it is an arrow, the start point, end point coordinates and arrow direction of the arrow are output;

[0066] It should be noted that the typical feature of an arrow is that one end will have an extra pixel, and the pixel will form a sharp end (such as a triangle or V shape);

[0067] Further, a small area composed of pixels, for example, a 10*10 pixel rectangle, is defined near the start point coordinate and the end point coordinate of each connection line; the number of dark pixels in the area is counted, and if the pixel density near either one of the end points is greater than or equal to the pixel density of the main body of the connection line, it is considered that there is an arrow, and if neither of the two end points has a clear arrow feature, that is, the pixel density near both of the end points is lower than the pixel density of the main body of the connection line, it is considered that there is no arrow;

[0068] It should be noted that for the existing arrow, a template of the arrow can be predefined, and the arrow is prevented from being misrecognized by comparing the template with the existing arrow;

[0069] Specifically, the direction of the arrow is determined according to the position of the arrow, and the start point coordinate, the end point coordinate and the direction of the arrow are output. In general, one end of the arrow is the end point, and the other end is the start point. For example, the start point coordinate of the connection line is (x1, y1), and the end point coordinate is (x2, y2). If there is an arrow near the end point coordinate (x2, y2), the direction of the arrow is from (x1, y1) to (x2, y2). If there is an arrow near the end point coordinate (x1, y1), the direction of the arrow is from (x2, y2) to (x1, y1).

[0070] It should be noted that the potential relationship is a connection structure between the form components on the user sketch, but lacks semantic depth, so it is necessary to understand the interaction mode between the form components, rather than only generating the connection between the form components.

[0071] S2, obtaining the logical relationship in the form components and constructing a dynamic relationship graph according to the potential relationship in the form components, and analyzing the logical relationship in the dynamic relationship graph by using a graph neural network;

[0072] Further, the text position and the text content of the position are located by using a text detection method, and the logical relationship in the form components is obtained according to the start point coordinate, the end point coordinate and the direction of the arrow;

[0073] It should be noted that the logical relationship in the form components can be data linkage, conditional jump and process control, respectively.

[0074] It should be noted that the text detection method can be implemented by using a text detection model EAST or an optical character recognition technology OCR, and each boundary box in the form is output. The text position and the text content are obtained according to the boundary box.

[0075] Specifically, for the arrow starting point, the distance between the arrow starting point and all the bounding boxes is calculated, the nearest bounding box is found, and it is considered as the "source" of the arrow; for the arrow ending point, the distance between the arrow ending point and all the bounding boxes is calculated, the nearest bounding box is found, and it is considered as the "target" of the arrow; if there are multiple bounding boxes or arrow arrangements, overlap may occur; at this time, further judgment can be made in combination with the arrow direction and the nearest text position of the bounding box; after determining the "source" and "target", each arrow is recorded as a "source text→target text" pair;

[0076] For example, assuming that the user sketch of the form design has the following elements: a drop-down box A labeled "country"; a drop-down box B labeled "city"; a check box C labeled "agree"; a text box D labeled "details"; a button E labeled "submit"; a text box F labeled "server"; arrow 1: from drop-down box A to drop-down box B; arrow 2: from check box C to text box D; arrow 3: from button E to text box F; according to arrow 1, the selection of drop-down box A will affect the change of drop-down box B, for example, when "country" is "China", the "city" drop-down box will display "Beijing, Shanghai", etc.; the logical relationship is represented as data linkage; according to arrow 2, the state of check box C will select the display content of text box D, for example, if check box C is in the checked state, text box D will change from the hidden state to the displayed state for user filling; the logical relationship is conditional jump; according to arrow 3, the triggering of button E will control the transmission state of the data; the logical relationship is flow control;

[0077] It should be noted that from the potential relationship to the logical relationship, the semantic depth between the form components is given, so that the form components have interactive logic;

[0078] Further, a dynamic relationship diagram is constructed, each form component is defined as a node in the dynamic relationship diagram, and the node attribute includes the bounding box coordinates;

[0079] Further, if a connection line is detected, a directed edge is added between the two nodes, and if no connection line is detected, but the spatial distance between the two nodes is less than a preset threshold and there is a dependency relationship between the categories, an undirected edge is added;

[0080] It needs to be explained that since the detected connection line is directly mapped as a directed edge in the relationship graph, for the connection line with an arrow, the direction of the arrow determines the direction of the directed edge; for example, if there is an arrow in the sketch from "country" to "city", a directed edge from the "country" node to the "city" node will be added in the relationship graph, indicating that "country" directly affects "city", indicating a direct relationship; if no connection line is detected, an undirected edge will be added, for example, if the "name" text box and the "address" text box are adjacent in the form design, and there is no arrow connection, but semantically "name" and "address" both belong to personal information, an undirected edge is added between them, indicating that there is an indirect relationship between them;

[0081] Specifically, the spatial distance (two-dimensional) The calculation formula is:

[0082]

[0083] Among them, and respectively represent the horizontal (horizontal) coordinates of nodes i and j, and respectively represent the vertical (vertical) coordinates of nodes i and j;

[0084] Specifically, the preset threshold refers to the middle value of the average distance and the maximum distance, and the distance is usually in units of pixels;

[0085] Specifically, the category dependency refers to the dependency relationship between the same category or different categories, for example, there is a same category dependency relationship between the drop-down box and the drop-down box (the "country" drop-down box and the "city" drop-down box), and there is a different category dependency relationship between the check box and the text box (the "agree" check box and the "detail information" text box);

[0086] Specifically, all the formed directed edges or undirected edges are connected to obtain a dynamic relationship graph;

[0087] It should be noted that by constructing the dynamic relationship graph, the user sketch can be converted from a static interactive logical relationship to a dynamic interactive logical relationship, so that non-professional users can also quickly complete form design;

[0088] Further, by analyzing the logical relationship in the dynamic relationship graph through the graph neural network, a node feature matrix is created according to the dynamic relationship graph, the node features are updated through the attention mechanism of the graph neural network, and each edge where the node is located is classified, and a labeled dynamic relationship graph is output;

[0089] It needs to be explained that the graph neural network (GNN) is a kind of deep learning model specially processing graph structure data, which can mainly make up for the deficiency of potential relationship and infer deeper logical relationship, and is used as an optimization operation of the dynamic relationship graph in the scheme of the application;

[0090] Specifically, the bounding box coordinates in the node attribute are normalized to the range of [0, 1], the width w and the height h of the bounding box coordinates are defined, and a 4-dimensional vector [x, y, w, h] is obtained:

[0091]

[0092]

[0093]

[0094]

[0095] wherein x min represents the x coordinate of the leftmost side of the bounding box, and represents the left boundary of the bounding box; x max represents the x coordinate of the rightmost side of the bounding box, and represents the right boundary of the bounding box; y min represents the y coordinate of the uppermost side of the bounding box, and represents the upper boundary of the bounding box; y max represents the y coordinate of the lowermost side of the bounding box, and represents the lower boundary of the bounding box;

[0096] Specifically, the form component is represented as a 3-dimensional vector by using one-hot encoding, and then the text content in the form component is converted into a text vector form, for example, "country" is converted into a 768-dimensional vector;

[0097] Further, the 4-dimensional vector, the 3-dimensional vector and the text vector are spliced to obtain a node feature vector;

[0098] Specifically, the splicing method is a summation operation between vectors;

[0099] Further, all the node feature vectors obtained by the above method are stacked into a node feature matrix F, which is represented as: wherein N represents the total number of nodes, D represents the feature dimension, and R is a real number;

[0100] Further, for each node, the attention weight of the node and the neighbor node of the node is calculated , assuming that j is a neighbor node, then:

[0101]

[0102] wherein,

[0103] wherein, and respectively represent the current node feature vector of node i and node j, represents a concatenation operation as a node feature vector, represents a parameter vector as an attention mechanism, is a "key" in the attention mechanism, represents an index of node i, and provides a query operation for node i;

[0104] Then, update the feature of node i:

[0105]

[0106] wherein, represents the updated node i;

[0107] Further, in the same way of updating the feature of node i, update the neighbor node j to obtain and again through concatenation, obtain the concatenation feature input the concatenation feature into a fully connected neural network in the graph neural network, the fully connected neural network assigns a corresponding logical relationship label to each edge, and outputs a labeled dynamic relationship graph;

[0108] It should be noted that by outputting the labeled dynamic relationship graph, the system can understand the overall logical structure of the form design in the sketch of the user, thereby optimizing the user interaction process;

[0109] S3, optimizing the user interaction path through reinforcement learning, generating an interactive form according to the analyzed logical relationship and the optimized interaction path;

[0110] It should be noted that in order to optimize the filling experience of the user in the interactive form, Markov Decision Process (MDP) is used to model the user interaction path in the present application scheme;

[0111] Specifically, the state in the Markov decision process is represented as the filling state of the form, and then the filling state of each component (text box, drop-down box, check box, etc.) in the form is abstracted as a vector; wherein, not filled: 0; filled: 1; for example, the form contains 3 components (text box: "name", drop-down box: "country", drop-down box: "city"), the state vector is: [1, 0, 0], which means the name has been filled, and the country and city have not been filled;

[0112] Specifically, the action in the Markov decision process is represented as the user operation process, for example, filling in the text box (inputting "name"), selecting the drop-down box (selecting "country" as "China"), etc.

[0113] It should be noted that the design goal of the reward is to encourage the user to quickly and efficiently complete the form;

[0114] Specifically, the reward in the Markov decision process is defined as the efficiency of the user filling out the form, which is divided into time efficiency and operation fluency, wherein the time efficiency is the filling time of the reward (that is, the shorter the time, the higher the reward); for operation fluency, if the user's action conforms to the logic (such as filling in "name" first and then filling in "country"), a positive reward is given; if the user's action does not conform to the logic (such as filling in the contact information without filling in any personal information), a negative reward is given;

[0115] It should be explained that in order to optimize the user interaction path under the MDP framework, the proximal policy optimization (PPO) algorithm is adopted in the present application, which is an efficient reinforcement learning method and can learn the optimal policy while maintaining the stability of the policy;

[0116] Specifically, the PPO algorithm learns through a policy network (Policy Network) and a value network (Value Network); the policy network outputs the probability distribution of the action according to the state in the Markov decision process, for example, the probability of "filling in name" is 0.6 and the probability of "selecting country" is 0.4; and the value network estimates the value of each state, which is used to evaluate the long-term return of the action taken in the current state;

[0117] Specifically, by collecting user interaction data with the form, calculating the generalized advantage function, and updating the parameters of the above policy network and value network, the optimal user interaction path is gradually learned;

[0118] It should be explained that the advantage function is the good or bad of performing a specific action in a certain state compared to the average case, which is usually defined as A(s, a) = Q(s, a) - V(s), wherein s represents the current state (such as the filling state of the form), a represents the action taken (a certain content of the form filling), and V(s) represents the expected cumulative return value in state s following the current policy; if A(s, a) > 0, it means that the action a in state s is better than the average case, and the policy needs to increase the probability of selecting the action; if A(s, a) < 0, it means that the action a in state s is worse than the average case, and the policy needs to reduce the probability of selecting the action; if A(s, a) = 0, it means that the action a in state s is the same as the average case, and the policy does not need to be changed;

[0119] Specifically, the generalized advantage function is represented as:

[0120]

[0121] wherein, represents the weighted contribution value of all future time steps (infinite steps) from the current time, in fact, due to the discount factor and the existence of the smoothing parameter , the influence of the time step will gradually decrease, so the summation can be approximated to a finite step; represents the time difference error, wherein , represents the time difference error (such as the efficiency of filling), represents the value estimation of the state, which is obtained by the value network output; represents the discounted value of the next state;

[0122] It should be noted that by continuously adjusting the generalized advantage function, the policy network continuously outputs the probability distribution of high advantage actions, thereby gradually optimizing the user interaction path;

[0123] Specifically, according to the optimal user interaction path learned gradually, the edge relationship in the labeled dynamic relationship graph is converted into JavaScript rules, and is embedded into static elements based on HTML / CSS rendering, so as to generate an interactive form;

[0124] It should be noted that the JavaScript rules are not manually written by the user, but are automatically generated by the system according to the output labeled dynamic relationship graph, and the <script>标签将该JavaScript规则嵌入到HTML / CSS渲染的静态元素中，实现表单页面前端的交互功能。

[0125] 本领域内的技术人员应明白，本发明实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质（包括但不限于磁盘存储器、CD-ROM、光学存储器等）上实施的计算机程序产品的形式。本申请实施例中的方案可以采用各种计算机语言实现，例如，面向对象的程序设计语言Java和直译式脚本语言JavaScript等。

[0126] 本申请是参照根据本申请实施例的方法、设备（系统）、和计算机程序产品的流程图和／或方框图来描述的。应理解可由计算机程序指令实现流程图和／或方框图中的每一流程和／或方框、以及流程图和／或方框图中的流程和／或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的装置。

[0127] 这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能。

[0128] 这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的步骤。

[0129] 尽管已描述了本申请的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。

[0130] 显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。< / script>

Claims

1. An AI image recognition-based zero-code form automatic generation method, characterized in that, The method comprises the following steps: Receiving a sketch or screenshot provided by a user, performing image preprocessing on the sketch or screenshot, and identifying form components in the preprocessed image using an object detection model to extract potential relationships in the form components; The step of extracting potential relationships in the form components comprises the following steps: Detecting a connection line in the form based on a Hough transform, setting a minimum line length and a maximum gap, and analyzing the distribution of pixels near the endpoints of the line segment on the connection line to determine whether the connection line is an arrow, and if it is an arrow, outputting the coordinates of the starting point and the ending point of the arrow and the direction of the arrow; According to the potential relationships in the form components, obtaining the logical relationships in the form components and constructing a dynamic relationship graph, and analyzing the logical relationships in the dynamic relationship graph using a graph neural network; The step of obtaining the logical relationships in the form components and constructing a dynamic relationship graph according to the potential relationships in the form components comprises the following steps: Locating the position of the text and the text content at the position by a text detection method, and obtaining the logical relationships in the form components according to the coordinates of the starting point and the ending point of the arrow and the direction of the arrow; Defining each form component as a node in the dynamic relationship graph, and the attribute of the node includes the coordinates of the bounding box; If a connection line is detected, a directed edge is added between the two nodes, and if no connection line is detected, an undirected edge is added between the two nodes if the spatial distance between the two nodes is less than a preset threshold and there is a dependency relationship between the categories; Connecting all the formed directed edges or undirected edges to obtain the dynamic relationship graph; The step of analyzing the logical relationships in the dynamic relationship graph using a graph neural network comprises the following steps: Creating a node feature matrix according to the dynamic relationship graph; Updating the node features through the attention mechanism of the graph neural network, classifying each edge where the node is located, and outputting a labeled dynamic relationship graph; Optimizing the user interaction path through reinforcement learning, and generating an interactive form according to the logical relationships obtained by analysis and the optimized interaction path; The step of optimizing the user interaction path through reinforcement learning comprises the following steps: Modeling the user interaction path through a Markov decision process, representing the state of the form as the state in the Markov decision process and representing the user operation process as the action, and defining the reward as the efficiency of the user filling out the form. 2.The AI image recognition-based zero-code form automatic generation method of claim 1, wherein The step of performing image preprocessing on the sketch or screenshot comprises the following steps: Regarding the sketch or screenshot as an image, suppressing image noise through Gaussian filtering, detecting the image edge through a simplified Canny algorithm, and binarizing the image through the Otsu method. 3.The AI image recognition based zero-code form automatic generation method of claim 1, wherein, The step of identifying form components in the preprocessed image using an object detection model comprises the following steps: Taking YOLOv5 as the object detection model, generating a labeled image set of the form components through the graphical image annotation tool labelme, inputting the preprocessed image, identifying the form components in the preprocessed image according to the labeled image set of the form components, and outputting the bounding box coordinates of each form component. 4.The AI image recognition-based zero-code form automatic generation method of claim 1, wherein, Further comprising the following steps: In the Markov decision process, using a proximal policy optimization algorithm to update the user interaction path in real time according to the state and action. 5.The AI image recognition based zero-code form automatic generation method of claim 1, wherein, The step of generating an interactive form according to the logical relationships obtained by analysis and the optimized interaction path comprises the following steps: Convert edge relationships in a labeled dynamic graph into JavaScript rules and embed them into static elements rendered based on HTML / CSS to generate an interactive form.

Images