Code-free platform development method based on image segmentation

Abstract

The invention discloses a code-free platform development method based on image segmentation, and relates to the technical field of computers. The method comprises the steps that a software interface sample set is acquired, positions and types of images, texts, lines and color block elements are labeled, and an image segmentation model is trained; after model training is completed, inputting a target software interface, and performing segmentation and type identification on various elements contained in the target software interface to obtain various component lists; a picture is generated and stored for a pixel region obtained by intercepting and segmenting the image component; identifying the text content, fonts and colors of the text component through OCR (Optical Character Recognition); the length, the color and the thickness of the line assembly are obtained directly through pixel values, and the line type of the line assembly is recognized through template matching; the color of the color block assembly is recognized through calculation of a histogram; and rendering and generating a target interface in a code-free platform according to the relative position relationship of each segmented component in the original image and the attribute of each component. The software development workload can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a code-free platform development method based on image segmentation. Background Art

[0002] Currently, information system development faces numerous challenges, including duplication, redundancy, and volatile requirements. Developers often face repetitive development workloads, while even minor changes in user requirements still require modifications and maintenance. Improving developer efficiency by enabling computers to automatically develop certain software interfaces has become a valuable new research topic.

[0003] Traditional low-code platforms can improve development efficiency by dragging and dropping various components and configuring them to generate the target software interface. In some scenarios, this can be done without the involvement of developers. However, when there are too many elements in the software interface, it is necessary to frequently drag and drop various components, sort and align their positions, and set the parameters of various components. When there are image elements that do not exist in the interface, it still takes time to search and download them on the Internet, and then upload them to the information system, which undoubtedly further increases the cost of using traditional low-code platforms. Summary of the Invention

[0004] In light of this, the present invention proposes a codeless platform development method based on image segmentation. This method can truly generate software interfaces in a codeless manner, locate, segment, and identify various components within the software interface, and automatically render the software interface, further enhancing codeless generation capabilities and improving development efficiency.

[0005] A code-free platform development method based on image segmentation includes the following steps:

[0006] Step 1: Label the software interface sample set and mark the location and type of each component in the software interface;

[0007] Step 2: Train the object detection model based on the labeled software interface sample set;

[0008] Step 3: Load the trained object detection model, segment the various component elements in the target software interface, and identify the location and type information of each component element;

[0009] Step 4: Based on the type determination result, traverse each element determined to be an image component, intercept the corresponding pixels and convert them into images based on their relative position in the software interface, save them to the object storage space, and record the ID values of these images;

[0010] Step 5: Based on the type determination result, traverse each element determined to be a text component, identify the text content, font, and color through OCR, and estimate the font size based on the length of the text block and the number of characters;

[0011] Step 6: Based on the type determination result, traverse each element determined to be a line component, obtain its length, color, and thickness through pixel values, and identify its line type through template matching;

[0012] Step 7: Based on the type determination result, traverse each element determined to be a color block component and identify its color by calculating the histogram;

[0013] Step 8: Based on the relative position of each component in the original software interface and the style attributes of each component, the color block is first rendered in the codeless platform and placed at the bottom of the canvas, and then other types of components are rendered in sequence;

[0014] Step 9: Publish the rendered software interface as a URL resource for use.

[0015] Furthermore, the specific method of step 1 is:

[0016] The acquisition software interface directly cuts the images, texts, lines, and color block elements and marks the component positions and types. Each component element is cut with a rectangular frame, which is tangent to the component element in the four directions of up, down, left, and right. For line elements, the rectangular frame is separated from the long and short ends of the line by one or more pixels. For multi-line text blocks with regular text arrangement and obvious line spacing, they are marked as one text element, otherwise they are marked as multi-line text elements.

[0017] Furthermore, the specific method of step 2 is:

[0018] Use image augmentation to expand the dataset, including randomly changing the brightness, contrast, and color of the image, randomly flipping the image, and randomly disrupting the order of the ground truth boxes;

[0019] Select yolov3 network structure as the target detection model;

[0020] Establish a loss function, which consists of three parts: a loss function for whether a component is included, a loss function for describing the component's location, and a loss function for describing the component's type.

[0021] Input sample images from the sample set of the software interface, extract image features, calculate the loss function, and train with the goal of minimizing the loss function.

[0022] Furthermore, the specific method of step 3 is:

[0023] The target software interface is fed into the trained object detection model to predict the location and category of the component.

[0024] Eliminate redundant rectangles through multi-classification non-maximum suppression;

[0025] Taking the upper left corner of the software interface image as the reference origin, the component segmentation prediction results of the software interface are obtained and output in the form of a Json list.

[0026] Furthermore, the specific method of step 4 is:

[0027] Obtain the element determined to be an image component and intercept the corresponding pixel area based on the position relationship attribute;

[0028] The captured pixel area is saved as a picture and stored in the object storage space to form a picture material;

[0029] Generate a UUID for each image, and finally get a collection of images belonging to the target software interface.

[0030] Furthermore, the specific method of step 5 is:

[0031] Get the element determined to be a text component, intercept the corresponding pixel area based on the position relationship attribute, and obtain the text image;

[0032] Recognize the text content, font, and color in text images through OCR;

[0033] Estimate the font size based on the length of the text block and the number of words;

[0034] Traverse each element determined to be a text component and finally obtain the text set belonging to the target software interface.

[0035] Furthermore, the specific method of step 6 is:

[0036] Get the element determined to be a line component, intercept the corresponding pixels based on the position relationship attribute, and obtain the line image;

[0037] The longer of the width and height of the line image is used as the length of the line, and the other as the width of the line;

[0038] Get the thickness of the line based on the pixels in the line width direction;

[0039] Match the line image with various standard line type templates to identify the line type;

[0040] Each element determined to be a line component is traversed, and finally a line set belonging to the target software interface is obtained.

[0041] Furthermore, the specific method of step 7 is:

[0042] Obtain the element determined to be the color block component, intercept and obtain the corresponding pixel according to the position relationship attribute, and obtain the color block;

[0043] Determine the arrangement hierarchy between multiple color blocks based on the inclusion relationship of the color blocks;

[0044] Calculate the color histogram of each color block area and select the color that appears most frequently in the area as the color of the color block;

[0045] Traverse each element determined to be a color block component, and finally obtain a set of color blocks belonging to the target software interface.

[0046] Furthermore, the specific method of step 8 is:

[0047] Generate a canvas of corresponding size according to the pixel size of the target software interface;

[0048] With the top left corner of the canvas as the origin, first render the elements in the color block set. Place the color blocks at the bottom of the canvas and render them sequentially based on their arrangement. Then, render the image set, text set, and line set sequentially based on their position information and element attributes.

[0049] After rendering is completed, the developer modifies the configuration of each component element and adds interaction events to each component element.

[0050] Furthermore, the specific method of step 9 is:

[0051] The style and data of the rendered software interface are saved into the database and published as a URL.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention can automatically segment and identify components in a software interface, and automatically render the software interface based on the identification results, thereby reducing software development workload and improving development efficiency.

[0054] 2. Compared with traditional low-code platforms, the present invention does not require the dragging, alignment and parameter configuration of components, which significantly reduces development costs.

[0055] 3. The software interface produced by the present invention can be easily published as a URL resource and provided to other business systems for use, thereby accumulating to form a software interface resource pool, thereby changing the traditional development model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a software interface to be identified.

[0057] Figure 2 It is the component segmentation recognition result of the software interface to be recognized.

[0058] Figure 3 It is an effect diagram that is published as a URL resource after identifying and rendering the target software interface through a codeless platform. DETAILED DESCRIPTION

[0059] The present invention is described in more detail and intuitively below with reference to the accompanying drawings. It should be noted that the embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described herein are merely examples and are not intended to be exhaustive. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of this application.

[0060] A code-free platform development method based on image segmentation includes the following steps:

[0061] Label the software interface sample set, marking the location and type of each component in the software interface;

[0062] Generate training sets and test sets based on the labeled sample sets to train target detection algorithms such as YOLO;

[0063] Load the trained model to segment various component elements in the target software interface and identify the location and type information of each component element;

[0064] Based on the type determination result, traverse each element determined to be an image component, intercept the corresponding pixels based on their relative position in the software interface, convert them into images, save them to object storage, and record the ID values of these images;

[0065] Based on the type determination result, traverse each element determined to be a text component, identify the text content, font, and color through OCR, and estimate the font size based on the length of the text block and the number of words;

[0066] Based on the type determination result, traverse each element determined to be a line component, obtain its length, color, and thickness through pixel values, and identify its line type through template matching;

[0067] Based on the type determination result, traverse each element determined to be a color block component and identify its color by calculating the histogram;

[0068] Based on the relative position of each component in the original software interface and the style attributes of each component, the color block is first rendered in the codeless platform and placed at the bottom of the canvas, and then other types of components are rendered in sequence;

[0069] The rendered software interface is published as a URL resource for use by the information system.

[0070] The software interface sample set is annotated to mark the location and type of each component in the software interface, mainly including:

[0071] In line with actual business scenario requirements, we collect software interfaces that are light on interaction and heavy on presentation, and require that the collected software interfaces are invisible and distortion-free.

[0072] No need to perform any image processing operations on the software interface, the images, texts, lines and color block elements can be directly cut and the component types can be marked;

[0073] Each component element is cut using a rectangular frame to ensure that the rectangular frame is tangent to the component element in the four directions of up, down, left, and right.

[0074] Cut each image element to ensure that the rectangular frame is tangent to the image in the four directions of up, down, left, and right;

[0075] Each text element is cut to ensure that the rectangular frame is tangent to the text block in the four directions of top, bottom, left, and right. The text block is not limited to one line. If the text is arranged regularly and there is a clear gap between the lines, the multi-line text block is marked as a single text element, otherwise it is marked as a multi-line text element.

[0076] Cut each line element, and the rectangle can be separated from the long end and short end of the line by several pixels;

[0077] Cut each color block element to ensure that the rectangular frame is tangent to the color block in the four directions of up, down, left, and right;

[0078] Generate training and test sets based on the labeled sample set to train target detection algorithms such as YOLO, mainly including:

[0079] Use image augmentation methods to expand the dataset, specifically by randomly changing the image brightness, contrast, and color, randomly flipping the image, and randomly disrupting the order of the ground truth boxes;

[0080] Select the network structure, such as the yolov3 network structure;

[0081] Establish a loss function. Taking yolov3 as an example, the loss function of yolov3 consists of three parts: the loss function of whether the component is included, the loss function describing the component position, and the loss function describing the component type.

[0082] Input the software interface image, extract image features, calculate the loss function, and start training by minimizing the loss function.

[0083] Load the trained model to segment various component elements in the target software interface and identify the location and type information of each component element, mainly including:

[0084] Input the target software interface and predict the location and category of the component;

[0085] Eliminate redundant rectangles through multi-classification non-maximum suppression;

[0086] Taking the upper left corner of the software interface image as the reference origin, the component segmentation prediction result of the software interface is obtained, and the List <json>For example, the form is:

[0087] [{"type"="picture","positionX":"20","positionY":"20","width":"112","height":"67"},{"type"="line","positionX":"20","positionY":"100","width":"56","height":"6"},{"type" ="text", "positionX": "780", "positionY": "377", "width": "120", "height": "60"}, {"type"="colorblock", "positionX": "780", "positionY": "377", "width": "120", "height": "60"}, {...}, ...];

[0088] Based on the type determination result, traverse each element determined to be an image component, intercept the corresponding pixels based on their relative position in the software interface, convert them into images, save them to object storage, and record the ID values of these images, mainly including:

[0089] Get all prediction results of type=pitcure and intercept the corresponding pixel area based on the position relationship attributes positionX, positionY, width, and height;

[0090] The captured pixel area is saved as a picture in the object storage to form a picture material;

[0091] Generate a UUID for each picture, record the picture collection belonging to this software interface, and use List <json>For example, the form is:

[0092] [{"id":"a1b2c3d4e5f64789abcdef0123456789", "positionX": "20", "positionY": "20", "width": "112", "height": "67"}, {…},…].

[0093] Based on the type determination result, we traverse each element determined to be a text component, identify the text content, font, and color through OCR, and estimate the font size based on the length of the text block and the number of characters. This mainly includes:

[0094] Get all prediction results for type=text, and based on the positional attributes positionX, positionY, width, and height, intercept the corresponding pixels to obtain the text image.

[0095] Recognize the text content, font, and color in text images through OCR;

[0096] Estimate the font size based on the length of the text block and the number of words;

[0097] Traverse each element determined to be a text component and record the text collection belonging to this software interface in List <json>Form example, expressed as:

[0098] [{"positionX":"780","positionY":"377","width":"120","height":"60","textList":{"content":"Happy New Year","font":"SimHei","size":"10","color":"#000000"}},{…},{…}].

[0099] According to the type judgment result, traverse each element judged as a line component, obtain its length, color and thickness through pixel values, and identify its line type through template matching, mainly including:

[0100] Obtain all prediction results with type = line, and intercept the corresponding pixel area according to the position relationship attributes positionX, positionY, width, and height to obtain the line picture;

[0101] Take the longer one of width and height as the length length of the line;

[0102] Obtain the thickness of the line according to the horizontal pixel points of the line;

[0103] Identify the line type of the line through template matching;

[0104] Traverse each element judged as a line component, and record the line set belonging to this software interface as List <json>For example, the form is:

[0105] [{"positionX": "20", "positionY": "100", "width": "56", "height": "6", "length": "56", "size": "4", "color": "#cc0033"}, {…},…].

[0106] Based on the type determination result, traverse each element determined to be a color block component and identify its color by calculating the histogram, which mainly includes:

[0107] Get all prediction results for type=colorblock, and intercept the corresponding pixels based on the positional attributes positionX, positionY, width, and height to get the color block;

[0108] Determine the arrangement hierarchy between multiple color blocks based on the inclusion relationship of the color blocks;

[0109] Calculate the color histogram of each color block area and select the color that appears most frequently in the area as the color of the color block;

[0110] Traverse each element that is determined to be a color block component, record the color block set belonging to this software interface, and use List <json>An example of the form is:

[0111] [{"positionX": "780", "positionY": "377", "width": "120", "height": "60", "color": "#cccc66", "level": "1"}, {"p positionX": "700", "positionY": "350", "width": "300", "height": "200", "color": "#ff3366", "level": "2"}, {…},…].

[0112] Based on the relative position of each component in the original software interface and the style attributes of each component, the codeless platform first renders the color block and places it at the bottom of the canvas, and then renders other types of components in sequence, mainly including:

[0113] Generate a canvas of corresponding size on the code-free platform according to the pixel size of the original software interface;

[0114] Taking the upper left corner of the canvas as the reference origin, first render the elements in the color block set on the canvas, placing the color blocks at the bottom of the canvas. Render them in sequence according to the arrangement of the color blocks. Render them in sequence according to the position information and element attributes of the image set, text set, and line set on the canvas.

[0115] After rendering is completed, the configuration of each component element can be modified through the codeless platform, and interactive events can be added to each component element.

[0116] Publishing the rendered software interface as a URL resource for use by the information system mainly includes:

[0117] Save the style and data of the rendered software interface into the database and publish it as a URL;

[0118] You can access this software interface directly through the URL in the browser;

[0119] Other business systems can also embed this software interface into their own systems by nesting iframes.

[0120] Here's a more concrete example:

[0121] To simplify the description, we will skip the training phase of the YOLO model and go directly to the image segmentation prediction phase of the software interface.

[0122] Input the target software interface image, such as Figure 1 As shown in the figure, the software interface consists of several pictures, some texts, boundary lines and base color blocks;

[0123] right Figure 1 The component elements in the segmentation prediction are performed to obtain the model output as follows Figure 2 As shown;

[0124] Output prediction results in List <json>The form can be expressed as:

[0125] [{"type"="picture","positionX":

[0126] "20","positionY":"137","width":"66","height":"66"},{"type"="text","positionX":"151","positio nY":"147","width":"58","height":"29"},{"type"="line","positionX":"13","positionY":"109","wid th":"548","height":"7"},{"type"="colorblock","positionX":"85",

[0127] "positionY":"144","width":"195","height":"59"},{…},…];

[0128] Get all prediction results for type=picture, and extract the pixel area corresponding to each picture based on the position relationship attributes positionX, positionY, width, and height;

[0129] Generate a UUID for each image and save each image to object storage to form image materials;

[0130] Get the picture collection belonging to this software interface, with List <json>Expressed as:

[0131] [{"id"="f47ac10b952e490489f1f1709d613c75","positionX":"20","positionY":"137","width":"66","height":"66"},{…},…];

[0132] Get all prediction results for type=text, and based on the positional attributes positionX, positionY, width, and height, extract the pixel area corresponding to each text box to obtain a text image.

[0133] Identify the text content, font, color, and number of lines in text images through OCR;

[0134] Estimate the font size based on the length of the text block and the number of words;

[0135] Get the text collection belonging to this software interface, using List <json>The form is expressed as:

[0136] [{"positionX":"151","positionY":"147","width":"58","height":"29","textList":{"content":"Ne Zha","font":"SimHei","size":"26","color":"#FFFFFF"}},{…},…];

[0137] Obtain all the prediction results with type = line, and intercept the corresponding pixel area according to the position relationship attributes positionX, positionY, width, and height to obtain a line picture;

[0138] Compare each line recognition result, and select the longer one of width and height as the length length of the line;

[0139] Identify the color in the line through pixel values;

[0140] The number of pixel points in the horizontal direction of the line is recorded as the thickness of the line;

[0141] Get the line set belonging to this software interface, in the form of List <json>Expressed as:

[0142] [{"positionX":"13","positionY":"109","width":"548","height":"7","length":"548","size":"5","color":"#0168b5e"},{…},…];

[0143] Get all prediction results of type=colorblock, and intercept the corresponding pixels to get the color blocks according to the position relationship attributes positionX, positionY, width, and height;

[0144] Determine whether there is an inclusion relationship between color blocks based on the position relationship attributes of the color blocks, and then determine the arrangement hierarchy of the color blocks

[0145] Calculate the color histogram of each color block and select the color that appears most frequently as the color of this color block;

[0146] Traverse each element that is determined to be a color block component, record the color block set belonging to this software interface, and use List <json>The form is:

[0147] [{"positionX":"85","positionY":"144","width":"195","height":"59","color":"#ff4500","level":"1"},{…},…];

[0148] So far, the recognition results of all elements of the target software interface have been obtained;

[0149] Generate a canvas of corresponding size on the code-free platform according to the pixel size of the original software interface;

[0150] Taking the upper left corner of the canvas as the reference origin, first render the color blocks in the recognition results, and generate color block components in sequence according to the color block hierarchy;

[0151] Render the image in the recognition result, search the image from the object storage according to the image ID, and generate image components in sequence;

[0152] Render the text in the recognition results and generate text components in sequence according to the content, font, color, and font size of the text;

[0153] Render the lines in the recognition results and generate line components in sequence according to the length, thickness, and color of the lines;

[0154] Finally, the rendering effect of the target software interface is saved in the database and published as a URL resource, which can be accessed in the browser, such as Figure 3 shown.

[0155] In short, the present invention can truly generate software interfaces in a code-free manner, locate, segment, and identify various components in the software interface, and automatically render the software interface, further enhancing the code-free generation capability and significantly improving development efficiency.< / json> < / json> < / json> < / json> < / json> < / json> < / json> < / json> < / json> < / json>

Claims

1. A codeless platform development method based on image segmentation, characterized in that: The following steps are involved: Step 1: Label the software interface sample set and mark the location and type of each component in the software interface; Step 2: Train the object detection model based on the labeled software interface sample set; Step 3: Load the trained object detection model, segment the various component elements in the target software interface, and identify the location and type information of each component element; Step 4: Based on the type determination result, traverse each element determined to be an image component, intercept the corresponding pixels and convert them into images based on their relative position in the software interface, save them to the object storage space, and record the ID values of these images; Step 5: Based on the type determination result, traverse each element determined to be a text component, identify the text content, font, and color through OCR, and estimate the font size based on the length of the text block and the number of characters; Step 6: Based on the type determination result, traverse each element determined to be a line component, obtain its length, color, and thickness through pixel values, and identify its line type through template matching; Step 7: Based on the type determination result, traverse each element determined to be a color block component and identify its color by calculating the histogram; Step 8: Based on the relative position of each component in the original software interface and the style attributes of each component, the color block is first rendered in the codeless platform and placed at the bottom of the canvas, and then other types of components are rendered in sequence; Step 9: Publish the rendered software interface as a URL resource for use.

2. The method for developing a codeless platform based on image segmentation according to claim 1, characterized in that: The specific method of step 1 is: The acquisition software interface directly cuts the images, texts, lines, and color block elements and marks the component positions and types. Each component element is cut with a rectangular frame, which is tangent to the component element in the four directions of up, down, left, and right. For line elements, the rectangular frame is separated from the long and short ends of the line by one or more pixels. For multi-line text blocks with regular text arrangement and obvious line spacing, they are marked as one text element, otherwise they are marked as multi-line text elements.

3. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 2 is: Use image augmentation to expand the dataset, including randomly changing the brightness, contrast, and color of the image, randomly flipping the image, and randomly disrupting the order of the ground truth boxes; Select yolov3 network structure as the target detection model; Establish a loss function, which consists of three parts: a loss function for whether a component is included, a loss function for describing the component's location, and a loss function for describing the component's type. Input sample images from the sample set of the software interface, extract image features, calculate the loss function, and train with the goal of minimizing the loss function.

4. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 3 is: Input the target software interface into the trained object detection model to predict the location and category of the component; Eliminate redundant rectangles through multi-classification non-maximum suppression; Taking the upper left corner of the software interface image as the reference origin, the component segmentation prediction results of the software interface are obtained and output in the form of a Json list.

5. The method for developing a codeless platform based on image segmentation according to claim 1, characterized in that: The specific method of step 4 is: Get the element determined to be an image component and intercept the corresponding pixel area based on the position relationship attribute; The captured pixel area is saved as a picture and stored in the object storage space to form a picture material; Generate a UUID for each image, and finally get a collection of images belonging to the target software interface.

6. The method for developing a codeless platform based on image segmentation according to claim 1, characterized in that: The specific method of step 5 is: Get the element determined to be a text component, intercept the corresponding pixel area based on the position relationship attribute, and obtain the text image; Recognize the text content, font, and color in text images through OCR; Estimate the font size based on the length of the text block and the number of words; Traverse each element determined to be a text component and finally obtain the text set belonging to the target software interface.

7. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 6 is: Get the element determined to be a line component, intercept the corresponding pixels based on the position relationship attribute, and obtain the line image; The longer of the width and height of the line image is used as the length of the line, and the other as the width of the line; Get the thickness of the line based on the pixels in the line width direction; Match the line image with various standard line type templates to identify the line type; Each element determined to be a line component is traversed, and finally a line set belonging to the target software interface is obtained.

8. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 7 is: Obtain the element determined to be the color block component, intercept and obtain the corresponding pixel according to the position relationship attribute, and obtain the color block; Determine the arrangement hierarchy between multiple color blocks based on the inclusion relationship of the color blocks; Calculate the color histogram of each color block area and select the color that appears most frequently in the area as the color of the color block; Traverse each element determined to be a color block component, and finally obtain a set of color blocks belonging to the target software interface.

9. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 8 is: Generate a canvas of corresponding size according to the pixel size of the target software interface; With the top left corner of the canvas as the origin, first render the elements in the color block set. Place the color blocks at the bottom of the canvas and render them sequentially based on their arrangement. Then, render the image set, text set, and line set sequentially based on their position information and element attributes. After rendering is completed, the developer modifies the configuration of each component element and adds interaction events to each component element.

10. The method for developing a codeless platform based on image segmentation according to claim 1, wherein: The specific method of step 9 is: The style and data of the rendered software interface are saved into the database and published as a URL.

Images