Low-code platform application intelligent generation system supporting multi-modal input

Abstract

The invention discloses a low-code platform application intelligent generation system supporting multi-modal input, which realizes intelligent fusion and rapid application development of sketches and voice instructions by integrating YOLOv7 image recognition, Transform voice semantic analysis and automatic code generation technologies. The core module comprises multi-modal input processing (supporting freehand sketches, voice instructions and mixed input), UI component analysis (hierarchical analysis based on a spatial relation algorithm), Vue.js code generation, RESTAPI automatic binding and MaterialDesign specification optimization, and supports responsive layout and automatic addition of form verification rules. Experiments show that compared with a traditional system, the development efficiency of the system is improved by 300%, the speech recognition accuracy rate reaches 92.7%, the code generation quality passing rate is 100%, the non-technical personnel development threshold is remarkably reduced, the development period is shortened to the minute level, butt joint of plug-in extension (such as AR input) and a mainstream cloud platform (AWS / Azure / GCP) is supported, and the system is suitable for rapidly constructing enterprise-level application scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and software development, and particularly relates to a low-code platform application intelligent generation system supporting multi-modal input, and particularly applies to a scene of quickly constructing an application program through natural interaction modes such as hand-drawn sketches and voice instructions. BACKGROUND

[0002] The current mainstream low-code development platforms (such as OutSystems and Mendix) generally adopt a text input and graphical drag-and-drop interaction mode, and the core defects are as follows:

[0003] Single interaction mode: relying on keyboard input and mouse operation, and unable to directly process natural interaction forms such as hand-drawn sketches and voice instructions;

[0004] Insufficient semantic understanding: the UI component recognition accuracy of the existing system for sketches is less than 65% (according to the 2024 Gartner report), and voice instructions need to go through a complex translation process;

[0005] Rigid generation logic: unable to automatically map semantic information of multi-modal input to application structure (such as interface layout, form field, and data binding logic);

[0006] For example, a user needs to manually configure more than 3 component attributes to describe "a page containing a login form and a registration button" through a text description, and it takes about 8 minutes (MIT experimental data). SUMMARY

[0007] This part aims to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this part and the abstract and title of the specification of the application to avoid obscuring the purpose of this part, the abstract and the title of the specification. Such simplifications or omissions cannot be used to limit the scope of the present application.

[0008] In view of the above problems existing in the existing low-code development platform, the present application is proposed.

[0009] Therefore, the technical problem solved by the present application is to solve the problems of single interaction mode, insufficient semantic understanding and rigid generation logic of the existing low-code development platform.

[0010] To solve the above technical problems, the application provides the following technical scheme: a low-code platform application intelligent generation system supporting multi-modal input, comprising the following components: a multi-modal input module for user input collection and input preprocessing; an image recognition submodule connected with the multi-modal input module, using a YOLOv7 model to identify UI elements and analyzing component levels based on a spatial relationship algorithm; a speech recognition submodule connected with the multi-modal input module, using a Transformer-based model for semantic understanding; a front-end code generation connected with the image recognition submodule and the speech recognition submodule, generating Vue.js component code based on the recognition results; a back-end logic binding connected with the front-end code generation, automatically creating a RESTAPI interface based on the generated front-end code to complete back-end logic binding; and a code logic optimization module connected with the front-end code generation and the back-end logic binding, completing layout optimization and interaction optimization based on the generated front-end code and the bound back-end logic.

[0011] As a preferred scheme of the low-code platform application intelligent generation system supporting multi-modal input, when the multi-modal input module collects user input, the following input types are supported: visual input: hand-drawn sketches, screenshots; speech input: continuous speech instructions in Mandarin / English; and mixed input: allowing simultaneous uploading of sketches and speech descriptions.

[0012] As a preferred scheme of the low-code platform application intelligent generation system supporting multi-modal input, the input preprocessing of the multi-modal input module specifically includes: image enhancement: using OpenCV for contrast enhancement and edge detection; and speech-to-text: using a DeepSpeech-based speech recognition engine.

[0013] As a preferred scheme of the low-code platform application intelligent generation system supporting multi-modal input, when the speech recognition submodule performs semantic understanding, it further includes using a dialogue state tracker to process continuous instructions and complete context association.

[0014] As a preferred scheme of the low-code platform application intelligent generation system supporting multi-modal input, the layout optimization completed by the code logic optimization module is specifically adaptive adjustment based on MaterialDesign specifications: responsive layout: media queries automatically adapt to mobile devices / desktops; and alignment correction: using a Flexbox algorithm to adjust the spacing between components.

[0015] As a preferred scheme of the low-code platform application intelligent generation system supporting multi-modal input, the code logic optimization module dynamically generates a user guide process and automatically adds a form verification rule.

[0016] The application provides a low-code platform application intelligent generation system supporting multi-modal input.

[0017] 1. Interaction efficiency is improved.

[0018] The sketch input generation speed is improved by 300% (compared with the traditional drag-and-drop method).

[0019] The voice command recognition accuracy is 92.7% (F1-score).

[0020] 2. The development threshold is lowered.

[0021] Non-technical personnel can complete application development through "drawing + speaking".

[0022] The development cycle is shortened from several days to minutes.

[0023] 3. System intelligence level.

[0024] Different device screen sizes are automatically adapted (responsive layout).

[0025] The best practices (such as form verification rules) are intelligently recommended.

[0026] 4. Extensibility advantage.

[0027] Support plug-in extension (such as adding an AR input module).

[0028] It can be connected to mainstream cloud platforms (AWS / Azure / GCP). BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor. Among them:

[0030] Figure 1 The system module diagram of the low-code platform application intelligent generation system supporting multi-modal input provided by the application. DETAILED DESCRIPTION

[0031] In order to make the above-mentioned purposes, characteristics and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0032] Based on the problems of single interaction mode, insufficient semantic understanding and rigid generation logic existing in the existing mainstream low-code development platform, referring to Figure 1 The present application provides a low-code platform application intelligent generation system supporting multi-modal input, comprising the following components:

[0033] The multi-modal input module 100 is used for user input collection and input preprocessing.

[0034] The image recognition submodule 200 is connected with the multi-modal input module 100 in data, uses the YOLOv7 model to identify UI elements, and analyzes the component hierarchy based on a spatial relationship algorithm.

[0035] The speech recognition submodule 300 is connected with the multi-modal input module 100 in data, and adopts a Transformer-based model for semantic understanding.

[0036] The front-end code generation 400 is connected with the image recognition submodule 200 and the speech recognition submodule 300 in data, and generates Vue.js component code based on the recognition results.

[0037] The back-end logic binding 500 is connected with the front-end code generation 400 in data, automatically creates a REST API interface based on the generated front-end code, and completes the back-end logic binding.

[0038] The code logic optimization module 600 is connected with the front-end code generation 400 and the back-end logic binding 500 in data, and completes layout optimization and interaction optimization based on the generated front-end code and the bound back-end logic.

[0039] Specifically, when the multi-modal input module 100 collects user input, the following input types are supported:

[0040] Visual input: hand-drawn sketch, screenshot;

[0041] Speech input: continuous speech instructions in Mandarin / English;

[0042] Mixed input: allows uploading sketches and voice descriptions at the same time.

[0043] It should be noted that:

[0044] 1.1 User input collection technology analysis

[0045] 1.1.1 Visual Input

[0046] (1) Hand-drawn Sketch (JPEG / PNG Format)

[0047] Technical Principle:

[0048] Image Format: Support common lossless compression formats (JPEG / PNG) to ensure sketch clarity and processability.

[0049] Image Content: User interface sketches drawn by hand using tools like tablets and styluses, possibly containing UI components (buttons, forms, tables, etc.) and layout information.

[0050] Technical Challenges:

[0051] Sketches may have blurry, irregular lines, or noise, requiring image enhancement techniques to improve recognition accuracy.

[0052] Differentiating between component types (e.g., buttons, input boxes) and their relative positional relationships.

[0053] (2) Screenshot with Annotation Markers

[0054] Technical Principle:

[0055] Image Source: User interface screenshots taken by mobile devices / computers, possibly containing annotations (e.g., arrows, text explanations).

[0056] Annotation Processing:

[0057] Using OCR technology (e.g., Tesseract OCR) to extract text information from screenshots.

[0058] Identifying the positions of annotation markers (e.g., red boxes, arrows) through image segmentation algorithms (e.g., U-Net).

[0059] Application Scenarios:

[0060] Users interact through screenshots + textual explanations (e.g., "add a search box in this area").

[0061] (3) Mixed Input (Sketch + Voice Description)

[0062] Technical Principle:

[0063] Multimodal Fusion: Combining visual information from sketches with semantic information from voice instructions to improve contextual understanding.

[0064] Example Scenarios:

[0065] User uploads a sketch (contains login interface), and says "add search box in the top right corner". The system needs to analyze the layout of the sketch and the semantics of the voice instruction at the same time.

[0066] Technical implementation:

[0067] Through cross-modal attention mechanism (Cross-Modal Attention) to associate analysis of sketch and voice information.

[0068] Specifically, the multi-modal input module 100 performs input preprocessing, which specifically includes:

[0069] Image enhancement: using OpenCV for contrast enhancement and edge detection;

[0070] Speech to text: DeepSpeech-based speech recognition engine.

[0071] It should be noted that:

[0072] 1.2 Input preprocessing technology analysis

[0073] 1.2.1 Image enhancement

[0074] (1) Contrast enhancement

[0075] Technical principle:

[0076] Histogram equalization (Histogram Equalization): Enhance the global contrast of the image, suitable for sketches with insufficient overall brightness.

[0077] Code example:

[0078]

[0079] CLAHE (CLAHE): dynamically adjust the contrast for local regions, avoid over-enhancement caused by global equalization.

[0080] Code example:

[0081]

[0082]

[0083] (2) Edge detection

[0084] Technical principle:

[0085] Canny edge detection: through Gaussian filter denoising, calculate gradient amplitude and direction, finally detect the edge of the image.

[0086] Code example:

[0087] def detect_edges(image):

[0088] edges = cv2.Canny(image, threshold1=50, threshold2=150)

[0089] return edges

[0090] Sobel Operator:

[0091] Used to detect edges in an image in horizontal and vertical directions.

[0092] Use Cases:

[0093] Extracting component outlines in a sketch (e.g., rectangular boundaries of buttons).

[0094] 1.2.2 Speech-to-Text

[0095] (1) DeepSpeech Speech Recognition Engine

[0096] Technical Principle:

[0097] Model Architecture: End-to-end speech recognition model based on deep learning, using a combination of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN).

[0098] Input Processing:

[0099] Preprocessing: Convert audio signals into Mel Spectrogram, which is used as input for the model.

[0100] Feature Extraction: Use MFCC (Mel Frequency Cepstral Coefficients) or FBANK (Filter Bank Energy) to extract acoustic features.

[0101] Output Result:

[0102] Converts speech commands into text (e.g., "Add search box in the top right corner").

[0103] (2) Speech Parameter Settings

[0104] Sampling Rate: 16 kHz

[0105] Reason: 16 kHz is a common sampling rate for speech recognition, balancing sound quality and computational cost.

[0106] Encoding Format: PCM (Pulse Code Modulation)

[0107] Reason: PCM is a lossless audio format suitable for speech recognition tasks.

[0108] (3) Speech-to-Text Code Example

[0109]

[0110] 1.3 Benefits of Technology Combination

[0111] (1) Synergistic Processing of Multi-modal Input

[0112] Complementarity of Vision and Speech:

[0113] Sketch provides spatial layout information, while speech instruction supplements semantic details (e.g., "add a search box in the top right corner").

[0114] Enhance overall understanding accuracy through cross-modal fusion (e.g., attention mechanism).

[0115] (2) Optimization of Preprocessing Techniques

[0116] Combination of Image Enhancement and Edge Detection:

[0117] First enhance contrast with CLAHE, then detect edges with Canny, to improve the recognition accuracy of UI components.

[0118] High Precision of Speech Recognition:

[0119] Use the pre-trained model of DeepSpeech (e.g., deepspeech-0.9.3-models.pbmm) combined with a 16kHz sampling rate to ensure the accuracy of speech-to-text ≥92%.

[0120] 1.4 Summary of Technical Details

[0121]

[0122]

[0123] 1.5 Application Examples of Technology

[0124] Scenario: User generates login interface through sketch + speech

[0125] Input Collection:

[0126] User uploads a hand-drawn sketch (containing username and password input boxes) and says "add a search box in the top right corner."

[0127] Preprocessing:

[0128] Image Enhancement: Use CLAHE to enhance the contrast of the sketch and Canny to detect the edges of the components.

[0129] Speech-to-Text: DeepSpeech converts the speech into the text "add a search box in the top right corner."

[0130] Parsing and Generation:

[0131] The image recognition module detects existing components (username, password input box).

[0132] The voice parsing module extracts location information (top right corner) and component type (search box).

[0133] The application generation module automatically adds a search box component in the top right corner and generates the corresponding code.

[0134] Additionally, the image recognition sub-module 200 uses the YOLOv7 model to identify UI elements and parses the component hierarchy based on spatial relationship algorithms, specifically:

[0135] 2. Modal Parsing Module Technology Analysis

[0136] 2.1 Image Recognition Sub-module

[0137] 2.1.1 Component Detection Technology

[0138] (1) Improved YOLOv7 Model

[0139] Technical Principle:

[0140] YOLOv7 Architecture: Based on the backbone network of YOLOv7 (such as E-ELAN structure), optimize the small target detection capability.

[0141] Improvement Direction:

[0142] Multi-scale feature fusion: through FPN (Feature Pyramid Network) to enhance the detection ability of components of different sizes.

[0143] Adaptive Non-Maximum Suppression (NMS): dynamically adjust the NMS threshold to reduce false positives and missed detections.

[0144] Loss function optimization: introduce CIoU Loss (Complete Intersection over Union Loss) to improve the accuracy of bounding box regression.

[0145] Performance Indicators: mAP@0.5 = 0.89 (performance on UI component dataset).

[0146] Code Example:

[0147]

[0148] (2) Button Detection

[0149] Technical Implementation:

[0150] Rectangle Detection: YOLOv7 outputs bounding boxes (BBs) to locate button positions.

[0151] Text Recognition: Extract button text (e.g., "Submit") via OCR.

[0152] Key Points:

[0153] Locate text regions using TextDetection (CTPN) or EAST algorithms.

[0154] Recognize text content with an OCR engine (e.g., Tesseract 5.0).

[0155] (3) Form Detection

[0156] Technical Implementation:

[0157] Field Label Recognition: Extract form field labels (e.g., "Username," "Password") via OCR.

[0158] Structural Analysis:

[0159] Identify field types (input boxes, dropdowns, etc.) using rule engines or deep learning models (e.g., CRNN).

[0160] Example:

[0161]

[0162] (4) Table Detection

[0163] Technical Implementation:

[0164] Row and Column Structure Localization:

[0165] Use Hough Transform to detect horizontal and vertical lines of the table. Segment cells based on image segmentation (e.g., U-Net).

[0166] Header Extraction:

[0167] Identify header rows by analyzing text position (e.g., row height, font size).

[0168] Example Algorithm:

[0169]

[0170] 2.1.2 Layout Analysis Techniques

[0171] (1) Spatial Relation Network

[0172] Technical principle:

[0173] Spatial relationship modeling:

[0174] Build a spatial relationship graph based on the coordinate information of the components (such as center points, bounding boxes). Use graph neural networks (GNN) or spatial attention mechanisms to analyze the hierarchical relationship of components. Hierarchical structure generation:

[0175] Determine the parent-child component relationship through KD Tree or spatial proximity search.

[0176] Sample code:

[0177]

[0178]

[0179] Mermaid diagram analysis:

[0180] graph LR

[0181] A[Top navigation bar]-->B[Main content area]

[0182] B-->C[Login Form]

[0183] C-->D[Username input box]

[0184] C-->E[Password input box]

[0185] Hierarchical relationship:

[0186] The navigation bar (A) is the parent component of the main content area (B).

[0187] The login form (C) is a child component of the main content area.

[0188] The username input box (D) and password input box (E) are subcomponents of the login form.

[0189] 2.2 Speech Recognition Submodule

[0190] 2.2.1 Intent Parsing Technology

[0191] (1)Transformer-based model (BERT-Base)

[0192] Technical principle:

[0193] Model architecture: Based on the BERT-Base pre-trained model, fine-tuned to adapt to the UI generation task. Input processing:

[0194] The text after speech-to-text conversion is input into the BERT model to extract semantic features.

[0195] Encode text using Token Embedding and Positional Encoding.

[0196] Output result:

[0197] Generate structured intent data (e.g., in JSON format).

[0198] Example code:

[0199]

[0200]

[0201] Structured intent example:

[0202]

[0203] Field meaning:

[0204] intent: Type of operation (e.g., create, modify).

[0205] component: Target component type (e.g., button, table).

[0206] position: Position of the component in the interface (e.g., top right).

[0207] properties: Component properties (e.g., text content, click action).

[0208] 2.2.2 Context association techniques (When the speech recognition submodule 300 performs semantic understanding, it also includes processing consecutive instructions using a dialogue state tracker to complete context association)

[0209] (1) Dialogue State Tracker (DST)

[0210] Technical principle:

[0211] State maintenance: Record dialogue history and current context information (e.g., components already created).

[0212] State update: Dynamically update the state according to new instructions.

[0213] Implementation method:

[0214] Use a state machine (State Machine) or memory network (Memory Network) to manage context. Example code:

[0215]

[0216] Continuous instruction processing example:

[0217] User instruction:

[0218] "Draw a table first"

[0219] "Then add a delete button in the second row"

[0220] System processing:

[0221] Step 1: Create a table component and record it to the state.

[0222] Step 2: Parse "second row" and "delete button", locate the second row of the table and add a button component.

[0223] 2.3 Technical details summary

[0224]

[0225]

[0226] 2.4 Technical application example

[0227] Scenario: User generates a table through voice instruction

[0228] Input collection:

[0229] User says "create a table with three columns of name, age, and gender"

[0230] Speech to text:

[0231] Recognized as text by DeepSpeech: "create a table with three columns of name, age, and gender"

[0232] Intention analysis:

[0233] BERT model analysis into structured intention:

[0234]

[0235] Layout analysis:

[0236] System generates table layout according to component detection results, automatically assigns column width and row height.

[0237] Output:

[0238] Generate table component with table header and prompt user "table created, do you need to add data?"

[0239] Need to explain:

[0240] 3. Application generation module technical analysis

[0241] 3.1 Frontend Code Generation Techniques

[0242] 3.1.1 Vue.js Component Code Generation

[0243] (1) Vue.js Template Syntax

[0244] Technical Principle:

[0245] Template Syntax: Vue.js uses an HTML-based template syntax with directives like v-model and @click for data binding and event handling.

[0246] Component Structure:

[0247]

[0248] Key Points:

[0249] v-model: Two-way data binding, synchronizing input value with username data property.

[0250] @click: Binding click event, calling submit method.

[0251] (2) Dynamic Code Generation

[0252] Technical Implementation:

[0253] Code Generation Engine:

[0254] Dynamic generation of Vue templates based on parsed component information (e.g., buttons, form fields).

[0255] Using template engines (e.g., EJS, Handlebars) or string concatenation to generate code.

[0256] Example Code:

[0257]

[0258]

[0259] 3.1.2 Backend Logic Binding Techniques

[0260] (1) REST API Interface Generation

[0261] Technical Principle:

[0262] Framework Selection: Use Python web frameworks like Flask or Django to define routes with decorators.

[0263] Interface Generation Logic:

[0264] Generate corresponding backend interfaces based on frontend component types (e.g., login form).

[0265] Example code:

[0266]

[0267] Key points:

[0268] @app.route: Define HTTP paths and methods (e.g., / login, POST).

[0269] request.json: Get data from JSON request body.

[0270] authenticate: Simulate authentication logic.

[0271] (2) Automatic interface generation

[0272] Technical implementation:

[0273] Interface mapping rules:

[0274] Generate corresponding backend logic based on frontend component types (e.g., login, registration).

[0275] Use code generation tools (e.g., Swagger) to automatically generate API documentation and interface code.

[0276] Example logic:

[0277]

[0278] Furthermore, the code logic optimization module 600 completes layout optimization, specifically based on MaterialDesign specifications for adaptive adjustment:

[0279] Responsive layout: Media queries automatically adapt to mobile / tablet devices;

[0280] Alignment correction: Use Flexbox algorithm to adjust component spacing.

[0281] Furthermore, the code logic optimization module 600 completes interaction optimization, specifically including:

[0282] Dynamically generate user guide processes;

[0283] Automatically add form validation rules.

[0284] It should be noted that:

[0285] 4. Intelligent optimization module technology analysis

[0286] 4.1 Layout optimization technology

[0287] (1) Responsive Layout (Media Query)

[0288] Technical Principle:

[0289] CSS Media Query: Adapt different device screen sizes through the @media rule.

[0290] Example Code:

[0291]

[0292] Key Points:

[0293] max-width: Define the maximum screen width (e.g., 768px for mobile devices).

[0294] flex-direction: column: Mobile devices use vertical layout.

[0295] (2) Alignment Correction (Flexbox Algorithm)

[0296] Technical Principle:

[0297] Flexbox Layout: Automatically adjust component spacing and alignment through the flexible box model.

[0298] Example Code:

[0299]

[0300] Key Points:

[0301] justify-content: Control the alignment of the main axis direction (e.g., space-between).

[0302] align-items: Control the alignment of the cross-axis direction (e.g., center).

[0303] (3) Material Design Specification

[0304] Technical Implementation:

[0305] Component Library Integration: Use Vuetify (Material Design component library for Vue) or AntDesign. Style Adaptation:

[0306] Colors, fonts, spacing, etc. conform to the Material Design design specification.

[0307] Example:

[0308] <v-btn color="primary">Login

[0309] 4.2 Interaction Optimization Techniques

[0310] (1) User Onboarding Tour

[0311] Technical Principle:

[0312] Onboarding Tour: Guide users through the operation by highlighting components, pop-up prompts, etc.

[0313] Implementation:

[0314] Use third-party libraries (such as Intro.js or Vue-IntroJS).

[0315] Example Code:

[0316]

[0317]

[0318] (2) Form Validation Rules

[0319] Technical Principle:

[0320] Validation Rules: Implement field validation through regular expressions or validation libraries (such as Vuelidate). Example code:

[0321]

[0322] Key Points:

[0323] emailRegex: Define the regular expression for email format.

[0324] test: Verify if the input matches the regular rule.

[0325] Technical Details Summary

[0326]

[0327]

[0328] Technical Application Example Scenario: Generate Login Interface and Optimize Layout Front-end Code Generation:

[0329] Generate Vue component code based on the analysis result:

[0330]

[0331] Backend Interface Binding:

[0332] Automatically generate REST API:

[0333]

[0334] Layout optimization:

[0335] Adjust the spacing between components using Flexbox:

[0336]

[0337] Interaction optimization:

[0338] Add email format verification:

[0339]

[0340]

[0341] To verify the beneficial effects of the invention, the following simulation experiments are conducted: I. Experimental design framework

[0342]

[0343] II. Core module verification experiment

[0344] 1. Multi-modal input module verification

[0345] Experiment scenario: mixed input scenario (sketch + voice)

[0346] # Experimental data set

[0347]

[0348]

[0349] Performance comparison table

[0350] Input type Processing time (s) Semantic parsing accuracy Cross-modal fusion success rate Traditional single-modal input 8.2±0.5 78.3％ N / A Hybrid input 2.1±0.3 94.6％ 91.2％

[0351] 2. Image recognition module verification YOLOv7 improved model comparison experiment graph TD

[0352] A [Original YOLOv7] --> B [Basic detection]

[0353] C [Improved YOLOv7] --> D [Multi-scale detection]

[0354] E [CIoU Loss] --> F [Bounding box optimization]

[0355] G [Spatial relationship network] --> H [Hierarchical analysis]

[0356] style D fill:#f9f,stroke:#333

[0357] style F fill:#bbf,stroke:#333

[0358]

[0359] 3. Voice recognition module verification

[0360] Continuous instruction processing experiment

[0361]

[0362] Context association accuracy

[0363] Instruction sequence length 1 2 3 4 Correct parsing rate 97.2％ 93.8％ 89.5％ 85.1％

[0364] III. System integration verification

[0365] 1. Full-process generation experiment

[0366] E-commerce background management system generation case

[0367] sequenceDiagram

[0368] participant User

[0369] User->>+System: Upload product management sketch + "add batch import function"

[0370] System->>ImageModule: YOLOv7 detection (0.6s)

[0371] System->>VoiceModule: Transformer analysis (0.3s)

[0372] System->>CodeGen: Generate Vue component (1.2s)

[0373] System->>APIGen: Create REST interface (0.8s)

[0374] System->>Optimizer: Layout optimization (0.5s)

[0375] System-->>User: Complete system code package (total time 3.6s)

[0376]

[0377] 2. Quality verification

[0378] Code generation quality evaluation

[0379] IV. User Experience Verification

[0380]

[0381] Developer Test Data

[0382] Typical User Feedback

[0383] "As a product manager, now I can directly generate basic code through prototype sketches and requirement descriptions, and the efficiency of requirement review has increased by 70%" - PM of an Internet Company

[0384] V. Technical Index Comparison Table

[0385]

[0386] VI. Experimental Conclusion

[0387] Core Verification Achieved

[0388] Multi-modal input processing speed reaches 2.1±0.3s / time, 300% higher than traditional methods;

[0389] Voice command recognition F1-score reaches 92.7%, 15.2% higher than industry benchmark;

[0390] Code generation quality pass rate is 100%, specification compliance is 98.7%;

[0391] Innovative Technology Verification

[0392] Cross-modal attention mechanism improves mixed input resolution accuracy to 91.2%;

[0393] Improved YOLOv7 model achieves mAP@0.5 of 89.0% in UI component detection;

[0394] Responsive layout optimization improves mobile adaptation efficiency by 4.2 times;

[0395] Economic Verification

[0396] Development cost is reduced by 83.6% (based on 100 person-days benchmark);

[0397] Iteration cycle is shortened to 1 / 7 of traditional methods;

[0398] Scalability Verification

[0399] New AR input module takes 3.2 days (traditional method takes 14 days);

[0400] Supports AWS Lambda automatic deployment (interface time <15 minutes).

[0401] The application provides a low-code platform application intelligent generation system supporting multi-modal input, and has the following beneficial effects:

[0402] 1. Interaction efficiency is improved:

[0403] The sketch input generation speed is improved by 300% (compared with the traditional drag method);

[0404] The voice command recognition accuracy is 92.7% (F1-score);

[0405] 2. The development threshold is lowered:

[0406] Non-technical personnel can complete application development through "drawing + speaking";

[0407] The development cycle is shortened from several days to minutes;

[0408] 3. System intelligence level:

[0409] Automatically adapt to different device screen sizes (responsive layout);

[0410] Intelligently recommend best practices (such as form validation rules);

[0411] 4. Extensibility advantage:

[0412] Support plug-in extension (such as adding an AR input module);

[0413] Can be connected to mainstream cloud platforms (AWS / Azure / GCP).

[0414] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be included in the scope of the claims of the present application.

Claims

1. A low-code platform application intelligent generation system supporting multimodal input, characterized in that: Includes the following components: A multimodal input module (100) for user input collection and input preprocessing; An image recognition submodule (200) is connected to the multimodal input module (100) and uses a YOLOv7 model to identify UI elements and parse component hierarchies based on a spatial relationship algorithm; A speech recognition submodule (300) is connected to the multimodal input module (100) and uses a Transformer-based model to perform semantic understanding; Front-end code generation (400), data connection with the image recognition submodule (200) and the speech recognition submodule (300), and generation of Vue.js component code based on the recognition results; Backend logic binding (500), connected with the frontend code generation (400) data, automatically creates a REST API interface based on the generated frontend code, and completes the backend logic binding; The code logic optimization module (600) is connected to the front-end code generation (400) and the back-end logic binding (500) data, and performs layout optimization and interaction optimization based on the generated front-end code and the bound back-end logic.

2. The low-code platform application intelligent generation system supporting multimodal input according to claim 1 is characterized in that When the multimodal input module (100) collects user input, it supports the following input types: Visual input: hand-drawn sketches, screenshots; Voice input: continuous voice commands in Mandarin / English; Mixed input: allows uploading sketches and voice descriptions at the same time.

3. The low-code platform application intelligent generation system supporting multimodal input according to claim 2 is characterized in that The multimodal input module (100) performs input preprocessing specifically including: Image enhancement: contrast enhancement and edge detection using OpenCV; Speech-to-text: Based on DeepSpeech's speech recognition engine.

4. The low-code platform application intelligent generation system supporting multimodal input according to claim 3 is characterized by: The speech recognition submodule (300) also includes using a maintenance dialogue state tracker to process continuous instructions and complete context association when performing semantic understanding.

5. The low-code platform application intelligent generation system supporting multimodal input according to claim 4 is characterized in that The code logic optimization module (600) completes the layout optimization by performing adaptive adjustment based on the Material Design specification: Responsive layout: media queries automatically adapt to mobile / desktop; Alignment correction: Use the Flexbox algorithm to adjust component spacing.

6. The low-code platform application intelligent generation system supporting multimodal input according to claim 5 is characterized in that The code logic optimization module (600) performs the interactive optimization specifically including: Dynamically generate user guidance processes; Automatically add form validation rules.

Images