Intelligent software development auxiliary platform fusing multi-modal information

Abstract

The invention relates to the technical field, and discloses an intelligent software development auxiliary platform fusing multi-modal information, which comprises a multi-modal information acquisition module, a multi-modal fusion module, an interaction and auxiliary generation module, an atlas construction module, a collaborative management module and a feedback closed loop module. According to the method, information of different forms such as requirement documents, design drafts, codes and operation and maintenance data is automatically understood and associated, a unified project knowledge graph is constructed, 'information isolated island 'in a traditional development tool chain is broken, complete and traceable contexts are provided for developers, manual switching and information splicing among multiple tools are not needed, and the development efficiency is improved. And the cognitive load and the context switching cost are obviously reduced. The high-quality code can be directly generated according to the natural language description or the design manuscript, and repeated work is liberated.

Description

Technical Field

[0001] This invention relates to the field of software development technology, specifically to an intelligent software development support platform that integrates multimodal information. Background Technology

[0002] Intelligent software development refers to a set of theories, methods, and practices that utilize artificial intelligence (AI) technology to enhance, automate, and even reshape various activities throughout the software engineering lifecycle. It focuses on how to leverage AI technology to enhance the capabilities of tools such as integrated development environments (IDEs) and code editors, and involves automatic code analysis, summarization, completion, and generation, as well as using AI for bug prediction, code refactoring suggestions, and impact analysis.

[0003] The intelligent software development assistance platform that integrates multimodal information is a unified platform that uses knowledge graphs as its core, integrates, understands and associates various forms of information throughout the entire software development lifecycle through artificial intelligence technology, and provides context-aware intelligent assistance and automated services.

[0004] Traditional software development support platforms have the following problems:

[0005] 1. Information fragmentation and silos result in a severe lack of development context, with development tools operating independently and data formats not being interchangeable;

[0006] 2. The ability to develop auxiliary tools is independent, passive, and lacks intelligence, remaining limited to the code syntax level and lacking an understanding of the overall semantics of the project;

[0007] 3. The development and operation feedback chain is broken, optimization is lagging and relies on human experience. Therefore, we need to propose an intelligent software development assistance platform that integrates multimodal information. Summary of the Invention

[0008] The purpose of this invention is to provide an intelligent software development assistance platform that integrates multimodal information. Through the coordinated design of a multimodal fusion module and a graph construction module, it automatically understands and associates information in different forms such as requirement documents, design drafts, code, and operation and maintenance data, and constructs a unified project knowledge graph to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: an intelligent software development auxiliary platform integrating multimodal information, comprising:

[0010] The multimodal information acquisition module is responsible for collecting data in different formats from different data sources;

[0011] The multimodal fusion module is responsible for processing and analyzing the collected data in different forms, extracting semantics from the data in different forms, and establishing cross-modal associations;

[0012] The interaction and auxiliary generation module interacts directly with the user, transforming the understanding capabilities of the multimodal fusion module into specific development assistance functions;

[0013] The graph construction module is used to store and manage the associated networks built by the multimodal fusion module;

[0014] The collaboration management module optimizes team collaboration processes;

[0015] The feedback loop module connects development and operations to enable data-driven continuous optimization.

[0016] The feedback closed-loop module, multimodal information acquisition module, multimodal fusion module, and map construction module are connected in sequence. The interaction and auxiliary generation module and the collaborative management module are both electrically connected to the map construction module. The multimodal information acquisition module and the multimodal fusion module are both electrically connected to the interaction and auxiliary generation module. The multimodal information acquisition module is also electrically connected to the collaborative management module.

[0017] Preferably, the multimodal information acquisition module includes a multi-source adapter set, a data standardization unit, and a data routing and distribution unit connected in sequence; wherein

[0018] A multi-source adapter set is a group of clients or listeners targeting different data sources, responsible for integrating with specific tools;

[0019] Data standardization units convert raw data from different sources into a unified intermediate format within the platform;

[0020] The data routing and distribution unit sends out the standardized data.

[0021] Preferably, the multimodal fusion module includes a modality understanding unit, a unified semantic representation unit, and a cross-modal fusion unit connected in sequence; wherein

[0022] The modality understanding unit contains multiple AI models specifically designed to process data of a particular modality.

[0023] A unified semantic representation unit maps the outputs of different modal understanders to the same semantic space;

[0024] The cross-modal fusion unit receives modal vectors from the unified semantic representation unit and performs deep reasoning, association, and alignment.

[0025] Preferably, the modal understanding unit includes:

[0026] The code analyzer performs lexical and syntactic analysis, generates an abstract syntax tree, and extracts code structure, dependencies, and semantics.

[0027] The NLP engine processes natural language text, performing entity recognition, keyword extraction, sentiment analysis, and summary generation.

[0028] The CV engine parses images, performs object detection, OCR recognition of layouts and text, and understands UI components and styles.

[0029] A speech recognizer converts audio streams into text and identifies different speakers.

[0030] Preferably, the interaction and auxiliary generation module includes a context management unit, an intent recognition unit, a service scheduling unit, and a result generation and rendering unit. The intent recognition unit and the service scheduling unit are both electrically connected to the context management unit, and the intent recognition unit and the result generation and rendering unit are both electrically connected to the service scheduling unit.

[0031] Preferably, the context management unit captures and maintains the user's current development context in real time;

[0032] The intent recognition unit parses the user's commands and determines their intent.

[0033] The service scheduling unit invokes different AI services based on the identified intent;

[0034] The results generation and rendering unit presents the results returned by the AI ​​service to the user in a user-friendly format.

[0035] Preferably, the graph construction module includes a graph construction unit, a storage engine unit, and a query and calculation unit connected in sequence; wherein

[0036] The knowledge graph construction unit is responsible for receiving related information from the fusion engine and updating the knowledge graph.

[0037] The storage engine unit is for graph databases;

[0038] The query and calculation unit provides complex map query and calculation capabilities.

[0039] Preferably, the collaborative management module includes a parallel collaborative task management unit, a collaborative context enhancement unit, and a process mining and optimization unit; wherein

[0040] The task management unit automates the management of the task lifecycle;

[0041] The collaboration context enhancement unit injects relevant information at the collaboration point;

[0042] The process mining and optimization unit analyzes team collaboration data to identify bottlenecks.

[0043] Preferably, the feedback closed-loop module includes a monitoring data integration unit, a root cause analysis unit, and a feedback injection unit connected in sequence; wherein

[0044] The monitoring data integration unit continuously consumes monitoring data from the production environment;

[0045] The root cause analysis unit automatically locates the source of the problem when a fault occurs;

[0046] The feedback injection unit feeds back the operation and maintenance conclusions to the development team.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. This invention, through the combined design of a multimodal fusion module and a graph construction module, automatically understands and associates information in different forms such as requirement documents, design drafts, code, and operation and maintenance data, and constructs a unified project knowledge graph. This breaks down the "information silos" in the traditional development toolchain, providing developers with a complete and traceable context without the need to manually switch and piece together information between multiple tools, significantly reducing cognitive load and context switching costs.

[0049] 2. Through the design of the interactive and auxiliary generation module, this invention can proactively provide accurate code completion, generation, Q&A and document updates based on the understanding of the developer's current working context. This upgrades development assistance from "only based on code syntax" to "based on comprehensive project semantics". It can directly generate high-quality code based on natural language descriptions or design drafts, thus liberating developers from repetitive labor.

[0050] 3. This invention, through the design of a feedback closed-loop module, automatically links monitoring data from the production environment back to the knowledge graph of the development stage, enabling intelligent root cause analysis and precise tracing of problems. It can also automatically generate technical debt work orders or optimize code generation strategies, achieving a true data-driven DevOps closed loop. This greatly shortens the average repair time for troubleshooting and continuously optimizes the software architecture and code quality based on real operational data. Attached Figure Description

[0051] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Please see Figure 1This invention provides a technical solution: an intelligent software development auxiliary platform that integrates multimodal information, comprising:

[0054] The multimodal information acquisition module is responsible for collecting data in different forms from different data sources; it builds a complete digital twin of the project, ensuring no valuable information is missed and reducing the burden of manually organizing and entering information.

[0055] Different forms of data include:

[0056] Code: Collect code change, commit information, and comments from IDEs and version control systems (such as Git);

[0057] Documents: Collection requirements document (Word / PDF), API documentation, Wiki page, meeting minutes;

[0058] Design drafts: Collect UI / UX design drafts from tools such as Figma and Sketch, including layers, dimensions, colors, and annotation information;

[0059] Audio and video: Collect audio and video streams from meetings and review sessions, and convert them into text and perform key action recognition;

[0060] Logs and Tracing: Collect system runtime logs, performance metrics, and distributed tracing data.

[0061] The multimodal information acquisition module includes a multi-source adapter set, a data standardization unit, and a data routing and distribution unit connected in sequence; wherein...

[0062] A multi-source adapter set is a group of clients or listeners targeting different data sources, responsible for integrating with specific tools; for example, listening for code commits, obtaining design changes via API, synchronizing task information, accessing audio and video streams, and pulling logs and metrics.

[0063] The data standardization unit converts raw data from different sources into a unified intermediate format within the platform; it performs data cleaning, format conversion (such as converting XML documents to JSON), and data completion (such as adding author information to code submissions), and outputs standardized data objects.

[0064] The data routing and distribution unit sends out standardized data. It adds tags based on data type (such as code, documents, and images) and publishes it to different message queues or data streams for consumption by the downstream "understanding and fusion engine".

[0065] The multimodal fusion module is responsible for processing and analyzing the collected data in different forms, extracting semantics from the data in different forms and establishing cross-modal associations; it not only stores information, but also understands the meaning of information, automatically discovers the inherent connections between requirements, designs, code, and test cases, and forms a "knowledge graph".

[0066] Specifically, data processing includes:

[0067] Natural Language Processing: Understanding the semantics of documents, code comments, commit messages, and meeting recordings converted to text;

[0068] Code analysis: Perform syntax analysis and static analysis to understand the code structure, dependencies, and logic;

[0069] Computer vision: Analyze UI design drafts, identify components such as buttons and input boxes and their attributes, and can even generate a rough front-end code skeleton;

[0070] Multimodal large model: Use models such as GPT-4V and Gemini to achieve joint understanding across modalities. For example, connect a description in a requirements document, a screenshot of a design draft, and a piece of related code implementation to understand that they are different manifestations of the same function.

[0071] The multimodal fusion module includes a modality understanding unit, a unified semantic representation unit, and a cross-modal fusion unit connected in sequence; wherein...

[0072] The modality understanding unit contains multiple AI models specifically designed to process data of a particular modality.

[0073] The modality understanding unit includes:

[0074] The code analyzer performs lexical and syntactic analysis, generates an abstract syntax tree, and extracts code structure, dependencies, and semantics.

[0075] The NLP engine processes natural language text, performing entity recognition, keyword extraction, sentiment analysis, and summary generation.

[0076] The CV engine parses images, performs object detection, OCR recognition of layouts and text, and understands UI components and styles.

[0077] A speech recognizer converts audio streams into text and identifies different speakers.

[0078] A unified semantic representation unit maps the outputs of different modal understanders to the same semantic space; embedding technology is used to convert code structure, text concepts, visual elements, etc. into high-dimensional vectors.

[0079] The cross-modal fusion unit receives modal vectors from the unified semantic representation unit and performs deep reasoning, association, and alignment. For example, it can determine whether a piece of code implements a certain function in the requirements document, or check whether the generated front-end interface matches the design draft.

[0080] The interaction and auxiliary generation module interacts directly with users, transforming the understanding capabilities of the multimodal fusion module into specific development assistance functions; freeing developers from repetitive tasks (such as writing boilerplate code and consulting documentation), and eliminating the need for developers to frequently switch between multiple tools and contexts.

[0081] This module not only provides suggestions for completion based on the current code context, but also combines the requirements document currently being implemented and related API documentation. It can also directly generate high-quality business code or test code based on natural language descriptions (such as "create a login component with email and password input fields") or design drafts. Furthermore, developers can ask questions, and the platform can directly locate the relevant code segment, and even associate it with relevant design discussion meeting minutes. Then, based on code changes and submission information, it can automatically update the API documentation or generate version release notes.

[0082] The interaction and auxiliary generation module includes a context management unit, an intent recognition unit, a service scheduling unit, and a result generation and rendering unit. The intent recognition unit and the service scheduling unit are both electrically connected to the context management unit, and the intent recognition unit and the result generation and rendering unit are both electrically connected to the service scheduling unit.

[0083] The context management unit captures and maintains the user's current development context in real time; it obtains the file the user is editing, the cursor position, the open task card, the document being viewed, etc., to form a snapshot of the current "work scene".

[0084] The intent recognition unit parses the user's instructions and determines their intent; when the user inputs natural language (such as code completion prompts or questions), it determines whether the user wants to generate code, search for documentation, or ask about the cause of a bug.

[0085] The service scheduling unit invokes different AI services based on the identified intent; if the intent is code generation, it invokes the code generator; if the intent is question answering, it invokes the intelligent question answering engine.

[0086] The results generation and rendering unit presents the results returned by the AI ​​service to the user in a user-friendly format. This includes displaying code completion suggestions in the IDE, formatting responses in the chat interface, and visually displaying code tracing results.

[0087] The graph construction module is used to store and manage the association networks built by the multimodal fusion module; it facilitates impact analysis (such as which functions will be affected by modifying this function) and root cause analysis (such as how this bug was introduced), provides new members with the ability to get started quickly, and gives them a clear overview of the entire project structure.

[0088] This module stores "entities" (such as requirements, tasks, code files, functions, API interfaces, design components, team members) and their "relationships" (such as "implementation", "dependency", "reference", "responsible for") in the form of a graph database, achieving end-to-end traceability. For example, clicking on a requirement clearly shows its design drafts, implementation code, test cases, deployment status, and all related discussions.

[0089] The graph construction module includes a graph construction unit, a storage engine unit, and a query and calculation unit connected in sequence; wherein

[0090] The graph construction unit is responsible for receiving related information from the fusion engine and updating the knowledge graph; it also persistently stores "entities" and "relationships" as nodes and edges in the graph database. It handles add, delete, and modify operations.

[0091] The storage engine unit is for graph databases; it efficiently stores and queries nodes, edges, and their attributes.

[0092] The query and calculation unit provides complex graph query and calculation capabilities. It can perform graph traversal queries (such as "find all modules that this function directly and indirectly depends on"), path lookups (such as "find the shortest path from this requirement to that bug"), etc.

[0093] The collaboration management module optimizes team collaboration processes and project management processes, reduces manual coordination, ensures team members communicate in a unified context, and minimizes misunderstandings.

[0094] This module automatically breaks down a complex requirements document into smaller development tasks, estimates the initial workload, automatically associates the requirements background and design constraints corresponding to the code implementation during code review, automatically summarizes the meeting content, identifies to-do items, and assigns them to relevant personnel.

[0095] The collaborative management module includes a parallel collaboration task management unit, a collaboration context enhancement unit, and a process mining and optimization unit; wherein...

[0096] The task management unit automates the lifecycle management of tasks; it automatically decomposes requirements, recommends task owners, estimates working hours, and tracks task status.

[0097] The collaboration context enhancement unit injects relevant information at collaboration points; automatically displays requirement documents and test cases in the review interface; and automatically displays the latest implementation of a function when it is @mentioned in a discussion.

[0098] The process mining and optimization unit analyzes team collaboration data to identify bottlenecks. It visualizes workflows, identifies bottlenecks (such as excessively long average code review times), and proposes suggestions for process improvement.

[0099] The feedback loop module connects development and operations to enable data-driven continuous optimization; it greatly shortens MTTR (Mean Time To Repair) and provides real data support for code quality and architecture optimization.

[0100] This module feeds back error logs and performance metrics from the production environment to the platform. When an anomaly occurs in the production environment, the platform can automatically trace back to the relevant code commits, the code review comments at the time, and even the customer requirements that may be affected, quickly locating the problem and the responsible party.

[0101] The feedback closed-loop module includes a monitoring data integration unit, a root cause analysis unit, and a feedback injection unit connected in sequence; wherein...

[0102] The monitoring data integration unit continuously consumes monitoring data from the production environment; it aggregates logs, metrics, and link tracing data, and performs preliminary preprocessing and alarm noise reduction.

[0103] The root cause analysis unit automatically locates the source of the problem when a failure occurs; it correlates abnormal indicators with code changes, traces back to the specific commit, author, and related requirements through a knowledge graph, and provides a root cause probability analysis.

[0104] The feedback injection unit relays operational findings to the development team. It automatically adds comments or creates technical debt tickets in relevant code sections and provides performance data to the code generator to optimize future code generation strategies.

[0105] The feedback closed-loop module, multimodal information acquisition module, multimodal fusion module, and map construction module are connected in sequence. The interaction and auxiliary generation module and the collaborative management module are both electrically connected to the map construction module. The multimodal information acquisition module and the multimodal fusion module are both electrically connected to the interaction and auxiliary generation module. The multimodal information acquisition module is also electrically connected to the collaborative management module.

[0106] The multimodal information acquisition module continuously sends various raw data to the multimodal fusion module. The multimodal fusion module acts as an information hub, parsing the data, extracting semantics, and establishing cross-modal relationships. This "knowledge" is then persisted to the graph construction module. The interaction and auxiliary generation module and the collaborative management module are the main "knowledge consumers." When developers operate within the IDE, they query the graph construction module to obtain rich contextual information, thereby providing intelligent completion, question answering, or project insights. Developer actions (such as writing code or holding meetings) are recorded by the multimodal information acquisition module, forming a new information flow and achieving a closed loop. The feedback loop module feeds data from the production environment back to the system, enabling "development" to optimize based on the actual effects of "operations," truly achieving a DevOps closed loop.

[0107] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent software development support platform integrating multimodal information, characterized in that, include: The multimodal information acquisition module is responsible for collecting data in different formats from different data sources; The multimodal fusion module is responsible for processing and analyzing the collected data in different forms, extracting semantics from the data in different forms, and establishing cross-modal associations; The interaction and auxiliary generation module interacts directly with the user, transforming the understanding capabilities of the multimodal fusion module into specific development assistance functions; The graph construction module is used to store and manage the associated networks built by the multimodal fusion module; The collaboration management module optimizes team collaboration processes; The feedback loop module connects development and operations to enable data-driven continuous optimization. The feedback closed-loop module, multimodal information acquisition module, multimodal fusion module, and map construction module are connected in sequence. The interaction and auxiliary generation module and the collaborative management module are both electrically connected to the map construction module. The multimodal information acquisition module and the multimodal fusion module are both electrically connected to the interaction and auxiliary generation module. The multimodal information acquisition module is also electrically connected to the collaborative management module.

2. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The multimodal information acquisition module includes a multi-source adapter set, a data standardization unit, and a data routing and distribution unit connected in sequence; wherein... A multi-source adapter set is a group of clients or listeners targeting different data sources, responsible for integrating with specific tools; Data standardization units convert raw data from different sources into a unified intermediate format within the platform; The data routing and distribution unit sends out the standardized data.

3. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The multimodal fusion module includes a modality understanding unit, a unified semantic representation unit, and a cross-modal fusion unit connected in sequence; wherein... The modality understanding unit contains multiple AI models specifically designed to process data of a particular modality. A unified semantic representation unit maps the outputs of different modal understanders to the same semantic space; The cross-modal fusion unit receives modal vectors from the unified semantic representation unit and performs deep reasoning, association, and alignment.

4. The intelligent software development auxiliary platform integrating multimodal information according to claim 3, characterized in that: The modality understanding unit includes: The code analyzer performs lexical and syntactic analysis, generates an abstract syntax tree, and extracts code structure, dependencies, and semantics. The NLP engine processes natural language text, performing entity recognition, keyword extraction, sentiment analysis, and summary generation. The CV engine parses images, performs object detection, OCR recognition of layouts and text, and understands UI components and styles. A speech recognizer converts audio streams into text and identifies different speakers.

5. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The interaction and auxiliary generation module includes a context management unit, an intent recognition unit, a service scheduling unit, and a result generation and rendering unit. The intent recognition unit and the service scheduling unit are both electrically connected to the context management unit, and the intent recognition unit and the result generation and rendering unit are both electrically connected to the service scheduling unit.

6. The intelligent software development auxiliary platform integrating multimodal information according to claim 5, characterized in that: The context management unit captures and maintains the user's current development context in real time; The intent recognition unit parses the user's commands and determines their intent. The service scheduling unit invokes different AI services based on the identified intent; The results generation and rendering unit presents the results returned by the AI ​​service to the user in a user-friendly format.

7. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The graph construction module includes a graph construction unit, a storage engine unit, and a query and calculation unit connected in sequence. in The knowledge graph construction unit is responsible for receiving related information from the fusion engine and updating the knowledge graph. The storage engine unit is for graph databases; The query and calculation unit provides complex map query and calculation capabilities.

8. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The collaborative management module includes a parallel collaborative task management unit, a collaborative context enhancement unit, and a process mining and optimization unit; in The task management unit automates the management of the task lifecycle; The collaboration context enhancement unit injects relevant information at the collaboration point; The process mining and optimization unit analyzes team collaboration data to identify bottlenecks.

9. The intelligent software development auxiliary platform integrating multimodal information according to claim 1, characterized in that: The feedback closed-loop module includes a monitoring data integration unit, a root cause analysis unit, and a feedback injection unit connected in sequence. in The monitoring data integration unit continuously consumes monitoring data from the production environment; The root cause analysis unit automatically locates the source of the problem when a fault occurs; The feedback injection unit feeds back the operation and maintenance conclusions to the development team.

Images