Voice intention understanding and intelligent software cooperation method in integrated development environment
Through the automatic interaction method driven by speech-text conversion and large language model, the problems of low visual interaction efficiency and poor speech interaction accuracy in the integrated development environment are solved, efficient human-computer collaboration is achieved, and the threshold for using complex interfaces is lowered.
Patent Information
- Application Number
- CN202411171255.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-06
AI Technical Summary
In a complex integrated development environment, traditional visual interaction methods are difficult to meet the developers' rapid positioning and use of menu bar functions, resulting in long learning curves and low efficiency. Voice interaction faces the challenge of accurately capturing interface elements and voice commands, making it difficult to achieve efficient collaboration in this way .
Natural language description instructions are obtained through voice-text conversion, combined with automated testing tools to record interaction paths and coordinates, and use large language models to determine user interface components to realize automatic interaction and collaboration at the voice level, lowering the threshold for use.
It improves the accuracy of the large language model to understand user voice commands, improves the efficient collaboration between developers and integrated development environments, and simplifies the use of complex interfaces.
Smart Images

Figure CN119088385A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of software engineering, and in particular to a method for voice intention understanding and intelligent software collaboration in an integrated development environment. Background Art
[0002] Software accelerates the empowerment of traditional industries and reshapes information infrastructure and services. As a basic tool for software development, the integrated development environment (IDE) is one of the most important tools for software developers in the process of writing software. It is an application used to provide a program development environment, integrating rich text code editing, code analysis functions, compilation and debugging, version control and historical code management, etc., providing software developers with comprehensive and convenient tools and interactive experience. Generally speaking, it displays all functions in the form of a graphical interactive interface for software developers to interact. At present, the mainstream integrated software development environments include Visual Studio, Visual Studio Code, Intellij IDEA, PyCharm, etc. The current integrated software development environment supports third-party plug-in extensions, thereby enriching the functions of the integrated development environment and providing a customized development environment for software developers' development needs and preferences.
[0003] As the integrated development environment further integrates more functions and third-party plug-in extensions introduce more customized functions, the user interaction interface becomes more and more complex. For example, for the popular IntelliJ IDEA environment, its main menu bar provides first-level menus such as File, Edit, View, Insert, Format, Tools, Table, Window, and Help. Each menu provides more than ten second-level menu functions, and some second-level menus have third-level menu functions. In addition, for the navigation window, debugging window, version management window, and third-party plug-in window, each window has customized interactive functions and options.
[0004] Therefore, it often takes skilled developers to be familiar with these menu bar options, and to be able to quickly locate the menu bar, use related functions, and improve development efficiency. This positioning relies on the developer's visual understanding and experience, and has problems such as a slow learning curve, a long learning cycle for developers, easy forgetting, and insufficient understanding of functions. In the current complex integrated development environment, traditional visual interaction methods are difficult to meet actual needs. Traditional visual interaction has shortcomings such as slow speed, requiring developers to visually locate, and understand the menu bar text functions, which is difficult to meet the complex work in the integrated development environment.
[0005] Voice interaction is an effective method. Currently, voice interaction mainly converts sound into code. In order to achieve interactive collaboration of user component functions in the integrated development environment, there are two challenges that hinder their effectiveness. The first challenge is how to accurately capture the interface elements and label texts in the IDE; the second challenge is how to accurately capture the user's voice command input; the third challenge is how to accurately convert the captured user voice input into the user's intention, so as to assist in triggering the functions in the integrated development environment and achieve efficient collaboration. Summary of the invention
[0006] In order to solve the above-mentioned problems existing in the prior art, the purpose of this case is to propose a method for voice intention understanding and intelligent software collaboration in an integrated development environment, which can help software developers to conduct automatic interaction and collaboration at the voice level with complex user interface components in the integrated development environment, lower the usage threshold, and improve software development efficiency.
[0007] In order to achieve the above technical objectives, the method steps of the present scheme include: performing speech-to-text conversion on the user voice command to obtain text data, and obtaining the target natural language description instruction corresponding to the user voice command from the text data; based on the target natural language description instruction, obtaining a candidate list of user interface components; based on the text data and the candidate list of user interface components, determining the user interface component through a trained large language model; based on the interaction path and / or trigger coordinates of the determined user interface component in the integrated development environment, automatically triggering the user interface component function through a mouse click.
[0008] In the above technical solution, the user voice commands are converted into pre-stored natural language description commands, which solves the interaction difficulties caused by the differences between the user's white expression or non-standard expression and professional terms. The pre-stored natural language description commands are used to obtain the candidate list of user interface components, which can effectively narrow the scope of confirming user interface components, thereby helping to improve the accuracy of the large language model in understanding user voice commands, and further helping to improve the efficiency of efficient human-computer collaboration between developers and integrated development environments.
[0009] In one implementation of the above technical solution, the steps of obtaining a target natural language description instruction corresponding to a user voice instruction from text data include:
[0010] The obtained text data is preprocessed, and the preprocessing includes word filtering, extracting verbs and objects, and obtaining a preprocessed target natural language description instruction.
[0011] In one embodiment of the above technical solution, the interaction path of the user interface component in the integrated development environment, the acquisition step includes: dynamically loading the integrated development environment, using automated testing tools and / or written scripts to record the interaction path and / or trigger coordinates from the start of the integrated development environment to triggering the user interface component elements.
[0012] In one implementation of the above technical solution, the user interface component candidate list is obtained by retrieving a preset mapping relationship between natural language description instructions and user interface component instruction names.
[0013] In one implementation of the above technical solution, the mapping relationship between the natural language description instructions and the user interface component instruction names is augmented by a natural language pre-training model, or is expanded by using known user interface component names through text-to-speech conversion.
[0014] In one implementation of the above technical solution, the user interface component instruction name, the acquisition step includes: dynamically loading the integrated development environment, using automated testing tools and / or written scripts to perceive the user interface components, and acquiring the user interface components; identifying the text labels and / or shortcut key information of the user interface components, and using the identified text labels and / or shortcut key information of the user interface components as the user interface component instruction name.
[0015] In one implementation of the above technical solution, the trained large language model is fine-tuned using a preset mapping relationship between natural language description instructions and user interface component instruction names.
[0016] In one implementation of the above technical solution, the types of user interface components include system default components of the top-level menu bar pane, warehouse-related components of the left navigation bar, and bottom-level toolbar-related components.
[0017] In order to achieve the above technical objectives, the present solution also proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any of the above methods.
[0018] The beneficial technical effects of this case are: improving the accuracy of large language models in understanding user voice commands, and improving the efficiency of human-computer collaboration between developers and integrated development environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0020] Figure 1 , one A schematic diagram of a method flow chart in an embodiment of the present invention. DETAILED DESCRIPTION
[0021] Traditional visual interaction is slow and requires developers to visually locate and understand the functions of menu bar text, making it difficult to meet the complex tasks in an integrated development environment. Existing voice interaction has the problem of difficulty in accurately capturing interface elements and label text in the IDE, and it is also difficult to accurately capture user voice command input and convert it into user intent.
[0022] This case proposes a method for voice intent understanding and intelligent software collaboration in an integrated development environment, which helps software developers to conduct automatic interaction and collaboration at the voice level with complex user interface components in the integrated development environment, lower the usage threshold, and improve software development efficiency.
[0023] Combine the following Figure 1 , the specific implementation method of this solution is explained using the IntelliJ IDEA integrated development environment developed by JetBrains.
[0024] 1. Exploration and perception of integrated development environment user interface
[0025] Explore the user interface component elements in the integrated development environment, dynamically load the integrated development environment, perceive the user interface components through automated testing tools and / or written scripts, obtain user interface components, record triggerable user interface elements such as buttons, lists, etc., and record the interaction path and trigger coordinates from the start of the integrated development environment to the triggering of the user interface component elements. Obtain the object of the user interface component through automated testing tools and / or written scripts, identify the text label name and / or shortcut key information of the object, which is the corresponding user interface component instruction name.
[0026] The above-mentioned user interface component elements refer to objects that are displayed on the screen in a graphical way and can interact with the user. The types of user interface components that can be explored include: system default component elements in the top-level menu bar pane, such as the "Override Methods" function under the Code menu bar in IntelliJ IDEA; warehouse-related component elements in the left navigation bar, such as directory names, file names, etc.; bottom-level toolbar-related component elements, such as the "Run" and "Stop" functions in the debug window, etc.
[0027] The above-mentioned automated testing tools, such as WinAppDriver, AutoIt, etc., can be used to write automatic execution scripts.
[0028] The trigger coordinates mentioned above refer to the coordinate range where the user interface component element is finally clicked.
[0029] The above interaction path refers to the sequence of graphical interface windows and the sequence of user interface component element clicks that need to be experienced from starting the integrated development environment to clicking a user interface element (such as a button, list, etc.).
[0030] The above trigger coordinates only involve a single interface displayed at a certain moment. Some complex interfaces may involve an interaction path in a time interval, such as clicking on a window first and then performing an operation. The interaction path records the temporal relationship of the trigger coordinates.
[0031] For the same integrated development environment, the above exploration and recording of interaction paths and trigger coordinates only need to be performed at the beginning of auxiliary development in preparation for calls in subsequent development stages.
[0032] For example, use automated user interface (UI) testing tools to explore user interface functions, such as uiautomation, an automated testing tool provided by Windows for testing Windows platform applications. The implementation steps include: first start the integrated development environment, use Process.GetProcessById to obtain the running process ID, and obtain the handle of the main window through MainWindowHandle; find the button element in the main window; record the UI execution path and position of the button element, and extract the text content of the UI element through UI Automation properties; use UIAutomationEvents to automatically simulate clicking the button and trigger the button event; based on whether a new window is generated, use a depth-first search method to explore all user interface components in the integrated development environment and record all user interface component command names and interaction paths.
[0033] In addition, according to the user interface component instruction name acquired through exploration of the integrated development environment, a natural language description instruction-user interface component instruction name relationship set is constructed, and the relationship set presets a mapping relationship between the natural language description instruction and the user interface component instruction name, so that a candidate user interface component can be retrieved based on the natural language description instruction. The relationship set can be stored in a storage element, which can be hardware or software.
[0034] For the relationship set, if there are too few mapping relationships between natural language description instructions and user interface component instruction names, natural language pre-training models such as BERT, RoBERTa, LlaMa, and Palm can be used to expand the corresponding relationships. The relationship set can also be automatically generated and expanded using the known user interface component names through the text-to-speech conversion TTSMaker method.
[0035] 2. User Voice Collection and Natural Language Description Command Generation
[0036] The user's voice command is converted from speech to text to obtain text data, and the target natural language description instruction corresponding to the user's voice command is obtained from the text data.
[0037] Specifically, the user can input voice through the ambient microphone. After collecting the user's voice, the system performs voice-to-text conversion to convert the user's input voice content into a natural language text description, which is a kind of text data. Based on the obtained natural language text description, the corresponding target natural language description instruction is obtained.
[0038] One of the above methods of collecting user voice uses the system's microphone device. After the user actively clicks a function button, the machine automatically records the user's input voice.
[0039] One method of the above-mentioned speech-to-text conversion uses a speech-to-text conversion tool speech-to-text-wavenet to convert user speech into natural language text.
[0040] Since the voice input by the user may contain repetitions, modal particles, etc., the converted natural language text description is preprocessed by filtering invalid words, extracting verbs and objects, etc. For example, Python's natural language processing libraries Spacy and Benepar are used to extract root verbs and direct objects from text data to obtain the preprocessed target natural language description instructions.
[0041] 3. Intelligent development collaboration
[0042] Based on the target natural language description instructions obtained above, using the preset mapping relationship between natural language description instructions and user interface component instruction names, a rough screening is performed through retrieval to obtain a list of relevant user interface component candidates, where each item in the table is a user interface component instruction name.
[0043] Based on the text data and the obtained user interface component candidate list, the trained large language model is used to understand the user interface component calling intention of the user's voice. Example: The text corresponding to the voice input by the user this time is "The code line 114 is too long, I want to see the following content", and the obtained user interface component candidate list is 3 items:
[0044] "I need the word wrap function to be turned on" - "Menu-View-Word Wrap" "I want to perform text wrapping function" - "Menu-Editor-Enter" "The file is too long, I want to put it in one window" - "Menu-View-Word Wrap"
[0045] The text corresponding to the user's voice, "Line 114 of code is too long. I want to see the following content," and the candidate list of the above-mentioned user interface components are input into the large language model. The model determines the function to be executed and outputs the user interface component: Menu-View-Word Wrap.
[0046] Then, based on the interaction path and trigger coordinates of the user interface component "Menu-View-Word Wrap" recorded during exploration, use Autogui to perform mouse operations (such as clicking the program to execute the interaction path) to automatically call the user interface component Menu-View-Word-Wrap to achieve intelligent software collaboration for voice intent understanding. The implementation of the calling function, such as using the automated testing framework UIautomation, can automatically execute button events and call related functions.
[0047] The large language model is trained in advance, and the input of the large language model is the constructed prompt word text, and the output is the structured user interaction path. The prompt word text is a mapping relationship between a natural language description and a user interface component instruction name, such as "I need the word wrap function to be turned on"----Menu-View-Word-Wrap.
[0048] In order to make the large language model suitable for tasks related to the programming field and better understand the user interface components that the user wants to call, the large language model can be fine-tuned using the above-mentioned preset stored relationship set to optimize the large language model.
[0049] One implementation of the above-mentioned fine-tuning uses a large model fine-tuning method (P-tuning) to specifically optimize the user interface component recognition task so that the model can achieve more accurate intent understanding for the task.
[0050] In summary, the method in this case automatically explores the integrated development environment, extracts the interaction paths and command names of user interface components, collects and converts user voice into natural language description instructions, accurately locates the user interface components that users need to trigger through a large language model-driven method, and helps developers and integrated development environments to carry out efficient human-computer collaboration in a voice collaboration manner, thereby lowering the threshold for using complex integrated development environments and improving developer development efficiency.
[0051] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation method in most cases.
[0052] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.
Claims
1. A method for speech intention understanding and intelligent software collaboration in an integrated development environment, characterized in that: The method steps include: Performing voice-to-text conversion on the user's voice command to obtain text data, and obtaining the target natural language description command corresponding to the user's voice command from the text data; Based on the target natural language description instruction, obtaining a candidate list of user interface components; Determine the user interface component by using a trained large language model based on the text data and the user interface component candidate list; Based on the determined interaction path and trigger coordinates of the user interface component in the integrated development environment, the user interface component function is automatically triggered by a mouse click.
2. The method according to claim 1, characterized in that Obtaining a target natural language description instruction corresponding to a user voice instruction from text data, the steps comprising: The obtained text data is preprocessed, and the preprocessing includes word filtering, extracting verbs and objects, and obtaining a preprocessed target natural language description instruction.
3. The method according to claim 1, characterized in that: The interactive path of the user interface component in the integrated development environment, the acquisition steps include: Dynamically load the integrated development environment, and use automated testing tools and / or written scripts to record the interaction path and triggering coordinates from the startup of the integrated development environment to the triggering of the user interface component elements.
4. The method according to claim 1, characterized in that: The user interface component candidate list is obtained by retrieving a preset mapping relationship between natural language description instructions and user interface component instruction names.
5. The method according to claim 4, characterized in that The mapping relationship between the natural language description instructions and the user interface component instruction names is augmented by a natural language pre-training model, or is expanded by using known user interface component names through text-to-speech conversion.
6. The method according to claim 5, characterized in that User interface component instruction name. The acquisition steps include: Dynamically load the integrated development environment, use automated testing tools and / or written scripts to perceive user interface components, and obtain user interface components; The text label and / or shortcut key information of the user interface component is identified, and the identified text label and / or shortcut key information of the user interface component is used as the user interface component instruction name.
7. The method according to claim 5, characterized in that The trained large language model is fine-tuned using a preset mapping relationship between natural language description instructions and user interface component instruction names.
8. The method according to claim 1, characterized in that: The types of user interface components include system default components of the top-level menu bar pane, warehouse-related components of the left navigation bar, and bottom-level toolbar-related components.
9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 8.
