Code repair method and related device based on intelligent agent and multimodal information processing

Abstract

This paper provides a code repair method and related apparatus based on intelligent agents and multimodal information processing for one or more scenarios. The method is applied to an integrated development environment (IDE), including: in response to receiving a code repair instruction for a first component, obtaining a design drawing of the first component from a graphical design service; based on the first code of the first component in the IDE, presenting a rendered image of the first component in the IDE; obtaining difference information based on the design drawing and the rendered image of the first component; processing the difference information using a first model to obtain code repair information for the first component, and presenting the code repair information of the first component in the IDE. Addressing the issue of low development efficiency for front-end components, this paper provides an automated process for component code repair, improving the consistency between the design and rendering of front-end components, reducing the frequency of user switching between different software, and improving the efficiency of front-end component code repair.

Description

Technical Field

[0001] One or more of the present invention relates to a code repair method based on intelligent agents and multimodal information processing, a code repair device based on intelligent agents and multimodal information processing, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the development of front-end components, the development process usually involves collaboration between various software, tools or environments. For example, developers need to draw the design of the component in a graphic design service, write the rendering code of the component in an integrated development environment, and preview the rendering of the component in a browser to ensure that the design and the rendering are visually consistent, thus realizing the transformation from the design to the final product.

[0003] The fragmented development process described above reduces the development efficiency of front-end components and makes it difficult to efficiently verify the visual fidelity between design drafts and rendered images. Summary of the Invention

[0004] This content section is provided to briefly introduce the concepts, which will be described in detail in the subsequent detailed description section. This content section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] This paper provides at least one scenario of a code repair method based on intelligent agents and multimodal information processing, applied to an integrated development environment (IDE). The method includes: in response to receiving a code repair instruction for a first component, obtaining a design drawing of the first component from a graphical design service; presenting a rendered image of the first component in the IDE based on first code of the first component, wherein the first code is used to render the design drawing of the first component; obtaining difference information based on the design drawing and the rendered image of the first component, wherein the difference information is used to indicate the difference between the design drawing and the rendered image of the first component; processing the difference information using a first model to obtain code repair information for the first component, and presenting the code repair information of the first component in the IDE, wherein the code repair information is used to repair a first sub-code in the first code, the first sub-code being the code that generates the difference between the design drawing and the rendered image of the first component.

[0006] This document provides at least one scenario a code repair device based on intelligent agents and multimodal information processing, deployed in an integrated development environment (IDE). The device includes: a first acquisition module configured to: acquire a design drawing of the first component from a graphical design service in response to acquiring a code repair instruction for a first component; a first presentation module configured to: execute first code of the first component in the IDE and present a rendered image of the first component in the IDE, wherein the first code is used to render the design drawing of the first component; a second acquisition module configured to: acquire difference information based on the design drawing and the rendered image of the first component, wherein the difference information is used to indicate the difference between the design drawing and the rendered image of the first component; and a second presentation module configured to: process the difference information using a first model to acquire code repair information of the first component, and present the code repair information of the first component in the IDE, wherein the code repair information is used to repair a first sub-code in the first code, the first sub-code being the code that generates the difference between the design drawing and the rendered image of the first component.

[0007] At least one scenario of this document provides an electronic device comprising: at least one processor; and at least one memory including one or more computer program instructions; wherein the one or more computer program instructions are executed by the processor to perform the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this document.

[0008] At least one aspect of this paper provides a computer-readable storage medium that non-transitory stores computer-readable instructions, wherein the code repair method based on intelligent agent and multimodal information processing provided by at least one aspect of this paper is implemented when the computer-readable instructions are executed by a processor.

[0009] At least one aspect of this document provides a computer program product, including a computer program that, when executed by a processor, implements the code repair method based on intelligent agent and multimodal information processing provided in at least one aspect of this document.

[0010] In the code repair method based on intelligent agents and multimodal information processing provided in at least one scenario of this paper, during the development of front-end component code, on the one hand, an automated process for component code repair is provided. By comparing the design diagram and the rendered diagram of the front-end component, difference information is obtained. Based on the difference information, the model is used to locate and repair the code in the front-end component code that causes visual differences, thereby improving the consistency between the design and rendering of the front-end component and reducing visual differences. On the other hand, the aforementioned front-end component preview, comparison, and repair steps are all integrated within the integrated development environment (IDE). Users (e.g., developers) can preview the rendered diagram of the front-end component and repair the front-end component code in the IDE, reducing the frequency of switching between different software during the front-end component code repair process and improving the efficiency of front-end component code repair. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the various scenarios described herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0012] Figure 1 This illustration shows an application scenario diagram of a code repair method based on intelligent agents and multimodal information processing, provided in at least one of the cases described in this paper.

[0013] Figure 2 The diagram schematically illustrates a system architecture for implementing a code repair method based on agent and multimodal information processing, as provided in at least one scenario of this paper.

[0014] Figure 3 The schematic diagram illustrates a flowchart of a code repair method based on intelligent agents and multimodal information processing provided in at least one scenario of this paper;

[0015] Figure 4 The illustration shows a schematic diagram of a user interaction with an integrated development environment for at least one scenario described herein;

[0016] Figure 5 The diagram illustrates the interaction between a first agent and an integrated development environment in at least one scenario described herein.

[0017] Figure 6 This paper schematically illustrates the interaction between an integrated development environment and a service layer in at least one scenario described herein.

[0018] Figure 7 The schematic diagram illustrates the structure of a code repair device based on intelligent agents and multimodal information processing, provided in at least one of the scenarios described herein.

[0019] Figure 8 A schematic diagram of the structure of an electronic device suitable for implementing at least one of the situations described herein is shown. Detailed Implementation

[0020] One or more scenarios described herein will now be described in more detail with reference to the accompanying drawings. While some scenarios are shown in the drawings, it should be understood that this document can be implemented in various forms and should not be construed as limited to the scenarios set forth herein; rather, these scenarios are provided to provide a more thorough and complete understanding of this document. It should be understood that the accompanying drawings and scenarios are for illustrative purposes only and are not intended to limit the scope of this document.

[0021] It should be understood that the steps described in the method embodiments herein may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this document is not limited in this respect.

[0022] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one situation" means "at least one situation"; the term "another situation" means "at least one additional situation"; the term "some situations" means "at least some situations". Definitions of other terms will be given in the following description.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this article are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "one" and "more" used in this document are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between the various devices in the embodiments herein are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0027] It is understood that before using the technical solutions disclosed in each scenario in this article, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this article and their authorization should be obtained through appropriate means in accordance with relevant laws and regulations. Relevant users may include any type of rights holder, such as individuals, enterprises, or groups.

[0028] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as electronic devices, applications, servers, or storage media that perform the operation of any of the technical solutions described herein.

[0029] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0030] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation method described in this article. Other methods that comply with relevant laws and regulations may also be applied to the implementation method described in this article.

[0031] In the development of front-end components, the development process usually involves collaboration between various software, tools, or environments. For example, developers need to draw the design of the component in a graphic design service, write the rendering code of the component in an integrated development environment (IDE), and preview the rendering of the component in a browser. If the design and the rendering are visually inconsistent, the rendering code needs to be modified, and the developer also needs to go back to the IDE to modify the rendering code to achieve the transformation from the design to the final product.

[0032] The development process of the aforementioned front-end components is fragmented, with different stages such as component design, coding, previewing, and debugging scattered across different software. Developers need to constantly switch between contexts, resulting in low development efficiency. For example, after modifying rendering code in the integrated development environment, a developer needs to switch to the browser to refresh the page, view the rendered image corresponding to the modified code, and then switch to the graphic design service to view the design image and compare the visual differences between the rendered image and the design image. The overall development process is lengthy and disjointed.

[0033] Furthermore, visual fidelity verification of design drafts and rendered images—that is, reducing visual differences between them—often relies on developers performing manual visual comparisons. This method is relatively subjective, prone to errors, and time-consuming. Moreover, for visual differences identified after verification, developers need to manually investigate the code locations in the rendering code that cause these differences. For example, which CSS property or component parameter in the rendering code leads to the visual differences between the design drafts and rendered images, making it difficult to efficiently verify visual fidelity between them.

[0034] In the development of some front-end components, artificial intelligence (AI) technology is introduced to assist in development. For example, AI-based code assistants can help generate rendering code. However, AI code assistants are disconnected from the integrated development environment and the actual preview effect. The rendering images corresponding to the rendering code generated by the AI ​​code assistant are difficult to produce visual effects that match expectations, making it difficult to use AI code assistants efficiently in the development of front-end components.

[0035] To at least partially solve the aforementioned technical problem, this paper provides a code repair method based on intelligent agents and multimodal information processing in at least one scenario. This code repair method is applied to an integrated development environment (IDE). The method includes: in response to receiving a code repair instruction for a first component, obtaining a design drawing of the first component from a graphical design service; presenting a rendered image of the first component in the IDE based on first code of the first component, the first code being used to render the design drawing of the first component; obtaining difference information based on the design drawing and the rendered image of the first component, the difference information indicating the difference between the design drawing and the rendered image of the first component; processing the difference information using a first model to obtain code repair information for the first component; and presenting the code repair information in the IDE, the code repair information being used to repair a first sub-code in the first code, the first sub-code being the code that generates the difference between the design drawing and the rendered image of the first component.

[0036] Based on the code repair method based on intelligent agents and multimodal information processing provided in at least one of the embodiments described herein, at least one of the embodiments described herein also provides a code repair device, electronic device, computer-readable storage medium, and computer program product based on intelligent agents and multimodal information processing.

[0037] In the code repair method based on intelligent agents and multimodal information processing provided in at least one scenario of this paper, during the development of front-end component code, on the one hand, an automated process for component code repair is provided. By comparing the design diagram and the rendered diagram of the front-end component, difference information is obtained. Based on the difference information, the model is used to locate and repair the code in the front-end component code that causes visual differences, thereby improving the consistency between the design and rendering of the front-end component and reducing visual differences. On the other hand, the aforementioned front-end component preview, comparison, and repair steps are all integrated within the integrated development environment (IDE). Users (e.g., developers) can preview the rendered diagram of the front-end component and repair the front-end component code in the IDE, reducing the frequency of switching between different software during the front-end component code repair process and improving the efficiency of front-end component code repair.

[0038] The following detailed description, with reference to the accompanying drawings, illustrates one or more scenarios and some examples thereof.

[0039] Figure 1 The illustration shows an application scenario diagram of a code repair method based on intelligent agents and multimodal information processing provided in at least one of the cases described in this paper.

[0040] like Figure 1 As shown, the application scenario provided in this case may include user 101, integrated development environment 102, graphic design service 103, and first model 104. User 101, for example, is a developer, and can use integrated development environment 102 and graphic design service 103 using a terminal device. The terminal device can be various electronic devices that can provide interactive pages, such as smart wearable devices, smart appliances, smart cars, mobile phones, tablets, laptops, or desktop computers.

[0041] For example, the integrated development environment 102 can be an application that provides a development environment. The integrated development environment 102 may include tools such as code editors, editors, and debuggers, as well as integrated development software service suites. During the development of front-end components, user 101 can write the code for the front-end components in the integrated development environment 102, such as rendering code for rendering the front-end components.

[0042] For example, the graphic design service 103 can be a front-end component design service, such as a front-end component design tool or prototyping software. During the development of the front-end component, the user 101 can draw the design diagram of the front-end component in the graphic design service 103.

[0043] For example, the integrated development environment 102 may include a client and a server, and the graphic design service 103 may include a client and a server. The server of the integrated development environment 102 can provide support for the operation of the client of the integrated development environment 102, and the server of the graphic design service 103 can provide support for the operation of the client of the graphic design service 103. The terminal device used by the user 101 may have the client of the integrated development environment 102 and the client of the graphic design service 103 installed. That is, the user 101 can interact with the integrated development environment 102 and the graphic design service 103.

[0044] For example, the integrated development environment 102 can communicate with the graphic design service 103, such as obtaining design drawings from the graphic design service 103.

[0045] For example, the integrated development environment 102 can communicate with the first model 104. For example, the integrated development environment 102 can call the first model 104 and use the first model 104 to perform repair analysis on the code in the integrated development environment 102.

[0046] For example, the code repair methods based on agent and multimodal information processing provided in one or more scenarios of this paper can be implemented in software, hardware, firmware, or any combination thereof.

[0047] For example, the code repair methods based on intelligent agents and multimodal information processing provided in one or more scenarios of this paper are applicable to an integrated development environment 102, which can load and execute the interactive method. For example, the integrated development environment 102 may include other forms of processing units with data processing capabilities and / or instruction execution capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), neural network processing unit (NPU), or storage units. The integrated development environment 102 may also have an operating system and various types of application programming interfaces (APIs) installed, enabling the implementation of the code repair methods based on intelligent agents and multimodal information processing provided in one or more scenarios of this paper by running code or instructions.

[0048] Figure 2 The diagram illustrates a system architecture for implementing a code repair method based on agent and multimodal information processing, as provided in at least one scenario of this paper.

[0049] In one or more of the scenarios described herein, a system for implementing a code repair method based on intelligent agents and multimodal information processing may include an integrated development environment, a service layer, a first intelligent agent, and a graphical design service, for example, a graphical design service that can be understood as a service for drawing design diagrams of front-end components.

[0050] For example, an integrated development environment (IDE) can provide an extension host. Using the extension host, component code repair plugins can be deployed within the IDE. These plugins then work collaboratively to repair component code under the unified management of the extension host.

[0051] For example, an integrated development environment (IDE) (such as a component code repair plugin deployed in an IDE) may include a design-to-code (D2C) coordination unit and a component rendering unit. The design-to-code coordination unit can be understood as a service process used to detect specific files in the IDE's workspace. Changes to these specific files can be used to synchronize the execution status of component code repair tasks. For example, changes to these specific files can trigger the start of the component code repair process. The component rendering unit can be understood as an isolated preview environment embedded in the IDE. For example, the component rendering unit can be understood as a browser (web) container used to independently render the rendering images of front-end components.

[0052] For example, the service layer can communicate with the integrated development environment (IDE). The service layer can provide a series of atomic capabilities that can be invoked by the IDE (e.g., the design-to-code coordination unit in the IDE). For example, the service layer may include design drawing retrieval services, dependency services, screenshot services, comparison services, and artificial intelligence services.

[0053] For example, the design draft acquisition service can be used to retrieve the design drafts of front-end components from the graphic design service; the dependency service can be used to resolve the dependencies of the front-end component code; the screenshot service can be used to take screenshots of the rendered images of the front-end components presented in the integrated development environment (e.g., the component rendering unit in the integrated development environment); the comparison service can be used to compare the design drafts and rendered images of the front-end components, such as pixel-level comparison and structural comparison, to obtain difference information; the artificial intelligence service can be understood as a multimodal analysis service that can communicate with an artificial intelligence model. The artificial intelligence service can be used to analyze the difference information to obtain code repair information. By calling the various services in the service layer, a coherent workflow is formed to realize the component code repair process.

[0054] For example, the first agent can be understood as a system used to drive and manage the component code repair process. For instance, the first agent can receive code repair requests sent by users in the integrated development environment (IDE), and write content related to component code repair into a specific file in the IDE's workspace to initiate the component code repair process.

[0055] In this way, during the component code repair process, relevant data flows in an orderly manner among the aforementioned modules, units, and services, forming a closed-loop process from the start to the end of the front-end component code repair.

[0056] The following will combine Figures 3 to 6 This paper provides a detailed description of a code repair method based on intelligent agents and multimodal information processing for at least one scenario.

[0057] Figure 3 The illustration shows a flowchart of a code repair method based on intelligent agents and multimodal information processing, provided in at least one scenario of this paper.

[0058] like Figure 3 As shown, the code repair method based on intelligent agents and multimodal information processing in this scenario includes steps S301 to S304. In some scenarios, the executing entity of the code repair method based on intelligent agents and multimodal information processing can be an integrated development environment (IDE). For example, the executing entity can be a client of the IDE (e.g., a component code repair plugin in the client of the IDE), or the executing entity can be a server of the IDE. This is not a limitation in one or more scenarios described herein. The steps included in the code repair method based on intelligent agents and multimodal information processing are explained below:

[0059] Step S301: In response to receiving a code repair instruction for the first component, obtain the design drawing of the first component from the graphic design service.

[0060] In one or more scenarios described in this article, the first component can be understood as a front-end component that has a visual fidelity verification requirement; that is, the first component has a requirement to "reduce the visual difference between the design draft and the rendered image." For example, the first component can be any front-end component whose code is written in an integrated development environment, or it can be any front-end component that draws the design draft in a graphic design service.

[0061] The code repair instruction for the first component can be understood as an instruction used to start the component code repair process for the first component. That is, the code repair instruction for the first component can be used to instruct the start of visual fidelity verification for the first component. For example, the code repair instruction for the first component can be used to instruct the start of locating the location of the part of the first code that causes visual errors and to repair the part of the first code that causes visual errors.

[0062] Graphic design services can be front-end component design services outside of the integrated development environment. For example, graphic design services can be front-end component design tools, front-end component design software, front-end component design plugins, etc.

[0063] The design draft of the first component can be understood as a graphic design sketch of the first component. For example, the design draft of the first component can describe the style of the first component in graphical form. The design draft of the first component can be drawn in a graphic design service. For example, in a graphic design service, developers can manually draw the design draft of the first component; or, for example, the graphic design service can integrate artificial intelligence technology, where developers can send the design requirements of the first component to the graphic design service, and the graphic design service can automatically draw the design draft of the first component based on the design requirements.

[0064] This article does not restrict the method of obtaining the design drawing of the first component from the graphic design service. For example, the graphic design service can provide an application programming interface (API), and the design drawing of the first component can be obtained from the graphic design service by calling the API of the graphic design service.

[0065] Step S302: Based on the first code of the first component in the integrated development environment, present the rendering diagram of the first component in the integrated development environment.

[0066] The first code of the first component corresponds to the design drawing of the first component. The first code can be used to render the design drawing of the first component. That is, the first code of the first component can be understood as the rendering code of the first component.

[0067] This article does not restrict the method of generating the first code in one or more scenarios. For example, the first code of the first component can be written manually by the developer; or, for example, the integrated development environment can also support the automatic generation of component code. The developer can send the code generation request of the first component to the integrated development environment, and the integrated development environment can automatically generate the first code of the first component based on the code generation request of the first component.

[0068] Executing the first code of the first component yields the rendered image of the first component. In other words, the rendered image of the first component can be understood as the execution result of the first code of the first component. Since the first code of the first component can be used to render the design of the first component, the rendered image of the first component can be understood as the rendering result of the design of the first component.

[0069] Presenting the rendered image of the first component in the integrated development environment (IDE) can be understood as presenting the rendered image of the first component in the interface within the IDE. That is, presenting the rendered image of the first component in the display interface provided by the IDE. By integrating the rendering image presentation function within the IDE, developers do not need to view the rendered image in a separate browser, but can view the rendered image within the IDE, reducing the frequency of switching between different software.

[0070] Step S303: Obtain the difference information based on the design drawing and the rendering drawing of the first component.

[0071] Although the first code of the first component is used to render the design of the first component, since the first code of the first component is based on the design of the first component, the first code of the first component cannot fully reflect the design of the first component. That is, there may be some differences between the rendered image of the first component and the design image of the first component. For example, there may be differences between the rendered image of the first component and the design image of the first component in terms of element size, element color, etc.

[0072] Therefore, by comparing the design drawing and the rendered image of the first component, difference information is obtained. This difference information can be used to indicate the difference between the design drawing and the rendered image of the first component. For example, the difference information can be understood as the visual difference between the design drawing and the rendered image of the first component.

[0073] Step S304: Use the first model to process the difference information to obtain the code repair information of the first component, and present the code repair information of the first component in the integrated development environment.

[0074] The code repair information for the first component can be used to repair the first sub-code in the first code. The first sub-code can be the code that causes the difference between the design drawing and the rendered drawing of the first component.

[0075] The first model can be understood as any artificial intelligence model with the ability to analyze differential information. For example, the first model can include any one or a combination of multiple large-scale models such as language models, speech models, vision models, and multimodal models. For example, the first model can be a model built on a transformer architecture, a model built on a recurrent neural network, a model built on an attention mechanism, etc. Alternatively, the first model can also be a model obtained by improving on the transformer architecture, such as a mixture of experts (MoE) model.

[0076] In other words, the first model is used to process the difference information, so that the first model can locate the first sub-code that causes the difference information in the first code based on the difference information, and provide a repair method for the first sub-code. That is, the first sub-code can be repaired by using the code repair information, so that the consistency between the rendered image corresponding to the repaired first code and the design image of the first component is enhanced, and the visual difference between the design image and the rendered image of the first component is reduced.

[0077] The code fix information for the first component is also presented in the integrated development environment (IDE). For example, it can be displayed in a designated area within the IDE's interface. By integrating this code fix information display functionality into the IDE, developers can directly view the fixes within the IDE, efficiently identifying the location in the first code that causes visual differences in the first component and how to fix it. This eliminates the need for developers to manually switch between graphic design services and browsers to compare design drafts and rendered images, and also eliminates the need to determine the location of code requiring repair. This reduces the frequency of switching between different software, improves component code fix efficiency, and enhances the consistency between the design drafts and rendered images of front-end components.

[0078] The code fix information for the first component can be presented in different forms in the integrated development environment. For example, the code fix information for the first component can be presented in the integrated development environment in the form of structured information. The code fix information for the first component may include the location information of the first sub-code (e.g., line number), the original first sub-code (i.e., the first sub-code before fix), the first sub-code after fix, and the explanation information for the fix of the first sub-code (e.g., information explaining the fix content of the first sub-code in natural language).

[0079] For example, the position information of the first sub-code can be "line": 12, the original first sub-code can be "oldCode": "padding-left: 24px;", the fixed first sub-code can be "newCode": "padding-left: 16px;", and the explanation information for the fixed first sub-code can be "explanation": "Adjust the left inner margin from 24px to 16px to match the spacing in the design".

[0080] In this way, the code fix information presented to users is no longer a vague description of code fixes, but a structured, directly applicable, and executable code fix suggestion, further reducing the workload of developers in the component code fix process.

[0081] In this way, the necessary functions in the integrated component code repair process are integrated within the integrated development environment, realizing a closed loop of the entire chain from front-end component development, front-end component preview, front-end component comparison, front-end component visual fidelity verification, front-end component repair, and front-end component repair application within the integrated development environment. This reduces the frequency of software switching in the front-end component repair process and, to some extent, solves the problems of fragmented front-end component development process, disconnect between visual verification and code repair, and difficulty in forming a closed loop with AI-assisted capabilities.

[0082] Furthermore, by comparing the design drafts and rendered images of the front-end components, the AI ​​model is used to perform correlation analysis on the differences, automatically locate the specific code location that causes the visual deviation, generate code repair suggestions that can be directly applied, realize the automatic mapping from visual differences to code repair, perform automated visual regression testing on the front-end components, enhance the consistency and quality of front-end component delivery, and prevent front-end components with visual differences from entering the production environment.

[0083] Figure 4 The illustration shows a schematic diagram of a user interaction with an integrated development environment for at least one scenario described herein.

[0084] In some cases, in response to receiving application instructions regarding code fix information, the first code is modified based on the code fix information, and the rendering graph of the first component is updated in the integrated development environment.

[0085] Application instructions for code fix information can be understood as instructions used to apply code fix information to the first code. For example, in an integrated development environment (IDE), an application control is provided, and in response to a trigger operation on the application control, an application instruction for code fix information is obtained. Or, in an IDE, application information input by the developer (e.g., information input in natural language form for applying code fix information to the first code) is received, and in response to receiving the application information, an application instruction for code fix information is obtained.

[0086] After applying the code fix information to the first code, the first code is modified. For example, when the first code is displayed in the integrated development environment (IDE), the modified first code is displayed in the IDE. At the same time, for the rendered image of the first component displayed in the IDE, after applying the code fix information to the first code, the rendered image of the first component is also updated, so that the rendered image of the first component displayed in the IDE is the rendered image obtained by executing the modified first code.

[0087] In this way, developers can not only view code fix information intuitively in the integrated development environment, but also apply the code fix information directly in the integrated development environment to achieve automatic repair of the first code of the first component. In addition, the rendering of the first component in the integrated development environment will also be hot-updated to ensure that the preview rendering in the integrated development environment corresponds to the first code, thus forming timely feedback.

[0088] like Figure 4 As shown, the interactive front-end component repair process between a user (e.g., a developer) and the integrated development environment (IDE) can be as follows: The user sends a code repair request (e.g., a code repair request for the first component) to the IDE. The IDE then presents a rendered image of the first component, allowing the user to preview it in real-time. Through comparison and AI analysis, the IDE presents code repair information for the first component and provides the user with executable repair suggestions for the first code. Finally, the user sends an application command (e.g., an application command for the code repair information) to the IDE. The IDE updates the rendered image of the first component, and after repairing the first code, synchronously provides the user with a rendered image of the first component that matches and corresponds to the current first code.

[0089] In some cases, in response to receiving a code repair instruction for the first component, the design drawing of the first component is obtained from the graphic design service, including: in response to detecting that first content is written to a first file in the integrated development environment, the design drawing of the first component is obtained from the graphic design service.

[0090] The first file can be a pre-set file in the workspace of the integrated development environment; for example, the first file can be a context.json file.

[0091] The first content can be understood as the content required in the code repair process of the first component. The first content may include task description information, first information and second information related to the code repair of the first component. The first information can be used to indicate the design diagram of the first component and the second information can be used to indicate the first code of the first component. For example, the first information may be the uniform resource locator (URL) of the design diagram of the first component and the second information may be the file path of the first code of the first component.

[0092] For example, the first content can be written to the first file by the first intelligent agent based on the code repair request sent in the integrated development environment. That is, the first intelligent agent writes the first content to the first file based on the code repair request sent in the integrated development environment.

[0093] The first intelligent agent can be understood as a system used to drive and manage the component code repair process; the code repair request can be understood as a request triggered by the developer in the integrated development environment (IDE) to initiate code repair for the first component. For example, the developer can open the first code of the first component in the IDE, and the IDE can provide repair controls. In response to the developer's triggering operation of the repair controls, the code repair request is sent to the first intelligent agent; or, for example, the developer can open the first code of the first component in the IDE, and in response to receiving the repair information input by the developer (e.g., information input in natural language form to initiate code repair for the first code), the code repair request is sent to the first intelligent agent.

[0094] For example, a manifest protocol is predefined, in which a standardized format for the first content is defined. The first intelligent agent writes the first content into a first file based on the manifest protocol, and the integrated development environment reads the first content from the first file based on the manifest protocol. The manifest protocol acts as a medium for decoupling, asynchronous communication, and state synchronization between the integrated development environment and the first intelligent agent.

[0095] In other words, the front-end component code repair process is driven by an external first intelligent agent. The integrated development environment (such as the design draft to code coordination unit in the integrated development environment) detects changes in the first file in real time. In response to the developer's code repair request, the first intelligent agent writes the first content into the first file. The integrated development system can detect the changes that have occurred in the first file and perceive the code repair needs of the first component in real time.

[0096] Thus, the code repair process of the front-end components is driven by the first intelligent agent, which is an intelligent agent capable of executing the complete front-end component development, front-end component verification and front-end component repair loop, thereby improving the degree of automation in the front-end component repair process.

[0097] Furthermore, the code repair information of the first component can be written into the first file, and the code repair decision sent by the first agent can be received.

[0098] Code repair decisions can be understood as information generated by the first agent to indicate the next decision in the front-end code development process. For example, the first agent can generate code repair decisions in response to code repair information in the first file. Code repair decisions can be used to indicate whether to apply code repair information, or to indicate the next step in the code repair process or the completion of the repair task.

[0099] In other words, after the first model generates code repair information, the code repair information is written back to the first file (e.g., context.json file), enabling the first agent to know the progress of the code repair process of the first component in a timely manner, and then make the next decision (i.e. code repair decision), forming a complete automated feedback loop.

[0100] In this way, by using the first intelligent agent to drive the overall code repair process of the front-end components in the integrated development environment, and by writing the first content, reading the first content, writing code repair information, and reading code repair information in the first file, just-in-time (JIT) synchronous interaction between the first intelligent agent and the integrated development environment is achieved. Based on the artificial intelligence capabilities of the first intelligent agent, efficient code repair is achieved in the integrated development environment.

[0101] In some cases, based on the first code of the first component in the integrated development environment (IDE), rendering the first component's image in the IDE includes: calling the browser component of the IDE to build a first sandbox environment and create a first view container; adding the first code of the first component to the file directory of the first sandbox environment; in the first sandbox environment, based on the first code of the first component, installing the dependency files associated with the first code and adding the configuration file associated with the first code; and in the first sandbox environment, based on the dependency files and configuration file, executing the first code to render the first component's image in the first view container.

[0102] A browser component (webview) can be understood as an embedded browser component provided by an integrated development environment (IDE). Using a browser component, a first view container can be rendered in the IDE, and the first view container can be used as the rendering interface of the first component to present the rendered image of the first component.

[0103] The process of rendering the first component's image in the first view container is performed in isolation within the first sandbox environment. For example, by copying the first code of the first component to a temporary file directory in the first sandbox environment, the first code of the first component is recursively parsed (e.g., recursively parsing the import statements in the first code of the first component), and the dependency files associated with the first code (e.g., npm packages) are identified. The dependency files are then installed in the first sandbox environment (e.g., by executing the nmp install command). In addition, the configuration files associated with the first code (e.g., configuration effects of the first component, including global styles, theme configurations, etc.) are identified and injected into the file directory of the first sandbox environment.

[0104] Thus, on the one hand, within the integrated development environment (IDE), a view container built using browser components serves as a real-time preview environment, integrated into the code repair process of front-end components. This allows for seamless connection between front-end component code editing and visual feedback, without requiring developers to leave the IDE. On the other hand, preparation work for rendering the first component (such as installing dependency files and adding configuration files) is performed in an isolated sandbox environment, ensuring that the rendering image of the first component can be rendered independently with the correct style, thus ensuring the accuracy of the preview.

[0105] Figure 5 The diagram illustrates the interaction between a first agent and an integrated development environment, as provided in at least one scenario of this paper.

[0106] like Figure 5 As shown, in the code repair process of the first component, the interaction process between the first agent and the integrated development environment (IDE) can be as follows: The first agent writes first content into the first file. In response to detecting that the first content has been written into the first file of the IDE, the IDE constructs a first sandbox environment, creates a first view container, and JIT-adds the first code to the file directory of the first sandbox environment. In the file directory of the first sandbox environment, it installs the dependency files associated with the first code and adds the configuration files associated with the first code. The first code is executed in the first sandbox environment to present the rendering of the first component in the first view container. In the first sandbox environment, the steps of comparing the design drawing and the rendering of the first component, obtaining difference information, and processing the difference information using the first model to obtain the code repair information of the first component are performed. The code repair information of the first component is obtained and written into the first file. In response to detecting that the code repair information has been written into the first file of the IDE, the first agent generates a code repair decision and sends the code repair decision to the IDE.

[0107] In some cases, the first model is used to process the difference information to obtain the code repair information of the first component, including: sending the design drawing of the first component, the rendering drawing of the first component, the difference information and the first code to the first model, and receiving the code repair information of the first component returned by the first model.

[0108] For example, the difference information can include difference maps and difference data. The difference map can be used to indicate the difference areas between the design map and the rendered map of the first component. For example, the difference map can be a heatmap that highlights the difference areas. The difference data can be used to describe the location information of the difference areas. For example, the difference data can be a JavaScript object notation (JSON) array. The JSON data includes the coordinate information of the difference areas, such as the vertex coordinates and the length and width information of the difference areas.

[0109] The multimodal information is input into the first model. The first model analyzes the multimodal information, finds the first sub-code in the first code that causes the difference information, and provides a way to repair the first sub-code to reduce the visual difference between the rendering of the first component and the design of the first component, thus realizing the transformation from "visual difference between design and rendering" to "location and repair of the first sub-code".

[0110] For example, the first model can generate code fix information for the first component based on prompt learning. Prompts can be used to guide the AI ​​model to make specific outputs in generative tasks (such as text generation, question answering, and dialogue tasks). By configuring prompts, the AI ​​model can understand the background and requirements of the task, enabling it to handle different types of processing tasks without retraining, thus increasing the scalability and flexibility of the AI ​​model.

[0111] For example, a first prompt word is generated and sent to the first model. The first model then receives code repair information for the first component. The first prompt word may include: the design draft of the first component, the rendered image of the first component, difference information, the first code, and the first prompt information. The first prompt information can be used to instruct the first model to analyze the difference information, understand the visual cues of visual differences between the design draft and the rendered image of the first component, and then use the visual cues to find the code fragments in the first code that cause the visual differences, such as a certain div element, a certain CSS selector, or a certain JSX property.

[0112] By sending the first prompt word to the first model, the first model can diagnose and locate the first code of the first component with the help of the prompt word's prompting capability, generate executable repair suggestions, and realize the automatic implementation of code location and code repair.

[0113] In this way, by leveraging AI capabilities, the differences between the design and rendering of front-end components can be automatically converted into code-level fix information for the front-end components, eliminating the need for developers to manually locate and fix them, thus improving the efficiency of front-end component code repair.

[0114] In some cases, in response to obtaining code repair instructions for the first component, the method further includes: generating a skill invocation sequence, which may include a first skill, a second skill, a third skill, and a fourth skill. The first skill may be used to obtain the design drawing of the first component from a graphic design service. The second skill may be used to render a rendering of the first component in the integrated development environment based on the first code of the first component in the integrated development environment. The third skill may be used to obtain difference information based on the design drawing and the rendering of the first component. The fourth skill may be used to process the difference information using a first model to obtain code repair information for the first component.

[0115] Figure 6 This paper schematically illustrates the interaction between an integrated development environment and a service layer, as provided in at least one scenario.

[0116] like Figure 6 As shown, in the code repair process of the first component, multiple services in the service layer can provide corresponding capabilities in the form of skills. The interaction process between the integrated development environment and the service layer can be as follows: In response to receiving the code repair instruction for the first component, the integrated development environment generates a skill call sequence. Then, the integrated development environment (e.g., the design draft to code coordination unit in the integrated development environment) sequentially calls the first skill, the second skill, the third skill, and the fourth skill in the service layer to obtain the design drawing of the first component from the graphic design service, to present the rendering drawing of the first component in the integrated development environment based on the first code of the first component, and to obtain the difference information and process the difference information using the first model based on the design drawing and the rendering drawing of the first component to obtain the code repair information of the first component.

[0117] In this way, the first intelligent agent drives the entire code repair process, and the integrated development environment decomposes the code repair process into specific skill call sequences. By orchestrating various predefined, atomic, and reusable skills (i.e., various services, such as code parsing, dependency installation, component rendering, visual comparison, AI analysis, etc.), the code repair process is automatically executed, the development and verification tasks of front-end components are realized, and end-to-end front-end component code repair driven by AI is achieved, integrating AI capabilities into the workflow of the integrated development environment.

[0118] Based on the code repair method based on intelligent agents and multimodal information processing provided in at least one aspect of this paper, a code repair device based on intelligent agents and multimodal information processing is also provided in at least one aspect of this paper. The following will combine... Figure 7 A detailed description is provided of a code repair device based on intelligent agents and multimodal information processing.

[0119] Figure 7 The schematic diagram illustrates the structure of a code repair device based on intelligent agents and multimodal information processing, provided in at least one of the scenarios described herein.

[0120] like Figure 7 As shown, the code repair device 700 based on intelligent agents and multimodal information processing is deployed in an integrated development environment. The code repair device 700 includes a first acquisition module 701, a first presentation module 702, a second acquisition module 703, and a second presentation module 704. For example, the first acquisition module 701, the first presentation module 702, the second acquisition module 703, and the second presentation module 704 can be implemented using hardware (e.g., circuit) modules or software modules. The following scenarios are similar and will not be elaborated further. For example, the first acquisition module 701, the first presentation module 702, the second acquisition module 703, and the second presentation module 704 can be implemented using a central processing unit (CPU), a general-purpose graphics processor (GPGPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other processing units with data processing capabilities and / or instruction execution capabilities, along with corresponding computer instructions.

[0121] The first acquisition module 701 is configured to: in response to receiving a code repair instruction for the first component, acquire the design drawing of the first component from the graphic design service. For example, the first acquisition module 701 can be configured to execute step S301 described above; its specific implementation principle can be found in the relevant description of step S301, and will not be repeated here.

[0122] The first presentation module 702 is configured to: execute the first code of the first component in the integrated development environment, and present a rendered image of the first component in the integrated development environment, wherein the first code is used to render the design drawing of the first component. For example, the first presentation module 702 can be configured to execute step S302 described above; its specific implementation principle can be found in the relevant description of step S302, and will not be repeated here.

[0123] The second acquisition module 703 is configured to: acquire difference information based on the design drawing and the rendered image of the first component, wherein the difference information is used to indicate the difference between the design drawing and the rendered image of the first component. For example, the second acquisition module 703 can be configured to execute step S303 described above; its specific implementation principle can be found in the relevant description of step S303, and will not be repeated here.

[0124] The second presentation module 704 is configured to: process the difference information using the first model, obtain code repair information for the first component, and present the code repair information for the first component in the integrated development environment. The code repair information is used to repair a first sub-code in the first code, where the first sub-code is the code that generates the difference between the design drawing and the rendered image of the first component. For example, the second presentation module 704 can be configured to execute step S304 described above. The specific implementation principle can be found in the relevant description of step S304, and will not be repeated here.

[0125] In at least one embodiment of this document, the first acquisition module 701 is further configured to: in response to detecting that first content is written into a first file of the integrated development environment, acquire the design drawing of the first component from the graphic design service, wherein the first content includes task description information, first information and second information related to code repair of the first component, the first information being used to indicate the design drawing of the first component, and the second information being used to indicate the first code of the first component; wherein the first content is written into the first file by a first agent based on a code repair request sent in the integrated development environment.

[0126] In at least one embodiment of this document, the code repair apparatus 700 based on intelligent agents and multimodal information processing further includes a communication module configured to: write code repair information of the first component into the first file; and receive a code repair decision sent by the first intelligent agent, wherein the first intelligent agent generates the code repair decision in response to the code repair information in the first file, the code repair decision indicating whether to apply the code repair information.

[0127] In at least one embodiment of this document, the first presentation module 702 is further configured to: invoke the browser component of the integrated development environment to construct a first sandbox environment and create a first view container; add the first code of the first component to the file directory of the first sandbox environment; in the first sandbox environment, based on the first code of the first component, install the dependency files associated with the first code and add the configuration file associated with the first code; in the first sandbox environment, based on the dependency files and the configuration file, execute the first code to present the rendered image of the first component in the first view container.

[0128] In at least one of the embodiments described herein, the second acquisition module 703 is further configured to: send the design drawing of the first component, the rendering drawing of the first component, the difference information and the first code to the first model, and receive the code repair information of the first component returned by the first model, wherein the difference information includes a difference map and difference data, the difference map is used to indicate the difference area between the design drawing of the first component and the rendering drawing of the first component, and the difference data is used to describe the location information of the difference area.

[0129] In at least one embodiment of this paper, the code repair device 700 based on intelligent agents and multimodal information processing further includes a generation module configured to: generate a skill invocation sequence in response to receiving a code repair instruction for a first component, wherein the skill invocation sequence includes a first skill, a second skill, a third skill, and a fourth skill; the first skill is used to obtain a design drawing of the first component from a graphic design service; the second skill is used to present a rendered image of the first component in the integrated development environment based on the first code of the first component in the integrated development environment; the third skill is used to obtain difference information based on the design drawing and the rendered image of the first component; and the fourth skill is used to process the difference information using a first model to obtain code repair information for the first component.

[0130] In at least one of the embodiments described herein, the first presentation module 702 is further configured to: in response to receiving an application instruction for the code repair information, modify the first code according to the code repair information; and update the rendering graph of the first component in the integrated development environment.

[0131] It should be noted that, for clarity and brevity, not all components of the code repair device 700 based on intelligent agents and multimodal information processing are shown in at least one instance of this document. To realize the necessary functions of the code repair device 700 based on intelligent agents and multimodal information processing, those skilled in the art can provide or set other components not shown according to specific needs, and the one or more instances described herein are not limited in this regard.

[0132] The code repair device 700 based on intelligent agent and multimodal information processing provided in at least one aspect of this paper is based on the same concept as the code repair method based on intelligent agent and multimodal information processing provided in at least one aspect of this paper. It can achieve the same technical effect and the same technical purpose as the code repair method based on intelligent agent and multimodal information processing provided in at least one aspect of this paper. For details, please refer to the relevant description above, which will not be repeated here.

[0133] This document also provides an electronic device, including a processing device and a storage device, the storage device including one or more computer program modules; wherein the one or more computer program modules are stored in the storage device and configured to be executed by the processing device, the one or more computer program modules being used to implement the code repair method based on intelligent agent and multimodal information processing provided in any of the present invention.

[0134] For example, the processing device may be a processor, such as a central processing unit (CPU), digital signal processor (DSP), image processor (GPU), general-purpose graphics processor (GPGPU), or other form of processing unit with data processing capabilities and / or instruction execution capabilities. It may be a general-purpose processor or a dedicated processor and may control other components in the electronic device to perform the desired functions.

[0135] For example, the storage device may be a memory, which may include one or more computer program products. These computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and a processing device may execute these program instructions to implement the functions described in at least one of the embodiments herein (implemented by the processing device) and / or other desired functions. Various application programs and various data may also be stored on the computer-readable storage medium, which is not limited in the embodiments described herein.

[0136] The following is for reference. Figure 8The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 800 suitable for implementing at least one of the embodiments described herein. The terminal device in at least one embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital televisions and desktop computers. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of at least one of the situations described herein.

[0137] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory 802 or a program loaded from a storage device 808 into a random access memory 803. The random access memory 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, the read-only memory 802, and the random access memory 803 are interconnected via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0138] Typically, the following devices can be connected to the input / output interface 805: input devices 806 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 808 including, for example, magnetic tape, hard disk, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0139] In particular, according to one or more embodiments herein, the processes described in the above-referenced flowcharts can be implemented as computer software programs. For example, one or more embodiments herein include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such a scenario, the computer program can be downloaded and installed from a network via communication device 809, or installed from storage device 808, or installed from read-only memory 802. When the computer program is executed by processing device 801, it performs the functions defined in the methods of at least one embodiment herein.

[0140] The electronic device 800 provided in at least one scenario of this paper is based on the same concept as the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this paper. It can achieve the same technical effect and the same technical purpose as the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this paper. For details, please refer to the relevant description above, which will not be repeated here.

[0141] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this document, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0142] The computer-readable storage medium provided in at least one of the embodiments described herein is based on the same concept as the code repair method based on intelligent agents and multimodal information processing provided in at least one of the embodiments described herein. It can achieve the same technical effect and the same technical purpose as the code repair method based on intelligent agents and multimodal information processing provided in at least one of the embodiments described herein. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0143] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as the Hypertext Transfer Protocol (HTTP), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0144] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0145] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the aforementioned code repair method based on intelligent agents and multimodal information processing.

[0146] Computer program code for performing the operations described herein may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] One or more embodiments of this document also provide a computer program product comprising one or more computer instructions. When these computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in any of these embodiments are generated.

[0148] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0149] When the computer program product is executed by a computer, the computer executes any of the aforementioned code repair methods based on intelligent agents and multimodal information processing. The computer program product can be a software installation package; when any of the aforementioned code repair methods based on intelligent agents and multimodal information processing needs to be used, the computer program product can be downloaded and executed on the computer.

[0150] The computer program product provided in at least one scenario of this paper is based on the same concept as the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this paper. It can achieve the same technical effect and the same technical purpose as the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this paper. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0151] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to the various scenarios described herein. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The units or modules described in at least one of the scenarios herein can be implemented in software or hardware. The names of the units or modules do not, in some cases, constitute a limitation on the unit or module itself.

[0153] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0154] In the context of this document, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] Based on one or more scenarios described in this paper, Example 1 provides a code repair method based on intelligent agents and multimodal information processing, applied in an integrated development environment. The method includes:

[0156] In response to receiving a code repair instruction for the first component, the design drawing of the first component is obtained from the graphic design service;

[0157] Based on the first code of the first component in the integrated development environment, a rendering of the first component is presented in the integrated development environment, wherein the first code is used to render the design drawing of the first component;

[0158] Based on the design drawing and the rendered image of the first component, difference information is obtained, wherein the difference information is used to indicate the difference between the design drawing and the rendered image of the first component;

[0159] The difference information is processed using the first model to obtain the code repair information of the first component, and the code repair information of the first component is presented in the integrated development environment. The code repair information is used to repair the first sub-code in the first code, and the first sub-code is the code that generates the difference between the design drawing and the rendering drawing of the first component.

[0160] Based on one or more scenarios described herein, Example 2 provides the response from Example 1 of receiving a code repair instruction for the first component, including retrieving the design drawing of the first component from the graphic design service, including:

[0161] In response to detecting that first content is written to the first file of the integrated development environment, the design drawing of the first component is obtained from the graphic design service, wherein the first content includes task description information, first information and second information related to code repair of the first component, the first information is used to indicate the design drawing of the first component and the second information is used to indicate the first code of the first component;

[0162] The first content is written into the first file by the first intelligent agent based on a code repair request sent in the integrated development environment.

[0163] Based on one or more scenarios described in this article, Example 3 provides the method from Example 2, and also includes:

[0164] Write the code repair information of the first component into the first file;

[0165] The first agent receives a code repair decision sent by the first agent, wherein the first agent generates the code repair decision in response to the code repair information in the first file, and the code repair decision is used to indicate whether to apply the code repair information.

[0166] According to one or more scenarios described herein, Example 4 provides the first code of the first component in Example 1 based on the integrated development environment, in which a rendered image of the first component is presented, including:

[0167] The browser component of the integrated development environment is invoked to build the first sandbox environment and create the first view container;

[0168] Add the first code of the first component to the file directory of the first sandbox environment;

[0169] In the first sandbox environment, based on the first code of the first component, the dependency files associated with the first code are installed, and the configuration files associated with the first code are added;

[0170] In the first sandbox environment, based on the dependency file and the configuration file, the first code is executed to render the rendering image of the first component in the first view container.

[0171] Based on one or more scenarios described herein, Example 5 provides the method of using the first model in Example 1 to process the difference information and obtain code repair information for the first component, including:

[0172] The design drawing of the first component, the rendered image of the first component, the difference information, and the first code are sent to the first model, and the code repair information of the first component returned by the first model is received. The difference information includes a difference map and difference data. The difference map is used to indicate the difference area between the design drawing and the rendered image of the first component, and the difference data is used to describe the location information of the difference area.

[0173] Based on one or more scenarios described herein, Example 6 provides the response to receiving code fix instructions for the first component as in Example 1, and also includes:

[0174] Generate a skill call sequence, wherein the skill call sequence includes a first skill, a second skill, a third skill, and a fourth skill;

[0175] The first skill is used to obtain the design drawing of the first component from the graphic design service;

[0176] The second skill is used to render a rendering of the first component in the integrated development environment based on the first code of the first component in the integrated development environment;

[0177] The third skill is used to obtain difference information based on the design drawing and the rendering drawing of the first component;

[0178] The fourth skill is used to process the difference information using the first model to obtain code repair information for the first component.

[0179] Depending on one or more scenarios described herein, Example 7 provides a method from any of the examples in Examples 1 through 6, and also includes:

[0180] In response to receiving an application instruction regarding the code repair information, the first code is modified according to the code repair information;

[0181] In the integrated development environment, update the rendering of the first component.

[0182] According to one or more scenarios described herein, Example 8 provides a code repair device based on intelligent agents and multimodal information processing, deployed in an integrated development environment, the device comprising:

[0183] The first acquisition module is configured to: in response to receiving a code repair instruction for the first component, acquire the design drawing of the first component from the graphic design service;

[0184] The first presentation module is configured to: execute the first code of the first component in the integrated development environment, and present the rendered image of the first component in the integrated development environment, wherein the first code is used to render the design image of the first component;

[0185] The second acquisition module is configured to: acquire difference information based on the design drawing and the rendering drawing of the first component, wherein the difference information is used to indicate the difference between the design drawing and the rendering drawing of the first component;

[0186] The second presentation module is configured to: process the difference information using the first model, obtain the code repair information of the first component, and present the code repair information of the first component in the integrated development environment, wherein the code repair information is used to repair the first sub-code in the first code, and the first sub-code is the code that generates the difference between the design drawing and the rendering drawing of the first component.

[0187] According to one or more of the provisions herein, Example 9 provides an electronic device comprising:

[0188] At least one processor; and

[0189] At least one memory, including one or more computer program instructions;

[0190] Among them, the one or more computer program instructions are executed by the processor at runtime, which is a code repair method based on intelligent agent and multimodal information processing provided in at least one of the scenarios described herein.

[0191] According to one or more of the scenarios described herein, Example 10 provides a computer-readable storage medium that non-transitory stores computer-readable instructions, wherein the code repair method based on intelligent agent and multimodal information processing provided in at least one scenario of this document is implemented when the computer-readable instructions are executed by a processor.

[0192] The above description is merely a preferred embodiment and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure herein is not limited to technical solutions formed by specific combinations of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed herein that have similar functions.

[0193] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain contexts, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of this paper. Certain features described in the context of a single case can also be implemented in combination within that single case. Conversely, various features described in the context of a single case can also be implemented individually or in any suitable sub-combination in multiple cases.

[0194] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A code repair method based on intelligent agents and multimodal information processing, applied in an integrated development environment, the method comprising: In response to receiving a code repair instruction for the first component, the design drawing of the first component is obtained from the graphic design service; Based on the first code of the first component in the integrated development environment, a rendering of the first component is presented in the integrated development environment, wherein the first code is used to render the design drawing of the first component; Based on the design drawing and the rendered image of the first component, difference information is obtained, wherein the difference information is used to indicate the difference between the design drawing and the rendered image of the first component; The difference information is processed using the first model to obtain the code repair information of the first component, and the code repair information of the first component is presented in the integrated development environment. The code repair information is used to repair the first sub-code in the first code, and the first sub-code is the code that generates the difference between the design drawing and the rendering drawing of the first component.

2. The method according to claim 1, wherein, The step of responding to receiving a code repair instruction for the first component and obtaining the design drawing of the first component from the graphic design service includes: In response to detecting that first content is written to the first file of the integrated development environment, the design drawing of the first component is obtained from the graphic design service, wherein the first content includes task description information, first information and second information related to code repair of the first component, the first information is used to indicate the design drawing of the first component and the second information is used to indicate the first code of the first component; The first content is written into the first file by the first intelligent agent based on a code repair request sent in the integrated development environment.

3. The method according to claim 2, further comprising: Write the code repair information of the first component into the first file; The first agent receives a code repair decision sent by the first agent, wherein the first agent generates the code repair decision in response to the code repair information in the first file, and the code repair decision is used to indicate whether to apply the code repair information.

4. The method according to claim 1, wherein, The step of presenting a rendered image of the first component in the integrated development environment based on the first code of the first component in the integrated development environment includes: The browser component of the integrated development environment is invoked to build the first sandbox environment and create the first view container; Add the first code of the first component to the file directory of the first sandbox environment; In the first sandbox environment, based on the first code of the first component, the dependency files associated with the first code are installed, and the configuration files associated with the first code are added; In the first sandbox environment, based on the dependency file and the configuration file, the first code is executed to render the rendering image of the first component in the first view container.

5. The method according to claim 1, wherein, The step of processing the difference information using the first model to obtain the code repair information of the first component includes: The design drawing of the first component, the rendered image of the first component, the difference information, and the first code are sent to the first model, and the code repair information of the first component returned by the first model is received. The difference information includes a difference map and difference data. The difference map is used to indicate the difference area between the design drawing and the rendered image of the first component, and the difference data is used to describe the location information of the difference area.

6. The method according to claim 1, wherein, In response to receiving code fix instructions for the first component, it also includes: Generate a skill call sequence, wherein the skill call sequence includes a first skill, a second skill, a third skill, and a fourth skill; The first skill is used to obtain the design drawing of the first component from the graphic design service; The second skill is used to render a rendering of the first component in the integrated development environment based on the first code of the first component in the integrated development environment; The third skill is used to obtain difference information based on the design drawing and the rendering drawing of the first component; The fourth skill is used to process the difference information using the first model to obtain code repair information for the first component.

7. The method according to any one of claims 1 to 6, further comprising: In response to receiving an application instruction regarding the code repair information, the first code is modified according to the code repair information; In the integrated development environment, update the rendering of the first component.

8. A code repair device based on intelligent agents and multimodal information processing, deployed in an integrated development environment, the device comprising: The first acquisition module is configured to: in response to receiving a code repair instruction for the first component, acquire the design drawing of the first component from the graphic design service; The first presentation module is configured to: execute the first code of the first component in the integrated development environment, and present the rendered image of the first component in the integrated development environment, wherein the first code is used to render the design image of the first component; The second acquisition module is configured to: acquire difference information based on the design drawing and the rendering drawing of the first component, wherein the difference information is used to indicate the difference between the design drawing and the rendering drawing of the first component; The second presentation module is configured to: process the difference information using the first model, obtain the code repair information of the first component, and present the code repair information of the first component in the integrated development environment, wherein the code repair information is used to repair the first sub-code in the first code, and the first sub-code is the code that generates the difference between the design drawing and the rendering drawing of the first component.

9. An electronic device, comprising: At least one processor; as well as At least one memory, including one or more computer program instructions; The one or more computer program instructions are executed by the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium for non-transitory storage of computer-readable instructions, wherein, The method of any one of claims 1 to 7 is implemented when the computer-readable instructions are executed by a processor.

Images