Interactive image processing method and device based on large language model

Abstract

The invention relates to an interactive image processing method and device based on a large language model. The method comprises the following steps: displaying an input area, an interaction area, a result display area and a plug-in area at a user side according to a preset layout; acquiring input information of a user in the input area, wherein the input information comprises one or more of the following information: text information, voice information and image information; analyzing the input information by the large language model, executing processing according to the input information, and generating output results in an image form and a character form; displaying an output result in an image form in the result display area; displaying an output result in a character form in the interaction area; and obtaining input information of a user from the plug-in area and / or the input area and / or the result display area, and performing subsequent processing on the output result in the image form according to the input information. According to the interactive image processing method and device based on the large language model, image generation and editing experience with low threshold, high flexibility and intuitive operation can be realized by combining plug-in function extension in a humanoid interaction mode.

Description

Technical Field

[0001] The present application relates to the field of computer information processing, and in particular to an interactive image processing method and device based on a large language model. Background Art

[0002] With the development of artificial intelligence technology, AI image generation applications are gradually being widely used in the field of image generation and editing. These applications generate images that meet the requirements by parsing the user's input instructions. However, the existing mainstream AI image generation applications mainly interact with users in the form of web pages or preset workflows.

[0003] Web form interaction method: Users need to enter detailed parameters by filling out the form, such as the content description, resolution, style, etc. of the generated image. Although this method is more flexible in function configuration, it requires users to have certain professional knowledge, the filling process is complicated, and the learning cost is high.

[0004] Preset workflow interaction mode: users complete the image generation operation according to fixed steps, and each step needs to be configured strictly according to the preset process. Although this method is more friendly to technology enthusiasts and enthusiasts, for ordinary users, the process is cumbersome and lacks flexibility, which can easily lead to operation failure or poor experience.

[0005] The complexity of the above traditional interaction methods makes it difficult for ordinary users to easily get started, which seriously limits the promotion and popularization of AI raw image applications among mass users. Therefore, designing an AI raw image application with low operation threshold, easy to use and more in line with human natural interaction habits has become an urgent need for current technological development.

[0006] The above information disclosed in the Background section is only for enhancement of understanding of the background of the present application and therefore it may contain information that does not constitute the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] In view of this, the present application provides an interactive image processing method and device based on a large language model, which can achieve a low-threshold, high-flexibility and intuitive operation image generation and editing experience through human-like interaction combined with plug-in function expansion.

[0008] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0009] According to one aspect of the present application, an interactive image processing method based on a large language model is proposed, the method comprising: displaying an input area, an interaction area, a result display area and a plug-in area according to a preset layout on a user side; obtaining user input information in the input area, the input information comprising one or more of the following information: text information, voice information, image information; parsing the input information by a large language model, performing processing according to the input information and generating output results in image form and text form; displaying the output results in image form in the result display area; displaying the output results in text form in the interaction area; obtaining the user's input information by the plug-in area and / or the input area and / or the result display area, and performing subsequent processing on the output results in image form according to the input information.

[0010] In an exemplary embodiment of the present application, it also includes: determining the input parameters and output parameters of the plug-in; developing the plug-in in the form of a RESTful API; registering the plug-in in a large language model; and displaying the registered plug-in in the plug-in area.

[0011] In an exemplary embodiment of the present application, obtaining the user's input information in the input area also includes: when the input information is text information, displaying the text information in the interactive area; when the input information is voice information, converting the voice information into text information to display in the interactive area.

[0012] In an exemplary embodiment of the present application, a large language model parses the input information, performs processing based on the input information and generates output results in image form and text form, including: parsing the input information to generate input instructions; generating reply content in text form as an output result based on the input instructions; and processing the input instructions through a large language model to generate an output result in image form.

[0013] In an exemplary embodiment of the present application, the input instruction is processed by a large language model to generate an output result in the form of an image, including: the large language model calls a target plug-in based on a function call technology; and the target plug-in is processed to generate an output result in the form of an image.

[0014] In an exemplary embodiment of the present application, the large language model calls the target plug-in based on the function call technology, including: the large language model performs multi-level intent analysis on the input instruction; the large language model understands the context; and dynamically determines the target plug-in according to the results of the multi-level intent analysis and context understanding.

[0015] In an exemplary embodiment of the present application, the user's input information is obtained by the plug-in area and / or the input area and / or the result display area, and the output result in image form is subsequently processed according to the input information, including: according to the user's selection, determining a target plug-in from multiple plug-ins displayed in the plug-in area; the large language model calls the target plug-in to subsequently process the output result in image form.

[0016] In an exemplary embodiment of the present application, the user's input information is obtained by the plug-in area and / or the input area and / or the result display area, and the output result in image form is subsequently processed according to the input information, including: obtaining the output result in image form selected by the user in the result display area; feeding back the selected output result in image form to the input area; obtaining the user's input information in the input area; and subsequently processing the output result in image form according to the input information.

[0017] In an exemplary embodiment of the present application, feeding back the output result in the form of the clicked image to the input area includes: feeding back the output result in the form of the clicked image to the input area in the form of a reference.

[0018] According to one aspect of the present application, an interactive image processing device based on a large language model is proposed, and the device includes: a display module, which is used to display an input area, an interaction area, a result display area, and a plug-in area according to a preset layout on the user side; an input module, which is used to obtain the user's input information in the input area, and the input information includes one or more of the following information: text information, voice information, and image information; a parsing module, which is used to parse the input information using a large language model, perform processing according to the input information, and generate output results in image form and text form; a display module, which is used to display the output result in image form in the result display area; an interaction module, which is used to display the output result in text form in the interaction area; and a processing module, which is used to obtain the user's input information from the plug-in area and / or the input area and / or the result display area, and perform subsequent processing on the output result in image form according to the input information.

[0019] According to one aspect of the present application, an electronic device is proposed, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0020] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0021] According to the interactive image processing method and device based on a large language model of the present application, an input area, an interaction area, a result display area and a plug-in area are displayed on the user side according to a preset layout; the user's input information is obtained in the input area, and the input information includes one or more of the following information: text information, voice information, image information; the large language model parses the input information, performs processing according to the input information and generates output results in image form and text form; the output results in image form are displayed in the result display area; the output results in text form are displayed in the interactive area; the user's input information is obtained by the plug-in area and / or the input area and / or the result display area, and the output results in image form are subsequently processed according to the input information. This can achieve a low-threshold, high-flexibility and intuitive operation image generation and editing experience through human-like interaction combined with plug-in function expansion.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present application will become more apparent by describing in detail the exemplary embodiments thereof with reference to the accompanying drawings. The accompanying drawings described below are only some embodiments of the present application, and it is clear to a person skilled in the art that other accompanying drawings can be obtained from these accompanying drawings without creative effort.

[0024] Figure 1 The invention is a flowchart of an interactive image processing method based on a large language model according to an exemplary embodiment.

[0025] Figure 2 It is a schematic diagram of an interactive image processing method based on a large language model according to an exemplary embodiment.

[0026] Figure 3 is a schematic diagram of an interactive image processing method based on a large language model according to another exemplary embodiment.

[0027] Figure 4 is a schematic diagram of an interactive image processing method based on a large language model according to another exemplary embodiment.

[0028] Figure 5 is a schematic diagram of an interactive image processing method based on a large language model according to another exemplary embodiment.

[0029] Figure 6 The invention is a flowchart of an interactive image processing method based on a large language model according to another exemplary embodiment.

[0030] Figure 7 It is a block diagram of an interactive image processing device based on a large language model according to an exemplary embodiment.

[0031] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0032] Fig. 9 It is a block diagram of a computer-readable medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0034] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0038] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0039] Figure 1 The flowchart of an interactive image processing method based on a large language model according to an exemplary embodiment is shown. The interactive image processing method 10 based on a large language model at least includes steps S102 to S112.

[0040] like Figure 1 As shown, in S102, the input area, the interaction area, the result display area and the plug-in area are displayed on the user side according to a preset layout.

[0041] Can be displayed as Figure 2 In the layout shown, the input area is used to provide an input box for users to input information. Users can submit multiple types of input information through the input area, including text information, voice information and image information.

[0042] The interactive area can be used to display information input by the user and output results in text form generated by the large language model, such as task feedback or operation prompts.

[0043] The result display area can be used to display the output results in the form of images generated by the large language model or plug-in. Furthermore, the result display area can arrange the output results generated each time in order for the user to confirm or click to use.

[0044] The plug-in area can display the currently available plug-in functions for users to select or directly call to expand task processing capabilities. According to the current task, the user is prompted with available plug-ins or functions. Users can add or reduce plug-ins in this area. Plug-ins are independent unit functions and can include functions such as automatic cutout, adding filters, etc.

[0045] It is worth mentioning that Figure 2 The above is only an exemplary description of the regional layout. In actual applications, the above regions can also be arranged in other layouts according to user or task requirements, and this application is not limited to this.

[0046] In S104, user input information is obtained in the input area, where the input information includes one or more of the following information: text information, voice information, and image information.

[0047] In one embodiment, it further includes: when the input information is text information, displaying the text information in the interactive area; when the input information is voice information, converting the voice information into text information to display in the interactive area.

[0048] In actual applications, when the input information is text information, the text information is directly displayed in the interactive area; when the input information is voice information, the voice information can be converted into text information by the voice recognition module and then displayed in the interactive area for users to read and confirm. Users can also upload pictures as the basic material for editing, and the pictures will be called and processed in the result display area or plug-in area.

[0049] More specifically, users can input natural language commands through the keyboard, such as "cut out", "add filter", etc. Users can also use the microphone to input voice, which the system can automatically recognize and convert into text commands. Users can upload pictures in this area as the basic material for processing.

[0050] In practical applications, in order to improve the accuracy of LLM's analysis of user input information, it is necessary to perform necessary preprocessing on the input data, which may include: (1) Natural language preprocessing Word segmentation: Split the text into key words or phrases and remove redundant content.

[0051] Entity Recognition: Identify key entities in text, such as color, style, object name, etc.

[0052] Sentiment analysis: Analyze users’ needs and intentions and infer possible editing purposes.

[0053] (2) Speech Input Preprocessing Speech Recognition: Convert speech to text, using the Google Speech API or the Speech-to-Text service.

[0054] Denoising: Eliminate background noise to improve speech recognition accuracy.

[0055] Speech segmentation: Split long speech input into smaller segments for incremental processing.

[0056] (3) Image input preprocessing Image format conversion: unify image formats (such as PNG or JPEG).

[0057] Image resizing: Optimize image resolution to suit the requirements of different processing tasks.

[0058] Image enhancement: Improve the clarity, brightness or contrast of an image.

[0059] In S106, the large language model parses the input information, performs processing according to the input information, and generates output results in the form of images and text. For example, the input information can be parsed to generate input instructions; a text reply content is generated as an output result according to the input instructions; and the input instructions are processed by the large language model to generate an output result in the form of images.

[0060] In a specific embodiment, user input information can be parsed to generate input instructions; based on the input instructions, text-based reply content is generated as an output result, such as "a picture that meets the requirements has been generated"; based on the input instructions, relevant plug-ins or functional modules are called through a large language model to generate output results in image form.

[0061] In one embodiment, it may include: the large language model calls the target plug-in based on the function call technology; and the target plug-in performs processing to generate an output result in the form of an image.

[0062] It is worth mentioning that in this application, the Function call capability refers to the capability possessed by some large models. When natural language is input into the large model, the large model has the ability to judge for itself whether to call the pre-registered API with real-time functions. If it is determined that a call is needed, it will not answer the question directly, but call the plug-in and return the result of the plug-in.

[0063] A plugin refers to a series of interfaces registered in a format agreed upon in advance by the LLM model. Usually, it is a network interface. A plugin can make up for the inability to update the LLM after training if the weights do not change. Plugins can be added and reduced dynamically. More specifically, plugin functions may include: Image generation: Generate images of specific scenes based on the description entered by the user; Image redrawing: modify existing image content, such as replacing the background or adjusting the style; Image editing: adjust brightness, contrast or remove background.

[0064] More specifically, the large language model can, for example, perform multi-level intent analysis on the input instruction; the large language model understands the context; and dynamically determines the target plug-in based on the results of the multi-level intent analysis and context understanding. The language model performs multi-level intent analysis on the input instruction and dynamically determines the target plug-in in combination with context information, for example: analyzing key task keywords in the user description (such as "redraw", "replace background"); and can also optimize the plug-in selection strategy in combination with the user's historical input information.

[0065] In specific applications, when LLM makes a plug-in call, it relies on its built-in Function Calling capability to select the appropriate API plug-in based on the user's input. To ensure the accuracy of the call, the following methods can be used: a. Multi-level intent analysis: By performing multi-level parsing of the input, it understands the user's multiple needs. For example, the user may require image style conversion and object recognition at the same time. LLM needs to be able to distinguish these different tasks and call multiple plug-ins.

[0066] b. Contextual understanding: Use the user's previous input as context to improve the accuracy of the current task. For example, if the user has uploaded an image, then the next task is likely to be image-related, and the image processing plug-in can be called first.

[0067] c. Error feedback and correction mechanism: A feedback mechanism is added to the API calling process. When the result is returned after the call, if the result does not meet the user's expectations, error correction suggestions can be provided or other plug-ins can be called again.

[0068] d. Optimize model selection: Dynamically select the most appropriate plugin based on the type of user input (natural language, voice, image) and context history. For example, for more complex image editing tasks, multiple plugins may need to be called, while for simple style transfer tasks, a dedicated style transfer plugin is called.

[0069] In S108, the output result in the form of an image is displayed in the result display area. The user can click on the result image with the mouse, and the clicked result will be automatically fed back to the input area as an input for further processing.

[0070] In S110, the output result in text form is displayed in the interactive area. The interactive area can display the instructions input by the user through text or voice. It can also display processing suggestions or feedback results of direct execution operations.

[0071] In S112, the user's input information is obtained from the plug-in area and / or the input area and / or the result display area, and the output result in the form of an image is subsequently processed according to the input information. For example, according to the user's selection, a target plug-in is determined from multiple plug-ins displayed in the plug-in area; the large language model calls the target plug-in and subsequently processes the output result in the form of an image.

[0072] In one embodiment, the user may also re-enter text or upload a picture in the input area to continue optimizing or adjusting the current result.

[0073] In one embodiment, for example, an output result in the form of an image selected by a user can be obtained in the result display area; the output result in the form of the image selected can be fed back to the input area; the user's input information can be obtained in the input area; and the output result in the form of an image can be subsequently processed according to the input information.

[0074] The output result in the form of the selected image can be fed back to the input area in the form of a reference.

[0075] In a specific application scenario, the user needs to generate a video file.

[0076] like Figure 3 As shown, first, the user can enter instructions in the input area to command the large language model (LLM) to generate a picture of a puppy. After the user's input information is sent in the input area, the user can see the content he just entered and the feedback text of the LLM model in the interactive area, and the generated picture will be fed back in the result display area above.

[0077] like Figure 4 As shown, the user can, for example, modify the generated image results. The user can select the image to be operated in the result display area, and then enter the command "redraw the background, a puppy in the garden", click Send, and the redrawn image can be obtained in the result display area after LLM processing.

[0078] like Figure 5 As shown, for example, if the user is satisfied with the image result and needs to further generate a video file, he can select the image template used to generate the video in the result display area, and then enter the instruction to generate a 20s video using the image as a template, click Send, and the generated video can be obtained in the result selection area after LLM processing.

[0079] According to the interactive image processing method based on a large language model of the present application, an input area, an interaction area, a result display area and a plug-in area are displayed on the user side according to a preset layout; the user's input information is obtained in the input area, and the input information includes one or more of the following information: text information, voice information, image information; the large language model parses the input information, performs processing according to the input information and generates output results in image form and text form; the output result in image form is displayed in the result display area; the output result in text form is displayed in the interactive area; the user's input information is obtained by the plug-in area and / or the input area and / or the result display area, and the output result in image form is subsequently processed according to the input information. This can achieve a low-threshold, high-flexibility and intuitive operation image generation and editing experience through human-like interaction combined with plug-in function expansion.

[0080] It should be clearly understood that the present application describes how to form and use specific examples, but the principles of the present application are not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in the present application, these principles can be applied to many other embodiments.

[0081] Figure 6 The invention is a flowchart of an interactive image processing method based on a large language model according to another exemplary embodiment. Figure 6 The process 60 shown is for Figure 1 Supplementary description of the process shown.

[0082] like Figure 6 As shown, in S602, the input parameters and output parameters of the plug-in are determined. The functional description, input and output formats, and parameter types of the plug-in are defined to help the LLM understand the calling method of the plug-in.

[0083] For example, the plugin metadata includes the API URL, request method (GET / POST), request parameters, return data type, etc.

[0084] In S604, the plug-in is developed in a RESTful API format.

[0085] In S606, the plug-in is registered in the large language model. The image processing API is registered as a plug-in through the API or SDK provided by the LLM. During the registration process, the following are required: a plug-in identifier (ID); an API interface document; and permission configuration (such as an API key or access control).

[0086] In one embodiment, in OpenAI's API, the API can be registered directly in the model configuration through an upload interface or through system instructions.

[0087] In one embodiment, the registered plug-in may also be tested to ensure that the plug-in can normally interact with the LLM, simulate the user call process and verify the correctness of the returned result.

[0088] In one embodiment, access permissions are configured for each API, for example: API Key: Generate unique calling credentials for different users.

[0089] OAuth authentication: ensures that only authorized users can access specific functional modules.

[0090] In S608, the registered plug-in is displayed in the plug-in area.

[0091] In order to realize the human-computer interactive image editing system, a series of APIs that can be called through the network are developed in this application. These APIs cover multiple core functional modules in the system, such as image uploading, picture generation and background removal. These APIs are designed with RESTful interfaces to achieve cross-platform and remote calling capabilities. The specific API designs are as follows: (1) Image Upload API Function description: Used to receive images uploaded by users and return the upload results (such as the URL or storage path of the image). It supports multiple common image formats, such as JPEG and PNG.

[0092] Request method: POST Request parameters: image_file: The uploaded image file, which supports JPEG and PNG formats (use multipart / form-data format when uploading multiple files).

[0093] image_format: specifies the format of the uploaded image (such as "JPEG", "PNG").

[0094] Returns: status: request status, the value is "success" or "error".

[0095] image_url: The image URL or storage path returned after successful upload for subsequent processing.

[0096] (2) Speech Analysis API Function description: Convert the voice files uploaded by users into text, supporting multiple common audio formats (such as MP3, WAV, etc.).

[0097] Request method: POST Request parameters: audio_file: The uploaded audio file, supporting MP3, WAV and other formats.

[0098] language: The target language for speech recognition (such as "en", "zh").

[0099] Returns: status: request status, the value is "success" or "error".

[0100] transcription: The text content after recognition.

[0101] (3) Generate Image API Functional description: Generate corresponding pictures according to the text description entered by the user, supporting custom image size, random seed and batch generation functions.

[0102] Request method: POST Request parameters: prompt: text describing the generated image (e.g. "a sunset over the ocean").

[0103] seed: Random seed used to generate consistent image content. The default value is -1, which means no fixed seed is used.

[0104] aspect: The aspect ratio of the image (e.g. "1:1" for square, "16:9" for widescreen).

[0105] batch_size: The number of images to be generated. The default value is 1, which supports batch generation (for example, 3 images are generated at the same time when set to 3).

[0106] Returns: status: request status, the value is "success" or "error".

[0107] generated_images: List of generated image URLs.

[0108] (4)Remove background API (remove_background) Function description: This API is used to remove the background of an image and only retain the main part of the image.

[0109] Request method: POST Request parameters: image_url: The URL of the image whose background needs to be removed.

[0110] Returns: status: request status, the value is "success" or "error".

[0111] processed_image_url: The URL of the image after removing the background.

[0112] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided in the present application are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0113] In addition, it should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0114] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0115] Figure 7 FIG. 1 is a block diagram of an interactive image processing device based on a large language model according to an exemplary embodiment. Figure 7 As shown, the interactive image processing device 70 based on the large language model includes: a display module 702 , an input module 704 , a parsing module 706 , a display module 708 , an interaction module 710 , and a processing module 712 .

[0116] Display module 702, used to display the input area, interaction area, result display area, and plug-in area according to a preset layout on the user side; An input module 704 is used to obtain user input information in the input area, where the input information includes one or more of the following information: text information, voice information, and image information; A parsing module 706, configured to parse the input information using a large language model, perform processing based on the input information, and generate output results in image and text forms; A display module 708, used to display the output result in the form of an image in the result display area; An interactive module 710 is used to display the output result in text form in the interactive area; The processing module 712 is used to obtain the user's input information from the plug-in area and / or the input area and / or the result display area, and perform subsequent processing on the output result in the image form according to the input information.

[0117] According to the interactive image processing device based on a large language model of the present application, an input area, an interaction area, a result display area and a plug-in area are displayed on the user side according to a preset layout; the user's input information is obtained in the input area, and the input information includes one or more of the following information: text information, voice information, image information; the large language model parses the input information, performs processing according to the input information and generates output results in image form and text form; the output result in image form is displayed in the result display area; the output result in text form is displayed in the interactive area; the user's input information is obtained by the plug-in area and / or the input area and / or the result display area, and the output result in image form is subsequently processed according to the input information. This can achieve a low-threshold, high-flexibility and intuitive operation image generation and editing experience through human-like interaction combined with plug-in function expansion.

[0118] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0119] Refer to the following Figure 8 hereinafter describes an electronic device 800 according to this embodiment of the present application. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), a display unit 840, etc.

[0121] The storage unit stores a program code, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 810 can perform the following steps: Figure 1 , Figure 6 Follow the steps shown in .

[0122] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0123] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0124] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 800 may also communicate with one or more external devices 800' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) so that a user can communicate with the device that interacts with the electronic device 800, and / or the electronic device 800 can communicate with any device (e.g., routers, modems, etc.) that communicates with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface 850. In addition, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 860. The network adapter 860 may communicate with other modules of the electronic device 800 through a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0126] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Fig. 9 As shown, the technical solution according to the implementation mode of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation mode of the present application.

[0127] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0128] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0129] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0130] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by a device, the computer-readable medium realizes the following functions: displaying the input area, interaction area, result display area and plug-in area according to a preset layout on the user side; obtaining the user's input information in the input area, and the input information includes one or more of the following information: text information, voice information, image information; parsing the input information with a large language model, performing processing according to the input information and generating output results in image form and text form; displaying the output results in image form in the result display area; displaying the output results in text form in the interaction area; obtaining the user's input information from the plug-in area and / or the input area and / or the result display area, and performing subsequent processing on the output results in image form according to the input information.

[0131] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0132] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0133] The exemplary embodiments of the present application are specifically shown and described above. It should be understood that the present application is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present application is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. An interactive image processing method based on a large language model, characterized in that: include: The input area, interaction area, result display area and plug-in area are displayed on the user side according to the preset layout; Acquire user input information in the input area, wherein the input information includes one or more of the following information: text information, voice information, and image information; The large language model parses the input information, performs processing according to the input information and generates output results in image form and text form; Displaying the output result in the form of an image in the result display area; Displaying the output result in text form in the interactive area; The user's input information is obtained from the plug-in area and / or the input area and / or the result display area, and the output result in the form of an image is subsequently processed according to the input information.

2. The method according to claim 1, characterized in that Also includes: Determine the input and output parameters of the plug-in; Developing the plugin in the form of a RESTful API; Registering the plug-in into the large language model; The registered plug-in is displayed in the plug-in area.

3. The method according to claim 1, characterized in that Acquiring the user's input information in the input area further includes: When the input information is text information, displaying the text information in the interactive area; When the input information is voice information, the voice information is converted into text information to be displayed in the interactive area.

4. The method according to claim 1, characterized in that The large language model parses the input information, performs processing based on the input information and generates output results in image and text form, including: Parsing the input information to generate input instructions; Generate a text reply content as an output result according to the input instruction; The input instruction is processed through a large language model to generate an output result in the form of an image.

5. The method according to claim 4, characterized in that The input instruction is processed by a large language model to generate an output result in the form of an image, including: The large language model calls the target plug-in based on the function call technology; The target plugin is used to process the output image.

6. The method according to claim 5, characterized in that The large language model calls the target plug-in based on the function call technology, including: The large language model performs multi-level intent analysis on the input instruction; Large language models understand context; The target plug-in is dynamically determined according to the results of the multi-level intent analysis and context understanding.

7. The method according to claim 1, characterized in that The user's input information is obtained from the plug-in area and / or the input area and / or the result display area, and the output result in the form of an image is subsequently processed according to the input information, including: According to a user's selection, determining a target plug-in from a plurality of plug-ins displayed in the plug-in area; The large language model calls the target plug-in to perform subsequent processing on the output result in the image form.

8. The method according to claim 1, characterized in that The user's input information is obtained from the plug-in area and / or the input area and / or the result display area, and the output result in the form of an image is subsequently processed according to the input information, including: Obtaining an output result in the form of an image selected by the user in the result display area; Feeding back the output result in the form of the clicked image to the input area; Acquire user input information in the input area; The output result in the form of an image is subsequently processed according to the input information.

9. The method according to claim 1, characterized in that Feeding back the output result of the selected image to the input area includes: The output result in the form of the clicked image is fed back to the input area in the form of a reference.

10. An interactive image processing device based on a large language model, characterized in that: include: The display module is used to display the input area, interaction area, result display area and plug-in area according to the preset layout on the user side; An input module, used to obtain user input information in the input area, wherein the input information includes one or more of the following information: text information, voice information, and image information; A parsing module, used for parsing the input information with a large language model, performing processing according to the input information and generating output results in image and text forms; A display module, used to display the output result in the form of an image in the result display area; An interactive module, used for displaying the output result in text form in the interactive area; The processing module is used to obtain the user's input information from the plug-in area and / or the input area and / or the result display area, and perform subsequent processing on the output result in the image form according to the input information.

Images