Method and system for expanding input and output capability of locally deployed large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a method and system for expanding the input and output capacity of a locally deployed large model, and the method comprises the steps: deploying a plurality of open source large models locally, and forming a large model service cluster of unified management; receiving a prompt instruction input by a user by using the interactive interface, identifying a modal type in the prompt instruction and a user demand, and matching a required target large model from the large model service cluster; calling the target large models in series according to user demand priorities, and integrating circulation data among the target large models to form a final output result; and finally, an output result is returned to the interactive interface and displayed to the user. By applying the method, the non-inductive input and output capability of a user is expanded, local deployment is carried out by adopting the open source large model, it is ensured that data cannot be out of a domain, the compliance and cost advantages of the open source model are considered, and meanwhile, a security mechanism of data circulation in a local area network is established.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a method and system for extending input and output capabilities of a locally deployed large model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large models have begun to be deeply differentiated in professional scenarios and subdivided into natural language large models (LLM, Large Language Model), visual large models (VLM, Visual Language Model), three-dimensional space large models (Three-Dimensional Model), and world large models (LWM, Large World Model) in multiple fields. Large models in each field have excellent capabilities in their respective fields, for example, LLM is good at text processing, and VLM is good at image input.

[0003] Currently, although commercial large models exhibit excellent professional capabilities, the local deployment process is complex, the hardware and license costs are high, and private data needs to be uploaded to a third-party server, which poses a security risk of data leakage, making it difficult to meet the needs of small and medium-sized enterprises. Although open source large models have low deployment costs and support local operation, they mostly only support single-modal input, such as only supporting text input or only supporting image input, and some multi-modal open source models (such as basic VLM) have weak inference capabilities, and the output results often deviate greatly from user expectations.

[0004] For small and medium-sized enterprises that are limited by cost and resources, they hope to obtain a large model solution with "low cost, high availability, multi-modal, local deployment, and accurate output", but the existing technology cannot simultaneously meet the above needs. How small and medium-sized enterprises can break through the limitations of single models by integrating the professional capabilities of multi-field large models is an important technical challenge in the current application of large models. SUMMARY

[0005] The purpose of the present application is to provide a method and system for extending the input and output capabilities of a locally deployed large model, which solves the problem of how small and medium-sized enterprises can safely and cost-effectively deploy open source large models, realizes low-cost, high-availability multi-modal input support, and at the same time enhances the accuracy and expected compliance of the output results of the large model, meeting the needs of small and medium-sized enterprises for locally deployed large models.

[0006] To achieve the above purpose, the present application provides the following technical solutions: According to one aspect of the present application, a method for extending the input and output capabilities of a locally deployed large model is provided, comprising the following steps: S1: locally deploying a plurality of open source large models to form a unified management large model service cluster; S2: receiving a prompt instruction input by a user through an interactive interface, the prompt instruction comprising at least multi-modal information; S3: identifying the modal type in the prompt instruction and the user demand, and matching a target large model from the large model service cluster according to the identification result; S4: calling the target large model in series according to the user demand priority through an API interface, integrating the data transferred among the target large models, and forming a final output result; S5: returning the final output result to the interactive interface and showing it to the user.

[0007] According to an embodiment of the present application, in step S1, the large model service cluster comprises at least a modal input type model, a text processing type model, and a strong reasoning type model; the modal input type model is a visual large model supporting image and audio non-text modal input and converting multi-modal information into structured data; the text processing type model is a natural language large model for processing pure text input or text data converted by the modal input type model; and the strong reasoning type model has deep reasoning ability of logical analysis and scheme generation for optimizing the accuracy of the output result.

[0008] According to an embodiment of the present application, in step S1, all large models in the large model service cluster are configured with standardized and adjustable API interfaces.

[0009] According to an embodiment of the present application, in step S2, the multi-modal information comprises text information, image information, audio information, and video segment information.

[0010] According to an embodiment of the present application, in step S3, the modal type is determined by a file format recognition algorithm or a content recognition algorithm to determine the modal attribute of the input information; and the user demand is determined by extracting a core demand label in the prompt instruction, the core demand label comprising "analysis", "generation", "inference", and "conclusion".

[0011] According to an embodiment of the present application, in step S4, the setting rule of the user demand priority is that the multi-modal input processing priority is higher than the pure text processing, and the core reasoning task priority is higher than the auxiliary data processing.

[0012] According to an embodiment of the present application, in step S4, the data fusion algorithm is used to integrate the data transferred among the target large models, the data fusion algorithm comprising key information extraction and logical verification to form a unified and coherent final output result.

[0013] According to one embodiment of the present application, in step S1, the locally deployed operating environment is an open source operating system, supports GPU computing power scheduling and multi-process concurrent processing, provides computing resources and storage resources for the large model service cluster, and all data only flows within the local area network.

[0014] According to one embodiment of the present application, in step S1, the large model service cluster supports dynamic model adjustment, adds or unloads large models according to actual needs, and the added large models are accessed to the cluster through a standardized API interface.

[0015] On the other hand, the present application also provides a system for expanding the input and output capabilities of a locally deployed large model, comprising an interactive interface module, a large model control module, a large model service cluster, and a local operating system module, wherein, The interactive interface module is used to receive multi-modal prompt instructions from a user, encapsulate them into a standardized format, and display the final results returned by the large model control module. The large model control module is used to analyze user prompt instructions, match target large models, serially call multiple target large models, and integrate data flow. The large model service cluster includes multiple types of large models, each model communicates with the large model control module through an API interface, and realizes the complementary capabilities of multiple large models. The local operating system module uses an open source operating system to provide computing power and storage resources for the above modules, and ensures that data flows within the local area network.

[0016] Compared with the prior art, the method and system for expanding the input and output capabilities of a locally deployed large model of the present application have the following advantages: 1. The input capability is expanded without user awareness. Specifically, by connecting modal input models and text models, the upgrade from a single model to multi-modal input is realized, and the user does not need to learn any new operation, achieving a non-aware experience.

[0017] 2. The output result is enhanced, and the expected compliance is improved. Specifically, the complementary capabilities of different large models are utilized to solve the problem of serious deviation of single model output from expectations, and the integrated output result is more in line with user expectations.

[0018] 3. The local deployment is safe, and the risk of data leakage is reduced. Specifically, the local cluster is built based on an open source model, all data only flows within the local area network, completely avoiding the risk of private data diffusion existing in the use of commercial large models, and meeting the needs of enterprise data security compliance.

[0019] 4. Low cost and high flexibility, suitable for small and medium-sized enterprises. The open source model can reduce deployment costs, no need for commercial license fees, and support dynamic addition or unloading of models, no need to restructure the system architecture, suitable for small and medium-sized enterprises with limited budget and flexible demand changes. BRIEF DESCRIPTION OF DRAWINGS

[0020] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation of the present application. In the drawings: Figure 1 is a method flowchart for extending the input and output capabilities of a locally deployed large model according to an embodiment of the present application; Figure 2 is a system schematic diagram for extending the input and output capabilities of a locally deployed large model according to an embodiment of the present application; Figure 3 is a locally deployed large model system service flow schematic diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same function and role are distinguished by using "first", "second", etc. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and do not limit the order. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. also do not necessarily mean different.

[0022] It should be noted that in the present application, "exemplary" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0023] In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of "and / or" describing the associated objects indicates that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. The following at least one or similar expressions mean any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b and c can be single or multiple.

[0024] The technical solution of the present application forms a local open source cluster, a multi-modal serial calling, and a large model input and output processing flow. The multi-modal input is extended without perception by dynamically adapting the user input information through the open source model. The output is extended without perception by integrating and transferring data among various large models and only showing the final result to the user. Among them, each large model dynamically matched according to the user input is chained and serially written according to the modal input type, text processing type, and strong reasoning type. On the one hand, it realizes the division of labor of large models, and on the other hand, it also realizes data fusion. The modal input type is only responsible for data conversion, and the reasoning type focuses on result optimization. The data flow among large models can use standardized protocols, such as outputting fixed format JSON by the modal input type for text type model to parse. By using open source large models for local deployment, it ensures that data does not leave the domain, and takes into account the compliance and cost advantage of open source models, while establishing a security mechanism for data flow within the local area network.

[0025] As shown in Figure 1 , a method flow chart for extending the input and output capabilities of locally deployed large models is given, which includes the following steps: Step S101, constructing a local large model service cluster; A plurality of open source large models with different professional capabilities are deployed locally to form a unified management large model service cluster. The cluster includes at least three types of models, specifically including: Modal input type model, such as visual large model VLM, supporting image, audio and other non-text modal input, used to convert multi-modal information into structured data, such as converting images into text descriptions.

[0026] Text processing type model, such as natural language large model LLM, used for text understanding, information extraction and generation, responsible for processing pure text input or text data converted by modal input type model.

[0027] Strong reasoning class models, such as QWEN-7B-Chat| and Llama3 inference optimization model, need to have deep reasoning capabilities such as logical analysis and scheme generation to optimize the accuracy of the output results.

[0028] All large models in the cluster are configured with standardized callable API interfaces to support data interaction and calling by other modules within the local area network.

[0029] Step S102, receiving a multi-modal prompt instruction input by a user; The prompt instruction input by the user through the interactive interface includes at least multi-modal information. The multi-modal information includes text information and image information, such as text of “analyze product defects” and photos of product defects. Further, it can be extended to audio information such as voice instructions or video segment information according to actual needs. The interactive interface is responsible for packaging the instruction into a standardized format, such as JSON format, to ensure that the subsequent module can correctly parse it.

[0030] Step S103, parsing the prompt instruction and matching the target large model according to the parsing and recognition result; The parsed user prompt instruction includes two types: Modality type analysis, which is specifically determined by file format recognition or content recognition algorithm to determine the modality attribute of the input information. For example, files with suffixes of JPG or PNG are determined as images, and files with suffixes of TXT or MD are determined as text.

[0031] User demand analysis, which is specifically implemented by extracting core demand tags in the prompt instruction, such as whether it includes keywords such as “analyze”, “generate”, “reason”, “summarize” and the like.

[0032] According to the recognition result obtained by parsing, the target large model is matched from the cluster according to the principles of modality adaptation and demand adaptation. For example, if the “defect analysis” instruction contains images, the VLM large model is preferentially matched for image processing, then the LLM large model is matched for image and text description, and finally the strong reasoning class model is matched to generate defect analysis conclusions.

[0033] Step S104, calling the matched target large model in series and integrating the data flow; The target large model is called in series according to the user demand priority using the API interface. The priority is set as follows: multi-modal input processing priority is higher than pure text processing, and core reasoning task priority is higher than auxiliary data processing. For example, for a prompt instruction with images and text, the VLM large model is called first, then the LLM large model, and finally the strong reasoning class model.

[0034] The data flow specifically includes: the output result of the previous target large model, standardized conversion according to the order of the next target large model input format, and elimination of redundant data. For example, the unstructured image description output by the VLM large model needs to be converted into the structured Prompt prompt word format recognizable by the LLM large model, and the repeated image detail description needs to be eliminated.

[0035] The result integration specifically includes: collecting the output data of all target large models, and forming a unified and coherent final output result through data fusion algorithms such as key information extraction or logical verification. For example, the output result is a complete report including image defect description, cause analysis, and improvement suggestions.

[0036] Step S105, the final output result is returned to the interactive interface and displayed to the user.

[0037] The final output result is returned to the interactive interface and displayed in a user-friendly document or list format. Throughout the process, the user does not need to manually switch models or adjust input and output methods, and has no perception of the expansion of the input and output capabilities of the background large model.

[0038] As shown in Figure 2 A system diagram for expanding the input and output capabilities of a locally deployed large model is shown, which is used to implement the above method, including an interactive interface module, a large model control module, a large model service cluster, and a local operating system module, and the functions of each module are as follows: The interactive interface module is used to receive the user's multi-modal prompt instructions, including text, images, audio, video clips, etc., encapsulate the instructions into a standardized format such as JSON, and display the final output result returned by the large model control module to the user. The user has no perception of multi-modal input expansion and does not need to change the original usage habits, and the system will automatically adapt to the processing of multi-modal input.

[0039] The large model control module is used to analyze the modal type (text / image / audio) and core demand (analysis / generation / reasoning) in the user prompt instruction; according to the analysis and identification result, the target large model is selected from the large model service cluster, and multiple target large models are called in series according to the demand priority through the API interface, realizing the data flow conversion and final result fusion between models, and ensuring the coherence and accuracy of the output result.

[0040] The large model service cluster includes modal input type models, text processing type models, and strong reasoning type models. Each model communicates with the large model control module through a standardized API interface, supports dynamic access and unloading, and can be added by registering the API of a new model. Models that are not needed can be removed without affecting other models in the cluster. Through the complementary capabilities of different models in the cluster, multi-modal input support and output result enhancement can be achieved, expanding the input and output capabilities.

[0041] The local operating system module uses an open-source operating system such as Linux and adapts to frameworks such as CUDA, supporting GPU computing power scheduling and multi-process concurrent processing. It can provide computing resources and storage resources for other modules and ensure that all data, such as user instructions, intermediate results, and final results, only circulate within the local or local area network, avoiding the risk of private data spreading outside.

[0042] Embodiment 1: Small and medium-sized enterprise product defect analysis scenario As shown in Figure 3 The local deployment of the large model system is shown in the schematic diagram. The local operating system uses the Linux Ubuntu 22.04 system, equipped with an NVIDIA GPU that meets the computing power requirements of multiple models, and a 1TB SSD for storing model files and user data.

[0043] Deploy the large model service cluster: modal input type models use the open-source visual large-scale model LLaVA-13B, which supports image input and can convert product defect photos into text descriptions; text processing type models use the open-source natural language large model Llama 3-8B, which can expand text processing and information extraction capabilities and optimize defect description output by LLaVA; strong reasoning type models use the open-source model Qwen-7B-Chat, which has logical analysis capabilities and can generate root cause analysis and improvement suggestions based on defect descriptions. The above three models all provide RESTful API interfaces, and the large model control module calls the API through the Python requests library.

[0044] First, the user uploads the "product defect photo" through the interactive interface and inputs the text command: "analyze the product defect position, reason, and generate improvement plan in the picture". The interactive interface will package the "photo (.jpg format) and text" into a JSON format command and send it to the large model control module.

[0045] Then, the large model control module analyzes the command. The modal type is "image + text", and the core requirement is "defect analysis and suggestion generation".

[0046] Perform target large model matching: LLaVA (image processing) -> Llama 3 (text description analysis) -> Qwen-7B-Chat (inference analysis).

[0047] The large model control module calls the API of LLaVA and passes in the defect photo. LLaVA outputs the result as: "The product in the picture is a plastic shell, with 2 obvious cracks on the surface, with a length of about 5 mm, located at the corner of the shell."

[0048] The large model control module converts the output of LLaVA into the Prompt format of Llama3: "Please organize the following product defect description: [LLaVA output content]".

[0049] The large model control module calls the API of Llama3, passing in the Prompt, and Llama3 outputs the structured description as: "Product type: plastic shell; defect location: edge corner; defect number: 2; defect morphology: 5mm long crack"; The large model control module integrates the Llama3 output with the user's original instruction into the Prompt format of Qwen-7B-Chat: "Based on the following product defect information, analyze the defect reasons and generate improvement suggestions: [Llama 3 output content]".

[0050] The large model control module calls the API of Qwen-7B-Chat, passing the Prompt into Qwen-7B-Chat.

[0051] Qwen-7B-Chat outputs: "Defect reason: The corner of the plastic shell has uneven wall thickness, which causes cracks due to stress concentration inside the plastic shell. Improvement suggestions: 1. Optimize mold design, increase wall thickness at the corner, make it uniform. 2. Adjust the plastic forming parameters, reduce the cooling speed, reduce the internal stress.

[0052] The large model control module integrates the outputs of the three models to generate a complete report (including defect description, reason analysis and improvement suggestions), and returns the final result to the interactive interface. The user can view the report on the interface. The whole process does not need to manually operate the model, and is not affected by multi-modal input or result expansion.

[0053] The input function in this embodiment realizes "image + text" multi-modal input, breaking through the limitation of single text input, and the output result evolves from "simple image description" to "structured analysis report", with a significant increase in accuracy and practicality. The installation cost is only one-fifth of the commercial model (no license fee), and all data (photos, orders, reports) are stored locally, with no risk of data leakage.

[0054] Although the present application has been described in connection with various embodiments thereof, it will be understood that other modifications and variations will be apparent to those skilled in the art in view of the foregoing disclosure. It is therefore contemplated that the application will encompass all such modifications and variations as fall within the scope of the claimed application. It is intended that the expressions "including", "includes" and "comprising" be open-ended and do not exclude other components or steps. The singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise. Individual processors or other units can implement several of the functions recited in the description. Some measures are described in mutually different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0055] Although the present application has been described in connection with specific features thereof, it will be evident to an artisan that various modifications and changes can be made to the application without departing from the spirit and scope thereof. Accordingly, it is intended that the specification and drawings be considered as illustrative only, with the true scope of the application being indicated by the appended claims. Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for extending input and output capabilities of a locally deployed open source large model, the method comprising: The method comprises the following steps: S1: locally deploying a plurality of open source large models to form a unified management large model service cluster; S2: receiving a prompt instruction input by a user through an interactive interface, wherein the prompt instruction at least comprises multi-modal information; S3: identifying the modal type and user demand in the prompt instruction, and matching a target large model from the large model service cluster according to the identification result; S4: calling the target large model in series according to the user demand priority through an API interface, integrating the data flow between the target large models, and forming a final output result; S5: returning the final output result to the interactive interface and showing it to the user.

2. The method of claim 1, wherein In step S1, the large model service cluster at least comprises a modal input type model, a text processing type model, and a strong reasoning type model; the modal input type model is a visual large model supporting image and audio non-text modal input and converting multi-modal information into structured data; the text processing type model is a natural language large model for processing pure text input or text data converted by the modal input type model; and the strong reasoning type model has deep reasoning ability for logical analysis and scheme generation, and is used to optimize the accuracy of the output result.

3. The method of claim 1, wherein In step S1, all large models in the large model service cluster are configured with standardized callable API interfaces.

4. The method of claim 1, wherein In step S2, the multi-modal information comprises text information, image information, audio information, and video segment information.

5. The method of claim 1, wherein In step S3, the modal type is determined by a file format recognition or content recognition algorithm; and the user demand is determined by extracting a core demand tag in the prompt instruction, wherein the core demand tag comprises "analysis", "generation", "inference", and "conclusion".

6. The method of claim 1, wherein In step S4, the setting rule of the user demand priority is that the multi-modal input processing priority is higher than the pure text processing, and the core reasoning task priority is higher than the auxiliary data processing.

7. The method of claim 1, wherein In step S4, the data flow between the target large models is integrated by using a data fusion algorithm, and the data fusion algorithm comprises key information extraction and logical verification to form a unified and coherent final output result.

8. The method of claim 1, wherein In step S1, the local deployment environment is an open source operating system supporting GPU computing power scheduling and multi-process concurrent processing, providing computing resources and storage resources for the large model service cluster, and all data only flows within the local or local area network.

9. The method of claim 1, wherein In step S1, the large model service cluster supports dynamic model adjustment, and new large models are added or unloaded according to actual needs, and the new large models are added to the cluster by registering standardized API interfaces.

10. A system for extending the input and output capabilities of a locally deployed large model to implement the method of any one of claims 1 to 9, characterized in that, The system comprises an interactive interface module, a large model control module, a large model service cluster and a local operating system module, wherein The interactive interface module is used for receiving a multi-modal prompt instruction of a user, encapsulating the instruction into a standardized format and displaying a final result returned by the large model control module; The large model control module is used for analyzing the user prompt instruction, matching a target large model, serially calling multiple target large models and realizing data flow integration; The large model service cluster comprises multiple types of large models, and each model communicates with the large model control module through an API interface to realize complementary capabilities of multiple large models; The local operating system module adopts an open source operating system, provides computing capacity and storage resources for the above modules and ensures data flow within the local area network.

Images