A large model design method for operation control in complex environment

Abstract

The application relates to the technical field of large model design, and discloses a large model design method for operation control in a complex environment, which comprises a hybrid model framework for realizing management and flexible combination of various operation models; the framework of the hybrid model is divided into four levels of perception, projection, analysis and generation; each level manages a corresponding model set; the multi-modal large model flexibly organized by the hybrid model framework can realize model training in a step-by-step training mode; the hybrid model modeling framework technology with the four levels of perception, projection, analysis and generation can effectively integrate different modal and different type models in a unified architecture, realize efficient and flexible analysis and processing of multi-modal data input, and generate effective decision or task script output, thereby supporting perception and decision generation required by a complex operation control system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of hybrid model design, in particular to a large model design method for operation control in complex environment. BACKGROUND

[0002] In large operation control scenarios, due to the involvement of a large number of complex perception devices, operation data and business logic, existing artificial intelligence technology can only play the role of assisting decision-making and analysis optimization at a single application point, and cannot provide effective assistance at the system level. In particular, in complex space-time environments, operation control needs to meet the information extraction and effective fusion of multiple data modalities, comprehensive analysis based on multi-modal information and flexible selection of analysis strategies, as well as flexible decision or task script output according to different scene requirements. In the prior art, although large language models and multi-modal large language models have been applied in the field of multi-modal knowledge analysis, they are difficult to construct efficient and flexible analysis strategies and form effective task outputs, especially lacking good task generalization ability. SUMMARY

[0003] (I) Technical problems solved

[0004] To solve the above-mentioned problems existing in the prior art, the present application provides a large model design method for operation control in complex environment, which has the advantages of being capable of realizing a multi-modal operation control hybrid model framework through a hybrid model modeling framework technology of four levels of perception, projection, analysis and generation, effectively integrating data perception of different modalities and decision analysis model architectures of different types, flexibly analyzing and processing and generating effective decision or task script output, thereby supporting the perception and decision generation required by complex operation control systems.

[0005] (II) Technical solutions

[0006] In order to achieve the above-mentioned multi-modal operation control hybrid model framework realized by the hybrid model modeling framework technology of four levels of perception, projection, analysis and generation, which can effectively integrate data perception of different modalities and decision analysis model architectures of different types, flexibly analyze and process and generate effective decision or task script output, thereby supporting the perception and decision generation required by complex operation control systems, the present application provides the following technical solutions: A large model design method for operation control in complex environment, comprising a hybrid model framework for realizing management and flexible combination of a plurality of operation models, the hybrid model framework comprising a perception layer responsible for feature analysis and extraction of different modal data;

[0007] a projection layer responsible for projecting features of different modal data to the same feature space for alignment;

[0008] Analysis layer: analyzing and predicting the input multi-modal data according to the aligned data features;

[0009] Generation layer: generating corresponding content according to the analysis prediction by the analysis layer, and verifying and correcting the generated content by interacting with the tool;

[0010] Wherein, the models in the perception, projection, analysis and generation layers are combined into a multi-modal large model, the mixed model framework is MMF, and is represented as:

[0011] MMF={MF,FA,MA,FO}

[0012] MF={mf0,mf1,…mf M}

[0013] FA={fa0,fa1,…fa K}

[0014] MA={ma0,ma1,…ma L}

[0015] FO={fo0,fo1,…fo C}

[0016] The calculation mode of the framework is represented as: O=MMF(D), the input of the framework is N-way different modal information D={d0,d1,…d N}, and the output is C-class output content O={o0,o1,…o C}.

[0017] Preferably, the perception layer manages a set of feature extraction models adapted to extract different modal data features, and the multi-modal information input {d0,d1,…d N} in the perception layer is extracted by the corresponding modal information extraction model {mf0,mf1,…mf M}, to realize feature extraction and generate corresponding features {f0,f1,…f N}.

[0018] Preferably, the perception layer provides M feature extraction models {mf0,mf1,…mf M}, and the feature extraction is represented as f i =mf i (d i ), each 1-way data can correspond to 0 to M feature extraction models, and the feature information output from the perception layer is N to Nx(M+1) ways, the projection layer provides K projection models {fa0,fa1,…fa K}, which is responsible for aligning the grouped r-way feature information in the same feature space and serving as the input of the analysis layer model, and the aligned features are represented as: {faj ,fa j+1 ,fa j+r-1}=fai(f j ,f j+1 ,…f j+r-1 )。

[0019] Preferably, the analysis layer provides L analysis models {ma0,ma1,…ma L}, and the analysis process is represented as: ao j =ma j ({fa j ,fa j+1 ,…fa j+r-1}).

[0020] Preferably, the generation layer is responsible for further verification and conversion of the output of the analysis layer, so as to realize the conversion from the script output by the analysis layer to the code that can be connected with other tools and realize task operation.

[0021] Preferably, the generation layer simultaneously provides C combination functions {fo0,fo1,…fo C} responsible for combining the results generated by different analysis models, represented as o i =fo i (ao j ,ao j+1 ,…ao j+r-1 ), wherein, for the results generated by the L analysis models, the generation layer can combine L to L! generated information.

[0022] Preferably, the models in the perception layer, the projection layer, the analysis layer and the generation layer are connected together through data channels to constitute the main part of the multi-modal large model, which can be expressed as:

[0023] M={MF,FA,MA,fo}

[0024] MF={mf0,mf1,…mf m}

[0025] FA={fa0,fa1,…fa k}

[0026] MA={ma0,ma1,…ma l}

[0027] Here, the model M is composed of m feature extraction models, k projection models, l analysis models and 1 generation model; if MMF is expressed in a graph manner, M belongs to the subgraph of MMF, and M needs to satisfy the following 5 constraints:

[0028] C1:Graph(M)={V,E},V represents nodes, and E represents edges

[0029] C2: V = Node(MF) U Node(FA) U Node(MA) U Node(fo);

[0030] C3: E = (Node(MF) X Node(FA)) U (Node(FA) X Node(MA)) U (Node(MA) X Node(fo)) = (Node(MF) X Node(MA)) U (Node(MA) X Node(fo))

[0031] C4: ((Node(MF) X Node(FA))!= {} && (Node(FA) X Node(MA))!= {} && (Node(MA) X Node(fo))!= {}) || (Node(MF) X Node(MA))!= {} && (Node(MA) X Node(fo))!= {}

[0032] C5: WeeklyConnected(Graph(M)) = TRUE

[0033] Wherein, Node represents taking the model in the layer as a node to generate a node set; A X B represents a set of all edges (associations between nodes) between two node sets A and B; WeeklyConnected represents taking M as a weakly connected graph.

[0034] Preferably, the projection layer and the analysis layer of the hybrid model framework can be fused at the neural network architecture level to be modeled in a tightly coupled manner; the analysis layer and the generation layer can be fused at the neural network architecture level to realize the generation model in a tightly coupled manner.

[0035] Preferably, the perception layer of the hybrid model framework can directly interface the data with the analysis layer to realize the solving processing based on logical rules and constraint conditions.

[0036] Preferably, the multi-modal large model flexibly organized by the hybrid model framework can realize model training in a step-by-step training manner, and a user can define the parameter range of each training, freeze a part of parameters each time, and update the training range of the model step by step until the training is completed according to the predefined order.

[0037] (Three) beneficial effects

[0038] Compared with the prior art, the present application provides a large model design method for operation control in a complex environment, which has the following beneficial effects:

[0039] 1. The hybrid model design technique for operation control in complex environments can flexibly combine different modalities and types of models through a hierarchically constructed hybrid model framework, achieving efficient and flexible analysis of complex operation control systems.

[0040] 2. The hybrid model design technique for operation control in complex environments supports information extraction and effective fusion of multiple data modalities in the hybrid model framework, enabling comprehensive analysis based on multi-modal information and flexible selection of analysis strategies to improve task generalization.

[0041] 3. The hybrid model design technique for operation control in complex environments generates flexible decision or task script output based on different scene requirements, and converts it to executable code after error correction by a verification tool, ensuring the accuracy and efficiency of task execution. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A hybrid model framework (MMF) diagram of the present application;

[0043] Figure 2 Another hybrid model framework (MMF) diagram of the present application;

[0044] Figure 3 A projection layer implementation process diagram of the present application;

[0045] Figure 4 An analysis layer implementation process diagram of the present application;

[0046] Figure 5 A generation layer implementation process diagram of the present application;

[0047] Figure 6 A whole training process FT(M) flowchart of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0049] As shown in Figures 1-6 A large model design method for operation control in complex environments includes a hybrid model framework for managing and flexibly combining multiple operation models. The hybrid model framework includes a perception layer responsible for feature analysis and extraction of different modal data. Figure 1 As shown in or 2, in the perception layer, multi-modal information input {d0, d1, … dN}By corresponding to different modal information extraction models{mf0,mf1,…mf M}to achieve feature extraction and generate corresponding features{f0,f1,…f N}Here, data of the same modality can use different feature extraction models to achieve focused feature extraction. Input data can also not necessarily pass through feature extraction (such as{d N+1 ,d N+2 ,…d N+r}) that is, traditional structured data can be directly used as input for rule solving or constraint solving. Each modality data is associated with a prefix prx i when entering the projection layer, indicating whether it participates in the projection conversion or participates in the calculation of which analysis model;

[0050] Projection layer: responsible for projecting the features of different modal data into the same feature space for alignment; as Figure 3 shown, the projection layer processes the alignment of data features of different modalities in the feature space. The specification of the feature space depends on the prefix prx i transmitted by the perception layer, to align the data features of different modalities in the same feature space. The alignment method is divided into loose coupling and tight coupling. The loose coupling method (see attached Figure 3 .b) only through an alignment model (usually a deep neural network) to convert and project the data features that need to be analyzed together ({f0,f1,…f m}) into token sequences for analysis by the large language model in the back end. The tight coupling method (see attached Figure 3 .c) is to realize projection and analysis by a multi-modal large model, that is, the projection model and the analysis model are completed by one model from projection to analysis (which is equivalent to the current form of multi-modal large language model). The projection layer can also directly transfer data or data features to the model in the analysis layer for processing (see attached Figure 3 .a).

[0051] Analysis layer: analyzes and predicts the input multi-modal data according to the aligned data features; as Figure 4 shown, the analysis layer has a certain coupling relationship with the projection layer and the generation layer in the calculation mode, and is divided into 4 calculation modes according to the coupling degree from low to high:

[0052] 1. Directly analyze the perception data, as described above, the perception data (usually structured data) can be directly processed by the analysis layer, processed by the analysis layer model (usually a rule or constraint solving model) and generate analysis results;

[0053] 2. In the projection layer, convert the data features that need to be analyzed into token sequences (loose coupling method), analyze and generate results by the large language model;

[0054] 3. The projection layer and the analysis layer are implemented by the same multimodal large model (tight coupling mode), and the model is used for analysis and result generation;

[0055] 4. The analysis layer and the generation layer are implemented by the same diffusion model, which is used to adapt to the input of text, image or other modal information provided by the perception layer, and generate text, image and other content output.

[0056] Generation layer: according to the analysis prediction of the analysis layer, the corresponding content is generated, and the generation layer verifies and corrects the generated content by interacting with the tool; The generation layer mainly outputs 3 types of information:

[0057] 1. Analysis results expressed in structured data table;

[0058] 2. Content generation expressed in text, image and other unstructured data, such as audio and video;

[0059] 3. Task script expressed in structured text, such as XML or Json format text output.

[0060] Further, as shown in Figure 5 , the third type of output of the generation layer is the main output mode for running control (see attached Figure 5 .a), which generates a script for users to implement task planning and execution through analysis and deduction of the analysis layer model. The script can be verified by a verification tool, corrected by a correction tool, and then converted into an execution script or code of an application program interface that can interact with tools to interface with other production tools to execute the planned tasks.

[0061] As mentioned before, the generation layer can realize the combination of different analysis model outputs by fo function: i =fo(ao j ,ao j+1 ,…ao j+r-1 ). Through combination, new scripts or contents are generated. The fo function can clip the input content {ao j ,ao j+1 ,…ao j+r-1} according to the heuristic algorithm customized in fo (see attached Figure 5 .b), so as to realize the controllable strategy of content generation and provide a technical basis for customizing more complex applications.

[0062] Among them, the models in the perception, projection, analysis and generation 4 layers are combined into a multimodal large model, and the mixed model framework is MMF, which is represented as:

[0063] MMF = {MF, FA, MA, FO}

[0064] MF = {mf0, mf1, … mf M}

[0065] FA = {fa0, fa1, … fa K}

[0066] MA = {ma0, ma1, … ma L}

[0067] FO = {fo0, fo1, … fo C}

[0068] The computing mode of the framework is represented as: O = MMF(D), the input of the framework is N-way different modal information D = {d0, d1, … d N}, and the output is C-class output content O = {o0, o1, … o C}.

[0069] In an embodiment of the present application, the perception layer manages a feature extraction model set suitable for feature extraction of different modal data, the multi-modal information input {d0, d1, … d N} in the perception layer is implemented for feature extraction and generates corresponding features {f0, f1, … f M} through corresponding different modal information extraction models {mf0, mf1, … mf N}.

[0070] In an embodiment of the present application, the perception layer provides M feature extraction models {mf0, mf1, … mf M}, the feature extraction is represented as f i = mf i (d i ), each 1-way data can correspond to 0 to M feature extraction models, and the feature information output from the perception layer is N to Nx(M+1) ways, the projection layer provides K projection models {fa0, fa1, … fa K}, which is responsible for aligning the grouped r-way feature information in the same feature space and serving as the input of the analysis layer model, and the aligned features are represented as: {fa j , fa j+1 , … fa j+r-1} = fai(f j , f j+1 , … f j+r-1 ).

[0071] In an embodiment of the present application, the analysis layer provides L analysis models {ma0, ma1, … ma L}, and the analysis process is represented as: ao j= ma j ({fa j j+1 ,…fa j+r-1})。

[0072] In an embodiment of the present application, the generation layer is responsible for further verification and conversion of the output of the analysis layer, to realize the script output from the analysis layer to the code that can be interfaced with other tools and realize task operations.

[0073] In an embodiment of the present application, the generation layer simultaneously provides C combination functions {fo0, fo1, … fo C} responsible for combining the results generated by different analysis models, denoted as o i = fo i (ao j , ao j+1 , …, ao j+r-1 ), where for the results generated by L analysis models, the generation layer can combine L to L! ways of generating information.

[0074] In an embodiment of the present application, the models in the perception layer, the projection layer, the analysis layer and the generation layer are connected together through data channels to constitute the main part of the multi-modal large model, which can be expressed as:

[0075] M = {MF, FA, MA, fo}

[0076] MF = {mf0, mf1, … mf m}

[0077] FA = {fa0, fa1, … fa k}

[0078] MA = {ma0, ma1, … ma l}

[0079] Here, the model M is composed of m feature extraction models, k projection models, l analysis models and 1 generation model; if MMF is expressed in a graph manner, then M belongs to the subgraph of MMF, and M needs to satisfy the following 5 constraints:

[0080] C1: Graph (M) = {V, E}, V represents nodes and E represents edges

[0081] C2: V = Node (MF) U Node (FA) U Node (MA) U Node (fo);

[0082] C3: E = (Node(MF) X Node(FA)) U (Node(FA) X Node(MA)) U (Node(MA) X Node(fo)) = (Node(MF) X Node(MA)) U (Node(MA) X Node(fo))

[0083] C4: ((Node(MF) X Node(FA))!= {} && (Node(FA) X Node(MA))!= {} && (Node(MA) X Node(fo))!= {}) || (Node(MF) X Node(MA))!= {} && (Node(MA) X Node(fo))!= {})

[0084] C5: WeeklyConnected(Graph(M)) = TRUE

[0085] Wherein, Node represents taking the model in the layer as a node to generate a node set; A X B represents a set of all edges (association between nodes) between two node sets A and B; WeeklyConnected represents taking M as a weakly connected graph.

[0086] In an embodiment of the present application, the projection layer and the analysis layer of the mixed model framework can be fused at the neural network architecture level to be modeled in a tightly coupled manner; the analysis layer and the generation layer can be fused at the neural network architecture level to realize the generation model in a tightly coupled manner.

[0087] In an embodiment of the present application, the perception layer of the mixed model framework can directly interface the data with the analysis layer to realize the solving processing based on logical rules and constraint conditions.

[0088] In an embodiment of the present application, the multi-modal large model flexibly organized by the mixed model framework can realize model training in a step-by-step training manner, and a user can define the parameter range of each training, freeze a part of parameters each time, and update the training range of the model step by step until the training is completed according to the predefined order.

[0089] Working principle: the modeling framework technology of the layered construction of the mixed model (referred to as the mixed model framework MMF) supports the above requirements for system flexibility and processing capacity, and the mixed model framework is divided into four layers:

[0090] Perception layer: responsible for feature analysis and extraction of different modal data;

[0091] Projection layer: responsible for projecting the features of different modal data into the same feature space for alignment;

[0092] Analysis layer: analyze and predict the input multi-modal data according to the aligned data features;

[0093] Generation layer: generate corresponding content according to the analysis prediction by the analysis layer, the generated content includes script represented by text, image, audio / video or other modal information, the generation layer realizes verification and error correction of the generated content by interacting with tools.

[0094] The perception layer realizes feature extraction of different modal data and different features of the same modal data by integrating different feature extraction models, and supports requirement 1. For requirement 2, the projection layer is responsible for the docking of data / data features and models in the analysis layer, which can directly dock the data to the analysis model, or can align the data features of different modalities in the corresponding feature space according to their combination and input the aligned intermediate information (usually represented as token sequence) into the corresponding analysis model in the analysis layer. The generation layer generates scripts or different modal data, for scripts, the generation layer is responsible for combining verification and parsing tools to correct the script generated by the analysis model and convert it into executable task code, which supports requirement 3.

[0095] As described in the mixed model framework overall architecture section, the models in the four levels in MMF can be flexibly combined into different multi-modal large models {M0, M1, … M N}. The training method of each model Mi is the same as that of the multi-modal large model, which can decouple and train each layer of the model in multiple steps (i.e. freeze a part of the model parameters in each step, train in multiple steps), or train all models at the same time. FT represents the training process, D represents the training data set, fs i represents the model set with frozen parameters, and FS represents the sequence of model sets with frozen parameters, i.e. step-by-step training. The overall training process FT(M) is represented as the process shown in Figure 6 .

[0096] In summary, the mixed model design technology for running control in complex environment can flexibly combine different modalities and types of models through the hierarchical mixed model framework, realize efficient and flexible analysis of complex running control system, support information extraction and effective fusion of multiple data modalities, realize comprehensive analysis based on multi-modal information and flexible selection of analysis strategy, improve task generalization ability, and generate flexible decision or task script output according to different scene requirements, and convert it into executable code through the verification tool to ensure the accuracy and efficiency of task execution.

[0097] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and implementations, it is to be understood that the terminology used is for the purpose of descriptive clarity and that it is intended to be limited only by the words recited in the appended claims. The scope of the present application shall be limited only by the claims.

[0098] While the embodiments of the application have been shown and described herein, it is to be understood that the application is not limited to these embodiments. Rather, numerous modifications are possible without departing from the spirit and scope of the present application as delineated by the claims and their equivalents.

Claims

1. A large model design method for operation control in a complex environment, comprising a hybrid model framework for realizing management and flexible combination of a plurality of operation models, characterized in that, The mixed model framework comprises a perception layer: responsible for feature analysis and extraction of different modal data; a projection layer: responsible for projecting the features of different modal data into the same feature space for alignment; an analysis layer: analyzing and predicting the input multi-modal data according to the aligned data features; a generation layer: generating corresponding content according to the analysis and prediction of the analysis layer, and the generation layer realizes verification and error correction of the generated content by interacting with tools; wherein the models in the four layers of perception, projection, analysis and generation are combined into a multi-modal large model, the mixed model framework is MMF, represented as: MMF = {MF, FA, MA, FO} MF = {mf0, mf1, … mf M} FA = {fa0, fa1, … fa K } MA = {ma0, ma1, … ma L } FO = {fo0,fo1, … fo C } The computing mode of the framework is represented as: O = MMF(D), the input of the framework is N different modal information D = {d0, d1, … d N }, and the output is C class output content O = {o0, o1, … o C }. The models in the perception layer, projection layer, analysis layer and generation layer are connected together through data channels to constitute the main part of the multi-modal large model, which can be expressed as: M = {MF, FA, MA, FO} MF = {mf0, mf1, … mf m} FA = {fa0, fa1, … fa k} MA = {ma0, ma1, … ma l } Here, the model M is composed of m feature extraction models, k projection models, l analysis models and 1 generation model; if MMF is expressed in a graph manner, M is a subgraph of MMF, and M needs to satisfy the following 5 constraints: C1: Graph(M) = {V, E}, V represents nodes, and E represents edges C2: V = Node(MF) U Node(FA) U Node(MA) U Node(fo) C3: E = (Node(MF) X Node(FA)) U (Node(FA) X Node(MA)) U (Node(MA) X Node(fo)) = (Node(MF) X Node(MA)) U (Node(MA) X Node(fo)) C4: ((Node(MF) X Node(FA)) ≠ {} && (Node(FA) X Node(MA)) ≠ {} && (Node(MA) X Node(fo)) ≠ {}) || (Node(MF) X Node(MA)) ≠ {} && (Node(MA) X Node(fo)) ≠ {}) C5: WeeklyConnected(Graph(M)) = TRUE Where Node represents the models in the layers as nodes to generate a node set; A X B represents the set of all edges (associations between nodes) between two node sets A and B; WeeklyConnected represents that M is a weakly connected graph.

2. The large model design method for operation control in a complex environment according to claim 1, characterized in that: The perception layer manages a feature extraction model set adapted to feature extraction of different modal data, and multi-modal information input {d0, d1, … d N} in the perception layer is extracted by corresponding modal information extraction model {m f 0, mf1, … mf M}, feature extraction is realized, and corresponding features {f0, f1, … f N} are generated.

3. The large model design method for operation control in a complex environment according to claim 1 or 2, characterized in that: The perception layer provides M feature extraction models {mf0, mf1, … mf M} and feature extraction is represented as f i = mf i (d i ). Each 1-way data can correspond to 0 to M feature extraction models, and the feature information output from the perception layer is N to Nx(M+1) ways. The projection layer provides K projection models {fa0, fa1, … fa K} responsible for aligning the grouped r-way feature information in the same feature space and serving as the input of the analysis layer model. The aligned features are represented as: {fa j , fa j+1 , … fa j+r-1} = fa i (f j , f j+1 , … f j+r-1 ).

4. The large model design method for operation control in a complex environment according to claim 1, characterized in that: The analysis layer provides L analysis models {ma0, ma1,... maL} and an analysis process is represented as: ao L = ma j ({fa j , fa j ,... fa j+1}). j+r-1 ​ 5. The method of claim 1, wherein: The generation layer is responsible for further verification and conversion of the output of the analysis layer to realize the conversion of the script output from the analysis layer to the code that can be connected with other tools and realize task operation.

6. The method of claim 1, wherein: The generation layer provides C combination functions {fo0, fo1,... fo C } responsible for combining results produced by different analysis models, denoted as o i = fo i (ao j , ao j+1 ,... ao j+r-1 ), where for L analysis model generated results, the generation layer can combine L to L! ways of generation information.

7. The method of claim 1, wherein: The projection layer and the analysis layer of the mixed model framework can be fused at the neural network architecture level to realize tight coupling modeling; the analysis layer and the generation layer can be fused at the neural network architecture level to realize the generation model in a tight coupling manner.

8. The method of claim 1, wherein: The perception layer of the mixed model framework can directly connect the data with the analysis layer to realize the solving process based on logical rules and constraint conditions.

9. The method of claim 1, wherein: The multi-modal large model flexibly organized by the mixed model framework can realize model training in a step-by-step training manner, and a user can define the parameter range of each training, freeze a part of parameters each time, and update the training range of the model step by step until the training is completed according to a predefined order.

Images