An adaptive training and inference performance optimization system for multimodal large models

Abstract

The present invention provides an adaptive training and reasoning performance optimization system for a multimodal large model, which relates to the field of electrical digital data processing, and comprises a multimodal data monitoring module, a model framework parsing module, a docking protocol module and a reasoning optimization module. The multimodal data monitoring module is used to monitor the characteristic information of the multimodal data, the model framework parsing module is used to parse the framework information of the training model and obtain the characteristic information of the training process, the docking protocol module is used to set the docking mode with the training system and the adjustment mode of the training model, and the reasoning optimization module optimizes the training model based on the characteristic information and the modification mode. The system can be docked with the model training system, monitor and analyze the training process of the model training system, perform reasoning optimization on the training framework, and improve the training efficiency of the docking system.

Description

Technical Field

[0001] The present invention relates to the field of electronic digital data processing, and in particular to an adaptive training and reasoning performance optimization system for a multi-modal large model. Background Art

[0002] With the development of artificial intelligence technology, various large model training systems are gradually applied to various occasions. In the existing technology, by training large models, the parameters in the model can be automatically optimized to achieve the final model function, but the optimization of the model framework itself still needs to be done manually. Therefore, a system that can optimize the model framework is needed, which can automatically complete the model optimization by docking with the training system and improve the training efficiency of the training system.

[0003] The foregoing discussion of the background art is intended only to facilitate an understanding of the present invention. This discussion does not acknowledge or admit that any of the material referred to is part of the common general knowledge.

[0004] Now many model optimization systems have been developed. After a lot of searching and reference, it is found that the existing model optimization systems are such as the system disclosed in the publication number CN116561542B. These system methods generally include: the data acquisition module acquires multimodal data based on the data acquisition configuration; the data evaluation module determines the degree of matching between the multimodal data and the large language model based on the model evaluation standard configuration matching different modal data to obtain a data quality evaluation report; the model training module adjusts the model structure and / or training hyperparameters of the large language model according to the data quality evaluation report, and generates model training demand information; the model training feedback module adds the collection adjustment method for multimodal data in the data acquisition configuration based on the model training demand information. However, the optimization direction of this system is training data, and the model itself is not optimized, so it is not suitable for model training systems in various occasions. Summary of the invention

[0005] The purpose of the present invention is to propose an adaptive training and reasoning performance optimization system for a multimodal large model in response to the existing deficiencies.

[0006] The present invention adopts the following technical solution:

[0007] An adaptive training and reasoning performance optimization system for a multimodal large model, including a multimodal data monitoring module, a model framework parsing module, a docking protocol module and a reasoning optimization module;

[0008] The multimodal data monitoring module is used to monitor the characteristic information of the multimodal data, the model framework parsing module is used to parse the framework information of the training model and obtain the characteristic information of the training process, the docking protocol module is used to set the docking method with the training system and the adjustment method of the training model, and the reasoning optimization module optimizes the training model based on the characteristic information and the modification method;

[0009] The multimodal data monitoring module includes a data monitoring unit and a data matching statistics unit, wherein the data monitoring unit is used to monitor the original multimodal data directly input into the training system and the training data input into the training model, and the data matching statistics unit is used to match the multimodal data with the training data and count the input features;

[0010] The model framework parsing module includes a model detection unit, a framework reproduction unit and a training monitoring unit. The model detection unit is used to detect the framework information of the training model. The framework reproduction unit sets the framework network based on the detection result and sets the storage area based on each network node. The training monitoring unit monitors the training process of the training model and stores the monitoring data in the corresponding node storage area.

[0011] The docking protocol module includes a system docking unit and a model protocol unit, wherein the system docking unit is used to set the connection mode with the corresponding interface of the training system, and the model protocol unit is used to record the adjustment mode of the training model;

[0012] The inference optimization module includes an inference analysis unit and an optimization output unit. The inference analysis unit optimizes and analyzes the data in the node storage area and obtains an optimization plan. The optimization output unit converts the optimization plan based on the protocol content in the model protocol unit and outputs it to the training system.

[0013] Further, the data monitoring unit includes an initial monitoring processor, a training monitoring processor and a data replication processor, wherein the initial monitoring processor is used to interface with the data input interface of the training system and monitor the input data, the training detection processor is used to interface with the data input interface of the training model and monitor the input data, and the data replication processor is used to replicate the monitored data;

[0014] Further, the reasoning and analysis unit includes a data parsing processor, a performance evaluation processor and a strategy generation processor, wherein the data parsing processor is used to perform preliminary parsing of the data in the storage space, the performance evaluation processor is used to evaluate the forward training capability of each activation function, and the strategy generation processor generates an optimization strategy based on the evaluation result;

[0015] Furthermore, the performance evaluation processor calculates the convergence degree Vc of each node according to the following formula:

[0016] ;

[0017] Among them, n is the number of data in the storage space, x i Represents the i-th data in the storage space, is the mean, is the standard deviation;

[0018] The performance evaluation processor calculates the node clipping degree Cr according to the following formula:

[0019] ;

[0020] in, represents the convergence degree of the i-th parent node, represents the convergence degree of the ith lower-level node, N1 is the number of upper-level nodes, N2 is the number of lower-level nodes, is the input interference parameter;

[0021] Furthermore, the optimization output unit includes a protocol parsing processor and an output conversion processor. The protocol parsing processor is used to obtain and parse the protocol information in the model protocol unit. The output conversion processor converts the information of the node to be deleted into corresponding optimization data based on the protocol parsing result and sends it to the docked training system.

[0022] The beneficial effects achieved by the present invention are:

[0023] This system analyzes the changes in parameters during model training, obtains the activation function information with low efficiency and deletes and reconstructs it to improve the overall training efficiency of the model. This system can be connected with the training system to achieve model optimization in multiple occasions and has universal applicability. This system also adjusts the efficiency evaluation of the activation function according to the characteristic information of each modal data in the input data, making the model reconstruction more targeted.

[0024] To further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall structural framework of the present invention;

[0026] Figure 2 This is a schematic diagram of the model framework analysis module of the present invention;

[0027] Figure 3This is a schematic diagram of the data monitoring unit of the present invention;

[0028] Figure 4 It is a schematic diagram of the reasoning and analysis unit of the present invention;

[0029] Figure 5 This is a schematic diagram of optimizing the output unit structure of the present invention. DETAILED DESCRIPTION

[0030] The following is an explanation of the embodiments of the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. It is stated in advance. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention.

[0031] Embodiment 1: This embodiment provides a multi-modal large model adaptive training and reasoning performance optimization system, combined with Figure 1 , including multimodal data monitoring module, model framework parsing module, docking protocol module and reasoning optimization module;

[0032] The multimodal data monitoring module is used to monitor the characteristic information of the multimodal data, the model framework parsing module is used to parse the framework information of the training model and obtain the characteristic information of the training process, the docking protocol module is used to set the docking method with the training system and the adjustment method of the training model, and the reasoning optimization module optimizes the training model based on the characteristic information and the modification method;

[0033] The multimodal data monitoring module includes a data monitoring unit and a data matching statistics unit, wherein the data monitoring unit is used to monitor the original multimodal data directly input into the training system and the training data input into the training model, and the data matching statistics unit is used to match the multimodal data with the training data and count the input features;

[0034] The model framework parsing module includes a model detection unit, a framework reproduction unit and a training monitoring unit. The model detection unit is used to detect the framework information of the training model. The framework reproduction unit sets the framework network based on the detection result and sets the storage area based on each network node. The training monitoring unit monitors the training process of the training model and stores the monitoring data in the corresponding node storage area.

[0035] The docking protocol module includes a system docking unit and a model protocol unit, wherein the system docking unit is used to set the connection mode with the corresponding interface of the training system, and the model protocol unit is used to record the adjustment mode of the training model;

[0036] The inference optimization module includes an inference analysis unit and an optimization output unit. The inference analysis unit optimizes and analyzes the data in the node storage area and obtains an optimization plan. The optimization output unit converts the optimization plan based on the protocol content in the model protocol unit and outputs it to the training system.

[0037] The data monitoring unit includes an initial monitoring processor, a training monitoring processor and a data replication processor, wherein the initial monitoring processor is used to interface with the data input interface of the training system and monitor the input data, the training detection processor is used to interface with the data input interface of the training model and monitor the input data, and the data replication processor is used to replicate the monitored data;

[0038] The reasoning and analysis unit includes a data parsing processor, a performance evaluation processor and a strategy generation processor, wherein the data parsing processor is used to perform preliminary parsing of the data in the storage space, the performance evaluation processor is used to evaluate the forward training capability of each activation function, and the strategy generation processor generates an optimization strategy based on the evaluation result;

[0039] The performance evaluation processor calculates the convergence degree Vc of each node according to the following formula:

[0040] ;

[0041] Among them, n is the number of data in the storage space, x i Represents the i-th data in the storage space, is the mean, is the standard deviation;

[0042] The performance evaluation processor calculates the node clipping degree Cr according to the following formula:

[0043] ;

[0044] in, represents the convergence degree of the i-th parent node, represents the convergence degree of the ith lower-level node, N1 is the number of upper-level nodes, N2 is the number of lower-level nodes, is the input interference parameter;

[0045] The optimization output unit includes a protocol parsing processor and an output conversion processor. The protocol parsing processor is used to obtain and parse the protocol information in the model protocol unit. The output conversion processor converts the information of the node to be deleted into corresponding optimization data based on the protocol parsing result and sends it to the docked training system.

[0046] Embodiment 2: This embodiment includes all the contents of Embodiment 1, and provides a multimodal large model adaptive training and reasoning performance optimization system, including a multimodal data monitoring module, a model framework parsing module, a docking protocol module and a reasoning optimization module;

[0047] The multimodal data monitoring module is used to monitor the characteristic information of the multimodal data, the model framework parsing module is used to parse the framework information of the training model and obtain the characteristic information of the training process, the docking protocol module is used to set the docking method with the training system and the adjustment method of the training model, and the reasoning optimization module optimizes the training model based on the characteristic information and the modification method;

[0048] The multimodal data monitoring module includes a data monitoring unit and a data matching statistics unit, wherein the data monitoring unit is used to monitor the original multimodal data directly input into the training system and the training data input into the training model, and the data matching statistics unit is used to match the multimodal data with the training data and count the input features;

[0049] Combination Figure 2 The model framework parsing module includes a model detection unit, a framework reproduction unit and a training monitoring unit. The model detection unit is used to detect the framework information of the training model. The framework reproduction unit sets the framework network based on the detection result and sets the storage area based on each network node. The training monitoring unit monitors the training process of the training model and stores the monitoring data in the corresponding node storage area.

[0050] The docking protocol module includes a system docking unit and a model protocol unit, wherein the system docking unit is used to set the connection mode of the interface corresponding to the training system, and the model protocol unit is used to record the adjustment mode of the training model;

[0051] The inference optimization module includes an inference analysis unit and an optimization output unit. The inference analysis unit optimizes and analyzes the data in the node storage area and obtains an optimization plan. The optimization output unit converts the optimization plan based on the protocol content in the model protocol unit and outputs it to the training system.

[0052] Combination Figure 3The data monitoring unit includes an initial monitoring processor, a training monitoring processor and a data replication processor, wherein the initial monitoring processor is used to interface with the data input interface of the training system and monitor the input data, the training detection processor is used to interface with the data input interface of the training model and monitor the input data, and the data replication processor is used to replicate the monitored data;

[0053] The data matching statistics unit includes a data matching processor and a feature statistics processor. The data matching processor is used to perform content matching between system input data and training input data. The feature statistics processor obtains input features based on matching result statistics.

[0054] The input features obtained by the feature statistics processor include a multimodal conversion vector and a multimodal input vector, the number of elements of the multimodal conversion vector and the multimodal input vector are both the number of modal types of the training system input data, the element value in the multimodal conversion vector represents the ratio of the corresponding modal system input data amount to the training input data amount, and the element value in the multimodal input vector represents the proportion of the corresponding modal training input data to the total training input data;

[0055] The model exploration unit includes an exploration transmission processor and a reflection analysis processor, wherein the exploration transmission processor is used to transmit an exploration data packet to the training model analysis interface, and the reflection analysis processor is used to analyze the positioning information fed back by the exploration data packet;

[0056] The detection data packet is used to circulate in the training model and detect the activation function. When the activation function is detected, a coefficient monitoring anchor point is inserted and the positioning information of the activation function and the number information of the coefficient monitoring anchor point are fed back to the reflection analysis processor. The coefficient monitoring anchor point monitors the parameter changes of the activation function during the training process and feeds back to the training monitoring unit;

[0057] The framework reproduction unit includes a network construction processor and a storage allocation processor, wherein the network construction processor constructs a node network based on the parsing result of the positioning information, and the storage allocation processor allocates corresponding storage space to each node;

[0058] Each node in the node network corresponds to an activation function in the training model. The connection between nodes is the same as the connection between activation functions. The identification number of the storage space is consistent with the number of the corresponding coefficient monitoring anchor point.

[0059] The training monitoring unit includes an anchor point receiving processor and a data synchronization processor, wherein the anchor point receiving processor is used to receive data fed back by the coefficient monitoring anchor point, and the data synchronization processor is used to synchronize the received data to the storage space of the corresponding node;

[0060] Combination Figure 4 The reasoning and analysis unit includes a data parsing processor, a performance evaluation processor and a strategy generation processor, wherein the data parsing processor is used to perform preliminary parsing of the data in the storage space, the performance evaluation processor is used to evaluate the forward training capability of each activation function, and the strategy generation processor generates an optimization strategy based on the evaluation result;

[0061] The analysis and processing process of the data in the storage space by the reasoning and analysis unit includes the following steps:

[0062] S1, the data analysis processor obtains data in a single storage space;

[0063] S2, the data analysis processor calculates the mean and standard deviation of the data in a single storage space;

[0064] S3, repeating step S1 and step S2 until the data analysis processor has processed all the data in the storage space;

[0065] S4. The performance evaluation processor calculates the convergence degree Vc of each node according to the following formula:

[0066] ;

[0067] Among them, n is the number of data in the storage space, x i Represents the i-th data in the storage space, is the mean, is the standard deviation;

[0068] S5. The performance evaluation processor calculates the node clipping degree Cr according to the following formula:

[0069] ;

[0070] in, represents the convergence degree of the i-th parent node, represents the convergence degree of the ith lower-level node, N1 is the number of upper-level nodes, N2 is the number of lower-level nodes, is the input interference parameter;

[0071] The input interference parameter is calculated according to the following formula:

[0072] ;

[0073] Among them, a i represents the value of the i-th element in the multimodal conversion vector, b i represents the value of the i-th element in the multimodal input vector, M is the number of elements in the multimodal conversion vector, and m iIt indicates the number of initial nodes of the i-th modal type that have inheritance with this node, M0 is the number of initial nodes;

[0074] S6. The strategy generation processor determines the number of nodes that need to be optimized, denoted as m;

[0075] S7, filter out the common nodes with the largest pruning degree and mark them as nodes to be deleted;

[0076] S8, repeat step S7 m times;

[0077] S9. The strategy generation processor calculates the density index P of the node to be deleted according to the following formula:

[0078] ;

[0079] Where d(i) represents the connection distance between the node to be deleted and the remaining i-th node to be deleted;

[0080] S10, re-mark the nodes to be deleted whose density index is greater than the threshold as immune nodes, and mark the area formed by the nodes within the distance r of the immune node as the immune area, where r is the area radius, and count the number of immune nodes, denoted as ;

[0081] S11, filter out the common nodes with the largest clipping degree in the non-immune area and mark them as nodes to be deleted;

[0082] S12, repeat Next step S11;

[0083] S13, the strategy generation processor sends the information of the node to be deleted to the optimization output unit;

[0084] The initial node refers to the node corresponding to the activation function that directly uses the training input data as the input source. If the output of activation function A is used as the input of activation function B, then the node corresponding to activation function A is the upper node of the node corresponding to activation function B, and the node corresponding to activation function B is the lower node of the node corresponding to activation function A. The node corresponding to activation function A and the node corresponding to activation function B are inherited, and the inheritance is one-way transmission;

[0085] Combination Figure 5 The optimization output unit includes a protocol parsing processor and an output conversion processor. The protocol parsing processor is used to obtain and parse the protocol information in the model protocol unit. The output conversion processor converts the to-be-deleted node information into corresponding optimization data based on the protocol parsing result and sends it to the docked training system;

[0086] The i appearing in the above text is an ordinal number used to indicate a sequence number.

[0087] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention specification and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. A multi-modal large model adaptive training and reasoning performance optimization system, characterized in that: It includes multimodal data monitoring module, model framework analysis module, docking protocol module and reasoning optimization module; The multimodal data monitoring module is used to monitor the characteristic information of the multimodal data, the model framework parsing module is used to parse the framework information of the training model and obtain the characteristic information of the training process, the docking protocol module is used to set the docking method with the training system and the adjustment method of the training model, and the reasoning optimization module optimizes the training model based on the characteristic information and the modification method; The multimodal data monitoring module includes a data monitoring unit and a data matching statistics unit, wherein the data monitoring unit is used to monitor the original multimodal data directly input into the training system and the training data input into the training model, and the data matching statistics unit is used to match the multimodal data with the training data and count the input features; The model framework parsing module includes a model detection unit, a framework reproduction unit and a training monitoring unit. The model detection unit is used to detect the framework information of the training model. The framework reproduction unit sets the framework network based on the detection result and sets the storage area based on each network node. The training monitoring unit monitors the training process of the training model and stores the monitoring data in the corresponding node storage area. The docking protocol module includes a system docking unit and a model protocol unit, wherein the system docking unit is used to set the connection mode of the interface corresponding to the training system, and the model protocol unit is used to record the adjustment mode of the training model; The inference optimization module includes an inference analysis unit and an optimization output unit. The inference analysis unit optimizes and analyzes the data in the node storage area and obtains an optimization plan. The optimization output unit converts the optimization plan based on the protocol content in the model protocol unit and outputs it to the training system. The model exploration unit includes an exploration transmission processor and a reflection analysis processor, wherein the exploration transmission processor is used to transmit an exploration data packet to the training model analysis interface, and the reflection analysis processor is used to analyze the positioning information fed back by the exploration data packet; The detection data packet is used to circulate in the training model and detect the activation function. When the activation function is detected, a coefficient monitoring anchor point is inserted and the positioning information of the activation function and the number information of the coefficient monitoring anchor point are fed back to the reflection analysis processor. The coefficient monitoring anchor point monitors the parameter changes of the activation function during the training process and feeds back to the training monitoring unit; The framework reproduction unit includes a network construction processor and a storage allocation processor, wherein the network construction processor constructs a node network based on the parsing result of the positioning information, and the storage allocation processor allocates corresponding storage space to each node; Each node in the node network corresponds to an activation function in the training model. The connection between nodes is the same as the connection between activation functions. The identification number of the storage space is consistent with the number of the corresponding coefficient monitoring anchor point. The training monitoring unit includes an anchor point receiving processor and a data synchronization processor, wherein the anchor point receiving processor is used to receive data fed back by the coefficient monitoring anchor point, and the data synchronization processor is used to synchronize the received data to the storage space of the corresponding node; The reasoning and analysis unit includes a data parsing processor, a performance evaluation processor and a strategy generation processor, wherein the data parsing processor is used to perform preliminary parsing of the data in the storage space, the performance evaluation processor is used to evaluate the forward training capability of each activation function, and the strategy generation processor generates an optimization strategy based on the evaluation result; The analysis and processing process of the data in the storage space by the reasoning and analysis unit includes the following steps: S1, the data analysis processor obtains data in a single storage space; S2, the data analysis processor calculates the mean and standard deviation of the data in a single storage space; S3, repeating step S1 and step S2 until the data analysis processor has processed all the data in the storage space; S4. The performance evaluation processor calculates the convergence degree Vc of each node according to the following formula: ； Among them, n is the number of data in the storage space, x i Represents the i-th data in the storage space, is the mean, is the standard deviation; S5. The performance evaluation processor calculates the node clipping degree Cr according to the following formula: ； in, represents the convergence degree of the i-th parent node, represents the convergence degree of the ith lower-level node, N1 is the number of upper-level nodes, N2 is the number of lower-level nodes, is the input interference parameter; The input interference parameter is calculated according to the following formula: ； Among them, a i represents the value of the i-th element in the multimodal conversion vector, b i represents the value of the i-th element in the multimodal input vector, M is the number of elements in the multimodal conversion vector, and m i It indicates the number of initial nodes of the i-th modal type that have inheritance with this node, M0 is the number of initial nodes; S6. The strategy generation processor determines the number of nodes that need to be optimized, denoted as m; S7, filter out the common nodes with the largest pruning degree and mark them as nodes to be deleted; S8, repeat step S7 m times; S9. The strategy generation processor calculates the density index P of the node to be deleted according to the following formula: ； Where d(i) represents the connection distance between the node to be deleted and the remaining i-th node to be deleted; S10, re-mark the nodes to be deleted whose density index is greater than the threshold as immune nodes, and mark the area formed by the nodes within the distance r of the immune node as the immune area, where r is the area radius, and count the number of immune nodes, denoted as ; S11, filter out the common nodes with the largest clipping degree in the non-immune area and mark them as nodes to be deleted; S12, repeat Next step S11; S13, the strategy generation processor sends the information of the node to be deleted to the optimization output unit; The initial node refers to the node corresponding to the activation function that directly uses the training input data as the input source. If the output of activation function A is used as the input of activation function B, then the node corresponding to activation function A is the upper node of the node corresponding to activation function B, and the node corresponding to activation function B is the lower node of the node corresponding to activation function A. The node corresponding to activation function A and the node corresponding to activation function B are inherited, and the inheritance is one-way transmission; The optimization output unit includes a protocol parsing processor and an output conversion processor. The protocol parsing processor is used to obtain and parse the protocol information in the model protocol unit. The output conversion processor converts the to-be-deleted node information into corresponding optimization data based on the protocol parsing result and sends it to the docked training system. Here, i is an ordinal number used to represent the sequence number.

2. The adaptive training and reasoning performance optimization system for a multimodal large model as claimed in claim 1, characterized in that: The data monitoring unit includes an initial monitoring processor, a training monitoring processor and a data replication processor. The initial monitoring processor is used to connect with the data input interface of the training system and monitor the input data. The training monitoring processor is used to connect with the data input interface of the training model and monitor the input data. The data replication processor is used to replicate the monitored data.

Images