The invention discloses an instruction
fine tuning method for a quantized multi-mode
large model. The instruction
fine tuning method comprises the following steps: step 1, scale learning of quantized
perception; 2, a multi-mode warm-up strategy used for scale learning; and step 3, carrying out specific
task learning by using the quantized multi-
modal large model. According to the method, scale learning of quantization
perception is adopted, quantization errors of
abnormal positions can be minimized in a self-adaptive manner, and particularly, the quantization errors in each quantization group can be effectively reduced under the condition that certain position activation represents abnormal value features; therefore, the problem that quantization errors caused by outliers cannot be effectively processed in the prior art is solved; according to the method, through a uniform quantification method, weight
cutting is adopted to relieve quantification difficulty occurring in a language task; according to the method, after iteration reaches a certain number of times, a multi-
modal instruction data set is replaced with a mixed
data set, so that language data is gradually introduced while multi-
modal task performance is maintained, and
overfitting is avoided.