The application discloses a kind of based on end side's big model key value cache dynamic optimization method and
system, method includes: S1, the context resource of multiple
inference sequences is centrally controlled, and corresponding context space is pre-allocated;S2, whether the currently available context space satisfies the allocation demand of
inference sequence is judged, if it satisfies, then the intelligent context allocation is carried out to
inference sequence, if it does not satisfy, then whether the remaining memory space of end side equipment is enough is judged, according to the judgment result, step S3 or S4 is executed;S3, if space is enough, then the available context space is increased by dynamic expansion, and the context
space allocation and context update of inference sequence are re-performed;S4, if space is insufficient, first carry out context
recovery, then discard and retain context by sequence
perception sliding window mechanism, and execute context update after allocation is completed.The application can realize the efficient inference of big model under the condition of end side limited resource.