Inference method based on large model and related equipment

Abstract

The invention discloses an inference method based on a large model and related equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining scene picture data in response to an inference instruction, carrying out the quantization operation of the scene picture data based on a quantization mode of the corresponding granularity of a preset large language model, and obtaining target image data, based on the reasoning quantification process of the preset large language model after the deployment mode is changed, reasoning quantification is conducted on the target image data, a reasoning quantification result is obtained, and the calculated amount of the corresponding reasoning quantification process after the deployment mode is changed is smaller than that of the reasoning quantification process before the deployment mode is changed. According to the method, the preset large language model is deployed in the deployment mode corresponding to the reasoning quantification process with smaller calculation amount, so that the calculation amount in the reasoning quantification process is reduced, and the reasoning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model-based reasoning method and related equipment. Background Art

[0002] With the rapid development of science and technology, large language models (LLMs) have shown excellent performance and demonstrated excellent reasoning capabilities in many tasks. When using large language models to perform various tasks, reasoning calculations are performed on the data to obtain reasoning results.

[0003] In the actual reasoning process, using a large language model for reasoning calculations requires a lot of hardware resources. As a result, when a large language model is used to perform reasoning calculations on data on edge devices, the computing efficiency is very low.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are related technologies. Summary of the invention

[0005] The main purpose of this application is to provide a large model-based reasoning method and related equipment, aiming to solve the technical problem of low computing efficiency when using a large language model to perform reasoning calculations on data on edge devices.

[0006] To achieve the above objectives, the present application proposes a reasoning method based on a large model, the method comprising: Responding to the inference instruction, acquiring scene image data; Based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data; Based on the inference quantization process of the preset large language model after the deployment mode is changed, the target image data is inferred and quantized to obtain an inference and quantization result, wherein the inference and quantization process corresponding to the change in the deployment mode has a smaller computational complexity than the inference and quantization process before the change in the deployment mode.

[0007] In one embodiment, the inference quantization process of the preset large language model after the deployment mode is changed only performs one multiplication operation and one shift operation on the target image data.

[0008] In one embodiment, the deployment method splits the quantization hyperparameters of the activation value to convert the first multiplication operation corresponding to the quantization hyperparameters of the activation value into a shift operation, and merges the balance factor and the quantization hyperparameters of the weight value obtained after the split to obtain a new second multiplication operation corresponding to the quantization hyperparameters of the weight value.

[0009] In one embodiment, the inference quantization process based on the preset large language model after the deployment mode is changed, before the step of performing inference quantization on the target image data to obtain the inference quantization result, further includes: Get sample image data; The sample image data is input into the preset large language model to obtain the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value corresponding to the deployment mode before the deployment mode is changed.

[0010] In one embodiment, the step of performing a quantization operation on the scene image data based on a quantization method of a corresponding granularity of a preset large language model to obtain target image data further includes: Based on a preset granularity, the scene image data is grouped to obtain multiple groups of initial image data; Determine the bit width type of each group of initial picture data to determine the quantization interval corresponding to each group of initial picture data; Based on the quantization interval, the corresponding group of initial picture data is quantized to obtain target image data.

[0011] In one embodiment, the step of quantizing the corresponding group of initial picture data based on the quantization interval to obtain target image data includes: quantizing the corresponding group of initial picture data based on the quantization interval; Based on the clip function and the bit width type, the quantized initial picture data is adjusted to obtain target image data.

[0012] In addition, to achieve the above purpose, the present application also proposes a large model-based reasoning device, the large model-based reasoning device comprising: An acquisition module, the acquisition module is used to acquire scene image data in response to the inference instruction; A quantization module, the quantization module is used to perform a quantization operation on the scene image data based on a quantization method of a corresponding granularity of a preset large language model to obtain target image data; An inference module, wherein the inference module is used to perform inference quantization on the target image data based on the inference quantization process of the preset large language model after the deployment mode is changed, and obtain an inference quantization result, wherein the inference quantization process corresponding to the change in the deployment mode has a smaller computational amount than the inference quantization process before the change in the deployment mode.

[0013] In one embodiment, the large model-based reasoning device may also implement: After the deployment mode is changed, the inference quantization process of the preset large language model only performs one multiplication operation and one shift operation on the target image data.

[0014] In one embodiment, the large model-based reasoning device may also implement: The deployment method splits the quantization hyperparameters of the activation value to convert the first multiplication operation corresponding to the quantization hyperparameters of the activation value into a shift operation, and merges the balance factor and the quantization hyperparameters of the weight value obtained after the split to obtain a new second multiplication operation corresponding to the quantization hyperparameters of the weight value.

[0015] In one embodiment, the large model-based reasoning device further includes a confirmation module, and the confirmation module includes: An acquisition unit, used for acquiring sample image data; An input unit is used to input the sample image data into the preset large language model to obtain the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value corresponding to the deployment mode before the deployment mode is changed.

[0016] In one embodiment, the quantization module includes: A grouping unit, used to group the scene image data based on a preset granularity to obtain multiple groups of initial image data; A determination unit, used to determine the bit width type of each group of initial picture data, so as to determine the quantization interval corresponding to each group of initial picture data; The first quantization unit is used to quantize the corresponding group of initial picture data based on the quantization interval to obtain target image data.

[0017] In one embodiment, the quantization module further includes: A second quantization unit, configured to quantize a corresponding group of initial picture data based on the quantization interval; The rounding unit is used to adjust the quantized initial picture data based on the clip function and the bit width type to obtain the target image data.

[0018] In addition, to achieve the above-mentioned objectives, the present application also proposes a large model-based reasoning device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the large model-based reasoning method as described above.

[0019] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the large model-based reasoning method described above are implemented.

[0020] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the large model-based reasoning method as described above.

[0021] One or more technical solutions proposed in this application have at least the following technical effects: The present application proposes an inference method and related equipment based on a large model. In the related technology, when a large language model is used on an edge device to perform inference calculations on data, the computational efficiency is very low. In the present application, first, in response to an inference instruction, scene image data is acquired, and then, based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data. Finally, based on the inference quantization process of the preset large language model after the deployment method is changed, the target image data is inferred and quantized to obtain an inference quantization result, wherein the inference quantization process corresponding to the changed deployment method has a smaller computational amount than the inference quantization process before the deployment method is changed.

[0022] It should be noted that edge devices refer to a type of device with processing power and network connection capabilities, which are mainly used to connect IoT terminal devices and cloud services. In the IoT, edge devices play the role of a transfer station. As a result, the data processing capabilities of edge devices are relatively weak. When using large language models on edge devices to perform inference calculations on data, the computing efficiency is very low.

[0023] It can be understood that in the present application, a preset large language model is deployed based on the deployment method corresponding to the inference quantization process with smaller computational complexity using a large language model inference device, thereby reducing the amount of computation in the inference quantization process. As a result, the amount of computation when using the large language model to perform inference calculations on data on the edge device is reduced, thereby improving the inference efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A flowchart diagram of the first embodiment of the reasoning method based on a large model of the present application; Figure 2A schematic diagram of the reasoning process provided for the first embodiment of the reasoning method based on a large model of the present application; Figure 3 A flowchart diagram of the second embodiment of the reasoning method based on a large model of the present application; Figure 4 This is a schematic diagram of the module structure of the reasoning device based on the large model in the embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the large model-based reasoning method in the embodiment of the present application.

[0027] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0028] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0029] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0030] The main solutions of the embodiments of this application are: In this embodiment, for ease of description, the following description is made using a large model-based reasoning device as the execution subject.

[0031] Due to related technologies, in the actual inference calculation process, due to the large scale and huge computing cost of LLMs, the computational complexity of the inference calculation is very high. As a result, when a large language model is used on the edge device to perform inference calculation on data, the computational efficiency is very low.

[0032] The present application provides a solution, so that: in response to an inference instruction, scene image data is obtained, based on the quantization method of the corresponding granularity of the preset large language model, a quantization operation is performed on the scene image data to obtain target image data, based on the inference quantization process of the preset large language model after the deployment method is changed, the target image data is inferred and quantized to obtain an inference quantization result, wherein the inference quantization process corresponding to the changed deployment method has a smaller amount of computation than the inference quantization process before the deployment method is changed. The present application deploys the preset large language model using a deployment method corresponding to an inference quantization process with a smaller amount of computation, reduces the amount of computation in the inference quantization process, and improves the efficiency of inference.

[0033] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a reasoning device based on a large model, etc. The following takes reasoning calculation as an example to illustrate this embodiment and the following embodiments.

[0034] Based on this, the embodiment of the present application provides a reasoning method based on a large model, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the large model-based reasoning method of the present application.

[0035] In this embodiment, the large model-based reasoning method includes steps S100 to S300: Step S100, in response to the inference instruction, acquiring scene image data; In this embodiment, the execution subject is an inference device based on a large model, which can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions.

[0036] In this embodiment, the specific application scenario may also be: refer to Figure 2 In the kitchen detection scenario, after the large model-based reasoning device receives an instruction to infer whether the chef's behavior is illegal, the large model-based reasoning device obtains the kitchen video data (scene image data) collected by the visual sensor in the kitchen. Furthermore, the large model-based reasoning device uses the preset large language model after the deployment method is adjusted to infer the scene image data, obtains the reasoning result, and judges whether the chef's behavior is illegal based on the reasoning result.

[0037] In this embodiment, the specific application scenario may also be: In the bank video surveillance scenario, after the big model-based reasoning device receives an instruction to infer whether the behavior of bank insiders is illegal, the big model-based reasoning device then obtains the bank lobby video data (scene image data) collected by the visual sensors in the bank. Furthermore, the big model-based reasoning device uses the preset big language model with adjusted deployment mode to infer the scene image data to obtain the reasoning result, and based on the reasoning result, determines whether the behavior of bank insiders is illegal.

[0038] It is understandable that in this application, the specific reasoning scenario can be any intelligent scenario, such as video surveillance (abnormal behavior detection), customer service dialogue, machine translation, automatic video generation and other tasks. For example, in the intelligent recognition system of the kitchen, it is necessary to monitor abnormalities in real time, such as whether the chef's movements are standardized, whether he is wearing a chef's hat, whether the trash can is overflowing, and whether there are animals such as cockroaches and rats.

[0039] It should be noted that the scene image data is image data collected under the corresponding scene. For example, in a customer service conversation scenario, the scene image data may be screenshot data of the customer service conversation record; in a scenario of detecting whether the kitchen trash can is overflowing, the scene image data may be kitchen trash can image data collected by a visual sensor.

[0040] Step S200, based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data; It should be noted that a Large Language Model (LLM) is an artificial intelligence model designed to understand and generate human language. A large language model is trained on large-scale text data. A large language model is a deep learning model containing billions or even hundreds of billions of parameters. They are able to perform a wide range of natural language processing tasks, such as text summarization and translation. Large language models are characterized by their large scale. They are trained on large amounts of text data to learn complex patterns in language data. For example, the GPT-3 model has 45TB of training data, while the entire Wikipedia data only accounts for 0.6% of its training data.

[0041] It should be noted that particle size is an important parameter for measuring particle size, and it is widely used in many fields and industrial applications. In this application, particle size is used to indicate the size of each set of data.

[0042] Specifically, based on the quantization method of the corresponding granularity of the preset large language model, the step of performing a quantization operation on the scene image data to obtain the target image data also includes steps S210 to S230: Step S210, grouping the scene image data based on a preset granularity to obtain multiple groups of initial image data; It should be noted that the granularity is set in advance. In the present application, the granularity is used to group the scene image data.

[0043] For example, there are 100 data in the scene image data, and the preset granularity is 20. Then the inference device based on the large model divides the 100 data into 5 groups to obtain 5 groups of initial image data, each of which contains 20 data.

[0044] It should also be noted that, in the present application, each set of initial image data corresponds to a quantization hyperparameter of a weight value and a quantization hyperparameter of an activation value.

[0045] Step S220, determining the bit width type of each group of initial picture data to determine the quantization interval corresponding to each group of initial picture data; It should be noted that the bit width type refers to the data type of each set of initial picture data. For example, the bit width corresponding to the binary number 00001100 is 8, and the bit width corresponding to the binary number 0000110000001100 is 16.

[0046] It should be noted that the quantization interval calculation formula needs to be used to determine the quantization interval corresponding to each group of initial picture data. Specifically, the quantization interval calculation formula is:

[0047] Where N represents the bit width type of each set of initial picture data. is the maximum value in each set of initial image data, It is the quantization interval and also the quantization hyperparameter.

[0048] It should be noted that the quantization interval refers to the time or space distance between two adjacent quantization points determined when converting a continuous variable into a discrete variable during the quantization process.

[0049] Among them, quantization is the process of mapping the infinite number of values ​​of a continuous signal to the finite number of values ​​of a discrete signal. In this process, the continuous signal value is converted into a finite precision value through quantization, usually represented by binary.

[0050] It is understandable that the quantization interval determines the accuracy with which a continuous signal is mapped to a discrete value set. The smaller the quantization interval, the smaller the quantization noise, but too many quantization values ​​will increase the complexity and economic cost of the system.

[0051] Step S230: quantize the corresponding group of initial picture data based on the quantization interval to obtain target image data.

[0052] It should be noted that a quantization formula is required to quantize the corresponding initial image data. Specifically, the quantization formula is:

[0053] in, is the target image data, and x is the initial image data.

[0054] It is understood that in this application, It can be expressed as " x Compared with the direct mapping method in related quantization techniques, this application uses In this way, the quantization interval is smaller and a more accurate quantization result is obtained.

[0055] Specifically, the step of quantizing the corresponding group of initial picture data based on the quantization interval to obtain the target image data includes steps S231-S232: Step S231, quantizing the corresponding group of initial picture data based on the quantization interval; It is understood that quantizing the corresponding set of initial image data means calculating , the calculated result should be in the interval [ , ].

[0056] Step S232: Based on the clip function and the bit width type, the quantized initial picture data is adjusted to obtain target image data.

[0057] It should be noted that the clip function is used to limit a value within a specified range. If the provided value exceeds the specified range, the clip function will set it to the maximum or minimum value of the range.

[0058] For example, when the value of N is 8, the specified range ( ) is (-127, 127), if at this time The result is 128, then the inference device based on the large model uses the clip function to modify 128 to 127; if The result is 126, then the inference device based on the large model will not modify 126.

[0059] It is understandable that when the value of N is 8, The result range is [-128, 128], and there are 256 binary numbers with a bit width of 8. The result range is 257 integers, which exceeds the binary number corresponding to the bit width type. Then, use the clip function to The results are adjusted.

[0060] Step S300, based on the inference quantization process of the preset large language model after the deployment mode is changed, the target image data is inferred and quantized to obtain an inference and quantization result, wherein the inference and quantization process corresponding to the changed deployment mode has a smaller computational complexity than the inference and quantization process before the changed deployment mode.

[0061] It should be noted that, after the deployment mode is changed, the inference quantization process of the preset large language model only performs one multiplication operation and one shift operation on the target image data.

[0062] It should be noted that the deployment method splits the quantization hyperparameters of the activation value to convert the first multiplication operation corresponding to the quantization hyperparameters of the activation value into a shift operation, and merges the balance factor and the quantization hyperparameters of the weight value obtained after the split to obtain a new second multiplication operation corresponding to the quantization hyperparameters of the weight value.

[0063] It can be understood that before changing the deployment method, the inference quantization process of the preset large language model performs two multiplication operations on the target image data. After changing the deployment method, the first multiplication operation corresponding to the quantization hyperparameter of the large activation value is converted into a shift operation, and the quantization hyperparameters of the balance factor and the weight value obtained after the split are merged to obtain the new second multiplication operation corresponding to the quantization hyperparameter of the weight value, converting two multiplication operations into one multiplication operation and one shift operation, effectively reducing the amount of calculation and improving the reasoning efficiency.

[0064] Furthermore, the inference device based on the large model performs two multiplication calculations on the data, which has a high calculation complexity, resulting in low calculation efficiency and occupying a large amount of hardware resources. When the inference device based on the large model performs a shift operation, it can directly obtain the shift operation result and occupy fewer resources. At the same time, one multiplication calculation is used instead of two multiplication calculations, which effectively reduces the intermediate bit width pressure, occupies less hardware resources, and effectively improves the calculation efficiency.

[0065] For example, for two two-digit numbers 1.1 and 1.1, multiply 1.1 by 1.1 to get a three-digit number 1.21. At this point, if we continue to use the two-digit number (1.2) to represent the product, there will be a difference, affecting the precision of the value. After the two numbers are multiplied, in order to ensure the precision of the data, a larger data type is needed to represent the product, which in turn occupies a larger intermediate bit width, more storage space and hardware resources.

[0066] In addition, it should be noted that the bit shift operation is a basic data processing method in computer science. The bit shift operation is mainly used to perform left or right shift operations on binary numbers. The bit shift operation can efficiently change the binary representation of a number in memory. Therefore, the bit shift operation is often used to speed up certain calculation processes. Shift operations are usually divided into left shift and right shift, and there are several different types of shifts, such as logical shift, arithmetic shift, and circular shift. For example, the binary number 00001100 (decimal value is 12) is shifted left by 2 bits, and the result is 00110000 (decimal value is 48), which is equivalent to 12x2x2=48; or, the binary number 00001100 (decimal value is 12) is shifted right by 2 bits, and the result is 00000011 (decimal value is 3), which is equivalent to 12 / (2x2)=3.

[0067] Specifically, the inference calculation formula is as follows:

[0068] Among them, y is the inference result, is the quantized weight, is the target image data, It is the quantization hyperparameter of the weight value of the preset large language model after changing the deployment mode. It is a quantization hyperparameter of the activation value of the preset large language model after the deployment mode is changed.

[0069] It should be noted that before changing the deployment method, the inference calculation formula is as follows:

[0070] in, It is the quantization hyperparameter of the weight value of the preset large language model mentioned before changing the deployment method. It is the quantization hyperparameter of the activation value of the preset large language model described before changing the deployment method.

[0071] Understandably, and corresponds to the multiplication operation, and Corresponding to the shift operation, Corresponding to the multiplication operation, the amount of calculation corresponding to the shift operation is significantly smaller than the amount of calculation corresponding to the multiplication operation. Therefore, after changing the deployment method, the amount of calculation corresponding to the reasoning process is reduced, and the reasoning efficiency is higher.

[0072] Specifically, obtain and The method is as follows: First, based on the large language model inference device, we obtain and Then, based on the large language model inference device will Split into and ,in, , is the balancing factor.

[0073] In the After splitting, and Perform the merge operation and get ,in, .

[0074] In addition, it should be noted that the calculation and The formula is as follows:

[0075] fl=bits-1-il

[0076] in, Represents the initial image data, The function is used to return the smallest integer value greater than or equal to the input parameter, for example, (5.1) = 6, bits represent The data type, for example, an 8-bit binary number, the corresponding bits is 8.

[0077] In addition, it should be noted that in the present application, the balance factor and the quantization hyperparameter of the weight value obtained after splitting the quantization hyperparameter of the activation value are merged to obtain the quantization hyperparameter of the weight value corresponding to the new second multiplication operation. At this time, the quantization hyperparameter of the merged weight value is optimized. Compared with the two optimizations required before changing the deployment method, only one optimization is required after changing the deployment method, and the optimization efficiency is higher. Moreover, one optimization acts on two parameters, which is equivalent to joint optimization of the two parameters, and the optimization effect is better.

[0078] One or more technical solutions proposed in this application have at least the following technical effects: The present application proposes an inference method and related equipment based on a large model. In the related technology, when a large language model is used on an edge device to perform inference calculations on data, the computational efficiency is very low. In the present application, first, in response to an inference instruction, scene image data is acquired, and then, based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data. Finally, based on the inference quantization process of the preset large language model after the deployment method is changed, the target image data is inferred and quantized to obtain an inference quantization result, wherein the inference quantization process corresponding to the changed deployment method has a smaller computational amount than the inference quantization process before the deployment method is changed.

[0079] It should be noted that edge devices refer to a type of device with processing power and network connection capabilities, which are mainly used to connect IoT terminal devices and cloud services. In the IoT, edge devices play the role of a transfer station. As a result, the data processing capabilities of edge devices are relatively weak. When using large language models on edge devices to perform inference calculations on data, the computing efficiency is very low.

[0080] It can be understood that in the present application, a preset large language model is deployed based on the deployment method corresponding to the inference quantization process with smaller computational complexity using a large language model inference device, thereby reducing the amount of computation in the inference quantization process. As a result, the amount of computation when using the large language model to perform inference calculations on data on the edge device is reduced, thereby improving the inference efficiency.

[0081] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 Before step S300, the large model-based reasoning method further includes steps S301-S302: Step S301, obtaining sample image data; It should be noted that the sample image data and the scene image data are data of the same scene type, and the sample image data is used to determine the quantization hyperparameters of the weight value, the quantization hyperparameters of the activation value, and the quantization weight. For example, in the scenario of detecting whether the chef's actions are standard, the scene image data is the image data of the current moment collected by the visual sensor, and the sample image data is the image data of the historical moment collected by the visual sensor.

[0082] Step S302: input the sample image data into the preset large language model to obtain the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value corresponding to the deployment mode before the deployment mode is changed.

[0083] It should be noted that the sample image data is input into the preset large language model, and the inference device based on the large model uses a statistical method to obtain the quantized hyperparameters of the activation value and the quantized hyperparameters of the weight value corresponding to the deployment method before changing the deployment method, wherein the statistics include dynamic statistics and static statistics.

[0084] It should be noted that the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value are both inference parameters of the large model. For image data of different scene types, when using the large model for inference, image data of different scene types correspond to different inference parameters.

[0085] It can be understood that in the process of deploying the big model, sample image data of the same scene type as the scene image data is input into the big model to obtain the inference parameters of the big model (the inference parameters corresponding to the inference parameters before the deployment method is changed), so that in the subsequent steps, based on the inference parameters of the big model, the inference device based on the big model uses the big model to infer the scene image data.

[0086] It should be noted that for the quantization hyperparameters of weight values, the inference device based on the large model can use static statistics to obtain the quantization hyperparameters of weight values; for the quantization hyperparameters of activation values, according to the actual scenario, the inference device based on the large model uses static statistics or dynamic statistics to obtain the quantization hyperparameters of activation values.

[0087] Among them, static statistics refers to the statistics of data at a fixed time point or time period (the change of data is independent of time), obtaining statistical results, and analyzing and describing the statistical results to obtain static statistical results. This statistical method does not involve the changes in time series data, but focuses on the characteristics of data at a specific moment or time period. Static statistics are usually used to describe characteristics such as data distribution, central tendency and degree of dispersion.

[0088] Among them, dynamic statistics refers to the statistical analysis and description of data that changes over time. This statistical method focuses on the time series characteristics of data, such as trends, seasonality, and periodicity. Dynamic statistics are usually used to reveal the changing patterns of data over time and provide a basis for prediction and decision-making.

[0089] It can be understood that in this embodiment, by using sample image data, the quantization hyperparameters of the activation values ​​and the quantization hyperparameters of the weight values ​​corresponding to the preset large language model before changing the deployment method are determined, thereby realizing the deployment of the preset large language model and providing a data basis for subsequent reasoning steps.

[0090] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the reasoning method based on the large model of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0091] This application also provides an inference device based on a large model, please refer to Figure 4 , the reasoning device based on the large model includes: An acquisition module 10, the acquisition module is used to acquire scene image data in response to an inference instruction; A quantization module 20, the quantization module is used to perform a quantization operation on the scene image data based on a quantization method of a corresponding granularity of a preset large language model to obtain target image data; An inference module 30 is used to perform inference and quantization on the target image data based on the inference and quantization process of the preset large language model after the deployment mode is changed, so as to obtain an inference and quantization result, wherein the inference and quantization process corresponding to the change in the deployment mode has a smaller computational complexity than the inference and quantization process before the change in the deployment mode.

[0092] In one embodiment, the large model-based reasoning device may also implement: After the deployment mode is changed, the inference quantization process of the preset large language model only performs one multiplication operation and one shift operation on the target image data.

[0093] In one embodiment, the large model-based reasoning device may also implement: The deployment method splits the quantization hyperparameters of the activation value to convert the first multiplication operation corresponding to the quantization hyperparameters of the activation value into a shift operation, and merges the balance factor and the quantization hyperparameters of the weight value obtained after the split to obtain a new second multiplication operation corresponding to the quantization hyperparameters of the weight value.

[0094] In one embodiment, the large model-based reasoning device further includes a confirmation module, and the confirmation module includes: An acquisition unit, used for acquiring sample image data; An input unit is used to input the sample image data into the preset large language model to obtain the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value corresponding to the deployment mode before the deployment mode is changed.

[0095] In one embodiment, the quantization module includes: A grouping unit, used to group the scene image data based on a preset granularity to obtain multiple groups of initial image data; A determination unit, used to determine the bit width type of each group of initial picture data, so as to determine the quantization interval corresponding to each group of initial picture data; The first quantization unit is used to quantize the corresponding group of initial picture data based on the quantization interval to obtain target image data.

[0096] In one embodiment, the quantization module further includes: A second quantization unit, configured to quantize a corresponding group of initial picture data based on the quantization interval; The rounding unit is used to adjust the quantized initial picture data based on the clip function and the bit width type to obtain the target image data.

[0097] The present application provides an inference device based on a large model, and the inference device based on a large model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the inference method based on the large model in the above-mentioned embodiment one.

[0098] Reference below Figure 5 , which shows a schematic diagram of the structure of a large model-based reasoning device suitable for implementing the embodiment of the present application. The large model-based reasoning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The large model-based reasoning device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0099] like Figure 5As shown, the reasoning device based on the large model may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the reasoning device based on the large model are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the large model-based reasoning device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a large model-based reasoning device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0100] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0101] The large-model-based reasoning device provided by the present application adopts the large-model-based reasoning method in the above-mentioned embodiment to solve the technical problems of reasoning calculation. Compared with the related art, the beneficial effects of the large-model-based reasoning device provided by the present application are the same as the beneficial effects of the large-model-based reasoning method provided by the above-mentioned embodiment, and the other technical features in the large-model-based reasoning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0104] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the large model-based reasoning method in the above-mentioned embodiment.

[0105] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0106] The computer-readable storage medium may be included in the reasoning device based on the large model; or may exist independently without being assembled into the reasoning device based on the large model.

[0107] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the reasoning device based on the large model, the reasoning device based on the large model: Responding to the inference instruction, acquiring scene image data; Based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data; Based on the inference quantization process of the preset large language model after the deployment mode is changed, the target image data is inferred and quantized to obtain an inference and quantization result, wherein the inference and quantization process corresponding to the change in the deployment mode has a smaller computational complexity than the inference and quantization process before the change in the deployment mode.

[0108] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0109] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0111] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large-model-based reasoning method, and can solve the technical problems of large-model-based reasoning. Compared with the related art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the large-model-based reasoning method provided in the above-mentioned embodiment, and will not be repeated here.

[0112] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned large model-based reasoning method when executed by a processor.

[0113] The computer program product provided by this application can solve the technical problem of reasoning calculation. Compared with the related art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the reasoning method based on the large model provided by the above embodiment, which will not be repeated here.

[0114] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A large model-based reasoning method, characterized in that: The inference calculation method comprises: Responding to the inference instruction, acquiring scene image data; Based on a quantization method of a corresponding granularity of a preset large language model, a quantization operation is performed on the scene image data to obtain target image data; Based on the inference quantization process of the preset large language model after the deployment mode is changed, the target image data is inferred and quantized to obtain an inference and quantization result, wherein the inference and quantization process corresponding to the change in the deployment mode has a smaller computational complexity than the inference and quantization process before the change in the deployment mode.

2. The large model-based reasoning method according to claim 1, characterized in that: After the deployment mode is changed, the inference quantization process of the preset large language model only performs one multiplication operation and one shift operation on the target image data.

3. The large model-based reasoning method according to claim 2, characterized in that: The deployment method splits the quantization hyperparameters of the activation value to convert the first multiplication operation corresponding to the quantization hyperparameters of the activation value into a shift operation, and merges the balance factor and the quantization hyperparameters of the weight value obtained after the split to obtain a new second multiplication operation corresponding to the quantization hyperparameters of the weight value.

4. The large model-based reasoning method according to claim 1, characterized in that: The inference and quantization process based on the preset large language model after the deployment mode is changed, before the step of performing inference and quantization on the target image data to obtain the inference and quantization result, it also includes: Get sample image data; The sample image data is input into the preset large language model to obtain the quantization hyperparameters of the activation value and the quantization hyperparameters of the weight value corresponding to the deployment mode before the deployment mode is changed.

5. The large model-based reasoning method according to claim 1, characterized in that: The step of performing a quantization operation on the scene image data based on a quantization method of a corresponding granularity of a preset large language model to obtain target image data further includes: Based on a preset granularity, the scene image data is grouped to obtain multiple groups of initial image data; Determine the bit width type of each group of initial picture data to determine the quantization interval corresponding to each group of initial picture data; Based on the quantization interval, the corresponding group of initial picture data is quantized to obtain target image data.

6. The large model-based reasoning method according to claim 5, characterized in that: The step of quantizing the corresponding group of initial picture data based on the quantization interval to obtain target image data includes: quantizing the corresponding group of initial picture data based on the quantization interval; Based on the clip function and the bit width type, the quantized initial picture data is adjusted to obtain target image data.

7. A reasoning device based on a large model, characterized in that: The large model-based reasoning device comprises: An acquisition module, the acquisition module is used to acquire scene image data in response to the inference instruction; A quantization module, the quantization module is used to perform a quantization operation on the scene image data based on a quantization method of a corresponding granularity of a preset large language model to obtain target image data; An inference module, wherein the inference module is used to perform inference quantization on the target image data based on the inference quantization process of the preset large language model after the deployment mode is changed, and obtain an inference quantization result, wherein the inference quantization process corresponding to the change in the deployment mode has a smaller computational amount than the inference quantization process before the change in the deployment mode.

8. A large model-based reasoning device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large model-based reasoning method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large model-based reasoning method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the large model-based reasoning method according to any one of claims 1 to 6 are implemented.

Images