Method, device, electronic device and storage medium for nl2sql modeling

Abstract

The application relates to a NL2SQL modeling method, device, electronic equipment and storage medium, the NL2SQL modeling method comprises the following steps: S1, data processing, a large number of unlabeled samples are generated after data processing, and a UDA technology is used to realize labeled training sample data set; S2, NL-to-SQL model training, a TernaryBERT model is used to quantize a weight layer and an activation layer; S3, model compression and generation module, a pre-training knowledge distillation mode is used to further compress and process the model, meanwhile, a bat algorithm is used for parameter training and intelligent search, and an optimal distillation network structure is found; S4, dialogue management, the model is fused with a preset SQL paradigm, a complete database query SQL is associated and generated, then the model is interacted with an intelligent customer service interface, and dialogue management application is realized. According to the NL2SQL modeling method, non-professional operators can use the method, model training does not need large-scale labeled sample data, and the method has strong adaptability to new environments and new tasks.

Description

Technical Field

[0001] This invention relates to the field of semantic parsing in natural language processing, and more specifically, to a method, apparatus, electronic device, and storage medium for NL2SQL (Natural Language to Structured Query Language) modeling based on unsupervised domain adaptation technology and the Bat algorithm optimization. Background Technology

[0002] With the continuous development of 5G and artificial intelligence technologies, telecommunications / mobile communication operators, as modern integrated information service providers, are rapidly updating their intelligent customer service systems across various product lines under the guidance of cloud transformation and digital transformation. Regarding Natural Language (NL) data for intelligent customer service, operations personnel may not be data experts. Accessing and querying databases through NL can significantly improve the operational efficiency of intelligent customer service, and this has become a hot research topic combining artificial intelligence and intelligent customer service. Currently, big data modeling techniques mostly focus on machine learning and deep learning, which have high barriers to entry and are not suitable for non-professional operations personnel.

[0003] NL-2-SQL (Natural Language to Structured Query Language) data modeling technology for intelligent customer service enables non-professional operations personnel to interact with the database via voice, lowering the barrier to database queries and greatly improving operational efficiency.

[0004] With in-depth research into deep learning technology, NL-to-SQL (NL2SQL) data modeling techniques based on supervised learning models have achieved great success in the NLP field. However, supervised learning relies on a large amount of manually labeled data, and the models are prone to problems such as spurious associations, generalization errors, and adversarial attacks. Furthermore, patents such as a method for converting complex natural language queries to SQL based on a tree-like model (patent number: 202110183393.4) ​​and a method and system for SQL conversion based on language model encoding and multi-task decoding (patent number: 202110505064.7) all require large-scale labeled sample data for model training. In real-world learning tasks, data labeling typically requires manual intervention and related professional knowledge, which is time-consuming, labor-intensive, and costly. In addition, learning models have weak adaptability to new environments and tasks; when faced with learning environments different from the training scenario, new data needs to be labeled, and the learning model needs to be retrained.

[0005] Therefore, there is an urgent need to develop a modeling method that is suitable for non-professional operators, does not require large-scale labeled sample data for model training, and is highly adaptable to new environments and tasks. Summary of the Invention

[0006] The technical problem this invention aims to solve is how to implement a modeling method that is suitable for non-professional operations personnel, does not require large-scale labeled sample data for model training, and has strong adaptability to new environments and tasks, thereby reducing the threshold for operations personnel to query databases and improving operational efficiency.

[0007] To address the aforementioned technical problems, according to one aspect of the present invention, a method for NL2SQL modeling is provided, comprising the following steps: S1, data processing, wherein the data is product operation data; after data processing, a large number of unlabeled samples are generated; and then, through a preset query SQL paradigm template, unsupervised domain adaptation (UDA) technology is used to label the training sample dataset; S2, NL-to-SQL model training, wherein the acquired natural language samples are used for sample data training to generate a structured query language model, the model including a weight layer and an activation layer, and employing a TernaryBERT model (Ternary Bidirectional Encoder Representation from...). Transformers (TernaryBERT, a pre-trained language representation model) quantizes the weight and activation layers; S3, Model Compression and Generation Module: To compensate for the insufficient performance of the TernaryBERT model in tasks including intelligent customer service consultation due to over-quantization, a pre-trained knowledge distillation method is used to further compress the model. At the same time, to prevent the weights of the distilled network structure from getting trapped in local optima, the Bat algorithm is used for parameter training and intelligent search to find the optimal distilled network structure; S4, Dialogue Management: After training the TernaryBERT model and compressing the PKD model, preprocessing is performed, and then the model is fused with a preset SQL paradigm to associate and generate complete database query SQL, which then interacts with the intelligent customer service interface to realize dialogue management applications.

[0008] According to an embodiment of the present invention, the product operation data in step S1 may include caller ID cards and hang-up SMS messages. The data processing methods include word segmentation, removal of stop words, replacement of English punctuation, construction of question-and-answer datasets, construction of sentence vector index databases, and thus generating a massive amount of unlabeled samples.

[0009] According to an embodiment of the present invention, the NL-to-SQL model training in step S2 can use the TernaryBERT model, which quantizes the weight layer and the activation layer. The weight layer may include a linear layer and an embedding layer. The parameters of the linear layer and the embedding layer account for the majority of the total parameters of the TernaryBERT model. Therefore, the TernaryBERT model performs relatively thorough quantization of the linear layer and the embedding layer. In the quantization of the activation layer, 8-bit symmetric and asymmetric methods can be used for quantization.

[0010] Furthermore, in the activation layer quantization, the matrix multiplication in the actual inference process can be changed from 32-bit floating-point operations to int8 integer operations, thus achieving the goal of acceleration; this model achieves performance comparable to the full-precision model with only 6.7% of the parameters of the BERT model.

[0011] According to an embodiment of the present invention, the pre-training knowledge distillation in step S3 can employ two strategies, PKD-Last and PKD-Skip, to extract hidden knowledge from the hidden layers of the "teacher" model, thereby completely eliminating the "student" model's imitation of the "teacher" model's output. Specifically, the PKD-Last strategy uses the knowledge contained in the last k layers of the "teacher" model, while the PKD-Skip strategy extracts and distills the knowledge from each of the k layers of the "teacher" model.

[0012] According to an embodiment of the present invention, step S3 can use the bat algorithm for parameter training and intelligent search to find the optimal distillation network structure, which may include the following steps: S31, inputting the network encoding vector into the trained structure generator to generate the weights of the corresponding distillation network, evaluating the distillation network on the validation set, and obtaining the accuracy of the corresponding distillation network; S32, in order to search for the distillation network with the highest accuracy that satisfies specific constraints, the bat algorithm is used to search for the distillation structure model with the highest accuracy that satisfies specific constraints, wherein the specific constraints include the number of floating-point operations.

[0013] According to an embodiment of the present invention, the preprocessing in step S4 may include processing for named entity recognition, typo correction, and user sentiment analysis.

[0014] According to a second aspect of the present invention, an apparatus for NL2SQL modeling is provided, comprising:

[0015] The massive data processing module uses UDA (User-Defined Analysis) technology to learn from training sample data. UDA helps a target domain without any labeled information learn from a source domain with a small amount of labeled data, thus enabling transfer learning and low-cost acquisition of labeled data. The NL-to-SQL model training module is based on the TernaryBERT network model and implements core technologies including quantization of network weights and activation layers. The model compression and generation module uses the PKD model compression algorithm, employing PKD-Last and PKD-Skip strategies to compress data from the "teacher" model. Hidden knowledge is extracted from the hidden layers of the model, completely eliminating the "student" model's imitation of the "teacher" model's output. This reduces the number of side layers in the NL-to-SQL model, decreases the number of training parameters, and improves the model's knowledge inference speed. Simultaneously, the Bat algorithm is used for network parameter training and intelligent search to find the optimal distillation network structure. The dialogue management module, after being trained by the TernaryBERT model and compressed by the PKD model, is used to implement processes including named entity recognition, typo correction, and user sentiment analysis. It then integrates preset SQL paradigms to associate and generate complete database query SQL, which is then used for interaction through the dialogue management system and intelligent customer service interface.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and an NL2SQL modeling program stored in the memory and executable on the processor, wherein the NL2SQL modeling program, when executed by the processor, implements the steps of the NL2SQL modeling method described above.

[0017] According to a fourth aspect of the present invention, a computer storage medium is provided, wherein an NL2SQL modeling program is stored on the computer storage medium, and when the NL2SQL modeling program is executed by a processor, it implements the steps of the NL2SQL modeling method described above.

[0018] Compared with the prior art, the technical solution provided by the embodiments of the present invention can achieve at least the following beneficial effects:

[0019] 1. This invention is based on Unsupervised Domain Adaptation (UDA) technology. The main goal of UDA is to help a target domain, which has no labeled information, learn by leveraging a source domain with a large amount of labeled data. Its main steps are to learn initial features of the source and target domains through self-supervised learning, and to fix some network parameters to preserve target domain information. Then, the sample comparison knowledge from the source domain is transferred to the target domain to assist the target domain in learning class-discriminative features. Therefore, UDA technology can utilize auxiliary tasks to mine its own supervisory information from large-scale unsupervised data, and use this information to train the network, thereby learning rich, general representations valuable for downstream tasks.

[0020] 2. This invention provides a data NL-to-SQL modeling method for intelligent customer service by integrating UDA technology. It solves the technical problem that existing NL-to-SQL technologies are limited to database queries and cannot be applied to complex intelligent customer service data, lowering the barrier for operations personnel to perform database queries and improving operational efficiency. Simultaneously, the use of UDA technology enables the low-cost acquisition of a large number of model training samples, has broad application scenarios, and possesses commercial application value.

[0021] 3. This invention is based on UDA task training and NL-to-SQL modeling technology. It uses UDA technology for language model pre-training tasks, and adopts the bat algorithm for optimal selection of model compression structure. By integrating NL-to-SQL modeling technology, it realizes automatic model training based on massive unlabeled data, which greatly reduces the cost of manual data labeling.

[0022] 4. The technical solution of this invention is based on UDA technology to realize self-supervised learning of samples, effectively reducing the dependence of the Text-to-SQL model on large-scale labeled sample data. It uses TernaryBERT model training and PKD model compression method, and optimizes the compressed network structure through the bat algorithm. It integrates the preset SQL paradigms of each category to predict and output complete SQL query statements.

[0023] 5. The technical solution of this invention adopts UDA technology to realize the autonomous generation of unsupervised domain adaptation data, reduce the model's dependence on a large amount of labeled data, and is compatible with single-domain and multi-domain scenarios.

[0024] 6. The technical solution of this invention achieves an accuracy matching rate and execution correctness of over 92% in a single domain, and an accuracy matching rate of over 75% and an execution correctness of over 86% in multiple domains.

[0025] 7. The technical solution of this invention performs a new round of reinforcement learning on erroneous cases, and the system has the ability to iteratively optimize.

[0026] 8. The technical solution of this invention adopts a TernaryBERT algorithm model, which has fewer parameters, improves the model's performance, requires less memory during model deployment, and has the advantages of rapid deployment and wide application. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention, and are not intended to limit the present invention.

[0028] Figure 1 This is a schematic diagram illustrating the model framework of the NL2SQL modeling method according to an embodiment of the present invention.

[0029] Figure 2 This is a flowchart illustrating the NL2SQL modeling method according to an embodiment of the present invention.

[0030] Figure 3 This is a schematic diagram illustrating a TernaryBERT pre-trained model according to an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram illustrating two strategies of the PKD method according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the described embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a limitation of quantity, but rather indicate the presence of at least one.

[0034] Figure 1 This is a schematic diagram illustrating the model framework of the NL2SQL modeling method according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the NL2SQL modeling method according to an embodiment of the present invention.

[0035] like Figure 1 and Figure 2 As shown, the NL2SQL modeling method includes the following steps:

[0036] S1. Data processing: The data consists of product operation data. After data processing, a large number of unlabeled samples are generated. Then, UDA technology is used to label the training sample dataset through a preset query SQL template.

[0037] S2 and NL-to-SQL model training: The acquired natural language samples are used to train the model and generate a structured query language model. The model includes a weight layer and an activation layer. The TernaryBERT model is used to quantize the weight layer and the activation layer.

[0038] S3, Model Compression and Generation Module: To compensate for the insufficient performance of the TernaryBERT model in tasks including intelligent customer service business consultation due to over-quantization, a pre-trained knowledge distillation method is used to further compress the model. At the same time, to prevent the weights of the distilled network structure from getting trapped in local optima, the Bat Algorithm is used for parameter training and intelligent search to find the optimal distilled network structure.

[0039] S4. Dialogue Management: After training with the TernaryBERT model and compressing with the PKD model, the system is preprocessed, and then the model is integrated with a preset SQL paradigm to associate and generate complete database query SQL, which then interacts with the intelligent customer service interface to realize dialogue management applications.

[0040] This invention is based on Unsupervised Domain Adaptation (UDA) technology. The main goal of UDA is to help a target domain, which lacks any labeled information, learn by leveraging a source domain with abundant labeled data. Its main steps involve self-supervised learning of initial features in both the source and target domains, and fixing some network parameters to preserve target domain information. Then, the sample comparison knowledge from the source domain is transferred to the target domain to assist in learning class-discriminative features. Therefore, UDA technology can utilize auxiliary tasks to mine its own supervisory information from large-scale unsupervised data, using this information to train the network and learn rich, general representations valuable for downstream tasks.

[0041] The technical solution of this invention adopts UDA technology to achieve autonomous generation of unsupervised domain adaptation data, reduce the model's dependence on a large amount of labeled data, and is compatible with single-domain and multi-domain scenarios.

[0042] According to one or more embodiments of the present invention, the product operation data in step S1 includes caller ID cards and hang-up SMS messages. The data processing methods include word segmentation, removal of stop words, replacement of English punctuation, construction of question-and-answer datasets, construction of sentence vector index databases, and thus generating a massive amount of unlabeled samples.

[0043] Figure 3 This is a schematic diagram illustrating a TernaryBERT pre-trained model according to an embodiment of the present invention.

[0044] like Figure 3 As shown, the NL-to-SQL model training in step S2 uses the TernaryBERT model. This model quantizes the weight layer and the activation layer. The weight layer can contain linear layers and embedding layers. The parameters of the linear layers and embedding layers account for the majority of the total parameters of the TernaryBERT model. Therefore, the TernaryBERT model performs relatively thorough quantization on the linear layers and embedding layers. In the quantization of the activation layer, 8-bit symmetric and asymmetric methods are used for quantization.

[0045] The technical solution of this invention adopts a TernaryBERT algorithm model, which has fewer parameters, improves the model's performance, requires less memory during model deployment, and has the advantages of rapid deployment and wide application.

[0046] Furthermore, in the activation layer quantization, the matrix multiplication is changed from 32-bit floating-point operations to int8 integer operations during the actual inference process, achieving the goal of acceleration; this model achieves performance comparable to the full-precision model with only 6.7% of the parameters of the BERT model.

[0047] Figure 4 This is a schematic diagram illustrating two strategies of the PKD method according to an embodiment of the present invention.

[0048] like Figure 4 As shown, in step S3, pre-trained knowledge distillation (PKD) uses two strategies, PKD-Last and PKD-Skip, to extract hidden knowledge from the hidden layers of the "teacher" model, completely eliminating the "student" model's imitation of the "teacher" model's output. Specifically, the PKD-Last strategy uses the knowledge contained in the last k layers of the "teacher" model, while the PKD-Skip strategy extracts and distills the knowledge from each of the k layers of the "teacher" model.

[0049] According to one or more embodiments of the present invention, in step S3, the bat algorithm is used for parameter training and intelligent search to find the optimal distillation network structure. Step S3 includes the following steps:

[0050] S31. Input the network encoding vector into the trained structure generator to generate the weights of the corresponding distillation network. Evaluate the distillation network on the validation set to obtain the accuracy of the corresponding distillation network.

[0051] S32. In order to search for the distillation network with the highest accuracy that satisfies specific constraints, the bat algorithm is used to search for the distillation structure model with the highest accuracy that satisfies specific constraints, including the number of floating-point operations.

[0052] This invention is based on UDA technology for language model pre-training tasks. At the same time, the bat algorithm is used to optimize the model compression structure selection. It also integrates NL-to-SQL modeling technology to realize automatic model training based on massive unlabeled data, which greatly reduces the cost of manual data labeling.

[0053] According to one or more embodiments of the present invention, the preprocessing in step S4 includes processing for named entity recognition, typo correction, and user sentiment analysis.

[0054] According to a second aspect of the present invention, an apparatus for NL2SQL modeling is provided, comprising: a massive data processing module, an NL-to-SQL model training module, a model compression and generation module, and a dialogue management module.

[0055] The massive data processing module trains sample data and uses the UDA (Unsupervised Data Acquisition) technology to achieve self-supervised sample data learning. UDA is used to help the target domain, which has no labeled information, learn by leveraging the source domain with a small amount of labeled data, thereby achieving low-cost acquisition of labeled data.

[0056] The NL-to-SQL model training module is based on the TernaryBERT network model and implements core technical solutions including the quantization of network weight layers and activation layers.

[0057] The model compression and generation module adopts the PKD model compression algorithm, using two strategies, PKD-Last and PKD-Skip, to extract hidden knowledge from the hidden layers of the "teacher" model, completely eliminating the "student" model's imitation of the "teacher" model's output. This reduces the number of side layers in the NL-to-SQL model, decreases the number of parameters trained, and improves the model's knowledge inference speed. At the same time, the bat algorithm is used for network parameter training and intelligent search to find the optimal distillation network structure.

[0058] After being trained by the TernaryBERT model and compressed by the PKD model, the dialogue management module is used to perform processing including named entity recognition, typo correction, and user sentiment analysis. It then integrates preset SQL paradigms to associate and generate complete database query SQL, and finally interacts with the dialogue management system and intelligent customer service interface.

[0059] The technical solution of this invention is based on UDA technology to achieve self-supervised learning of samples, which effectively reduces the dependence of the Text-to-SQL model on large-scale labeled sample data. It uses TernaryBERT model training and PKD model compression method, and optimizes the compressed network structure through the bat algorithm. It integrates the preset SQL paradigms of each category to predict and output complete SQL query statements.

[0060] According to the technical solution of the present invention, it can be effectively deployed and applied in intelligent operation scenarios such as intelligent customer service application scenarios and intelligent query system application scenarios, reducing the usage threshold for operation personnel and ensuring uninterrupted service 24 / 7.

[0061] According to another aspect of the present invention, an apparatus for NL2SQL modeling is provided, comprising: a memory, a processor, and an NL2SQL modeling program stored in the memory and executable on the processor, wherein the NL2SQL modeling program, when executed by the processor, implements the steps of the NL2SQL modeling method described above.

[0062] The present invention also provides a computer storage medium.

[0063] The computer storage medium stores the NL2SQL modeling program, which, when executed by the processor, implements the steps of the NL2SQL modeling method described above.

[0064] The method implemented when the NL2SQL modeling program running on the processor is executed can be referred to in various embodiments of the NL2SQL modeling method of the present invention, and will not be repeated here.

[0065] The present invention also provides a computer program product.

[0066] The computer program product of the present invention includes an NL2SQL modeling program, which, when executed by a processor, implements the steps of the NL2SQL modeling method as described above.

[0067] The method implemented when the NL2SQL modeling program running on the processor is executed can be referred to in various embodiments of the NL2SQL modeling method of the present invention, and will not be repeated here.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0069] The technical solution of this invention performs a new round of reinforcement learning on erroneous cases, and the system has the ability to iteratively optimize. In a single domain, the accurate matching rate and execution accuracy both reach over 92%, and in multiple domains, the accurate matching rate reaches over 75%, and the execution accuracy reaches over 86%.

[0070] This invention provides a data-driven NL-to-SQL modeling method for intelligent customer service by integrating UDA technology. It solves the technical challenge of existing NL-to-SQL technologies being limited to database queries and thus unsuitable for complex intelligent customer service data. This lowers the barrier to database queries for operations personnel and improves operational efficiency. Simultaneously, the use of UDA technology enables the low-cost acquisition of a large number of model training samples, making it applicable to a wide range of scenarios and possessing commercial value.

[0071] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the scope of protection of the present invention, which is determined by the appended claims.

Claims

1. An NL2SQL modeling device, comprising: a mass data processing module, which trains sample data and realizes self-supervised sample data learning by using an UDA technical solution, the UDA being used to help a target domain learn without any labeled information by means of a source domain with a small amount of labeled data, so as to realize low-cost acquisition of labeled data by using a transfer learning technology; an NL-to-SQL model training module, which realizes a core technical solution including network weight quantization and activation layer quantization based on a TernaryBERT network model; a model compression and generation module, which uses a PKD model compression algorithm to extract hidden knowledge from the hidden layers of a "teacher" model by using two strategies of PKD-Last and PKD-Skip, completely eliminates the output of a "student" model imitating the "teacher" model, is used to reduce the number of layers of the NL-to-SQL model, reduces the parameter amount of model training, and improves the knowledge reasoning speed of the model; at the same time, a bat algorithm is used for network parameter training and intelligent search to find the optimal distillation network structure; a dialogue management module, which is used to realize processing including named entity recognition, wrong word correction, and user emotion analysis after TernaryBERT model training and PKD model compression, further fuses a preset SQL paradigm, associates and generates a complete database query SQL, and then interacts with a dialogue management system and an intelligent customer service interface.

2. An NL2SQL modeling method, which is realized based on the NL2SQL modeling device of claim 1 and is realized based on unsupervised domain adaptation (UDA) technology and a bat algorithm optimization, and comprises the following steps: S1, data processing, the data being product operation data, after the data processing, a large amount of unlabeled samples are generated, and then a preset query SQL paradigm template is used to realize labeled training sample data set by using the UDA technology; S2, NL-to-SQL model training, the natural language samples obtained are subjected to sample data training to generate a structured query language model, the model including a weight layer and an activation layer, and the weight layer and the activation layer are quantized by using a TernaryBERT model; S3, model compression and generation module, in order to make up for the insufficient effect of the TernaryBERT model due to excessive quantization in the task of intelligent customer service business consultation, the model is further compressed by using a pre-training knowledge distillation method, and at the same time, in order to prevent the distillation structure network weight from falling into a local optimal value, a bat algorithm is used for parameter training and intelligent search to find the optimal distillation network structure. S4, dialogue management, after the ternaryBERT model training and PKD model compression, preprocessing, then the model is fused with the preset SQL paradigm, correlation and generation of complete database query SQL, and then interact with the intelligent customer service interface, realize the dialogue management application.

3. The method of claim 2, wherein, The product operation data in S1 includes incoming business cards and hang-up messages, and the data processing method includes word segmentation, stop word removal, English punctuation replacement, question and answer data set construction, and sentence vector index database construction, and then generating a large number of unlabeled samples.

4. The method of claim 2, wherein, The NL-to-SQL model training in S2 adopts a TernaryBERT model, which quantizes the weight layer and the activation layer, The weight layer includes a linear layer and an Embedding layer, and the parameters of the linear layer and the Embedding layer account for a large part of the total parameters of the TernaryBERT model, so the TernaryBERT model is more thoroughly quantized in the linear layer and the Embedding layer. In the quantization of the activation layer, 8-bit symmetric and asymmetric methods are used for quantization.

5. The method of claim 4, wherein, In the quantization of the activation layer, in the actual inference process, the matrix multiplication is changed from 32-bit floating point operation to int8 integer operation, achieving the goal of acceleration; this model achieves the performance comparable to the full-precision model with only 6.7% of the parameters of the BERT model.

6. The method of claim 2, wherein, In S3, the pre-training knowledge distillation adopts PKD-Last and PKD-Skip strategies to extract hidden knowledge from the hidden layer of the "teacher" model, and completely eliminates the "student" model from imitating the output of the "teacher" model, The PKD-Last strategy uses the knowledge contained in the last k layers of the "teacher" model; the PKD-Skip strategy extracts and distills the knowledge in every k layers of the "teacher" model.

7. The method of claim 2, wherein, In S3, the bat algorithm is used for parameter training and intelligent search to find the optimal distillation network structure, including the following steps: S31, input the network coding vector into the trained structure generator to generate the weight of the corresponding distillation network, evaluate the distillation network on the validation set, and obtain the precision of the corresponding distillation network; S32, in order to search for the highest precision distillation network that meets the specific constraint condition, the bat algorithm is used to search for the distillation structure model with the highest precision that meets the specific constraint condition, wherein the specific constraint condition includes the number of floating point operations.

8. The method of claim 2, wherein, The preprocessing in S4 includes implementing named entity recognition, misspelling correction, and user emotion analysis.

9. An electronic device comprising: The memory, the processor, and the NL2SQL modeling program stored on the memory and executable on the processor, when the processor executes the NL2SQL modeling program, the steps of the NL2SQL modeling method in any one of claims 2 to 8 are implemented.

10. A computer storage medium, wherein, The computer storage medium has stored thereon an NL2SQL modeling program, which, when executed by the processor, implements the steps of the NL2SQL modeling method according to any one of claims 2 to 8.

Images