Intelligent shopping guide voice recognition method and system based on transfer learning

Abstract

The invention relates to the field of data processing, in particular to an intelligent shopping guide voice recognition method and system based on transfer learning, and the method comprises the steps: carrying out the transfer learning fine tuning through employing a pre-training language model in a voice recognition and optimization stage of intelligent shopping guide; the comprehensive adaptation index is obtained by dynamically combining the entity novelty score of the training batch and the model prediction confidence coefficient, the learning step length is adjusted in real time by further combining the model loss ratio, finally, the model can more intelligently adapt to new commodities, promotion information and query with different difficulties, and the overall shopping guide experience is optimized. According to the method, a pre-training language model is finely adjusted by applying transfer learning to carry out a shopping guide NLU task, and a multi-stage self-adaptive learning rate adjustment strategy fusing entity novelty, prediction confidence and loss dynamics is adopted in fine adjustment.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more particularly to an intelligent shopping guide speech recognition method and system based on transfer learning. Background Art

[0002] With the development of modern retail industry, large comprehensive supermarkets (hypermarkets) have become the main places for urban residents' daily shopping. Such hypermarkets usually cover a vast area, with tens of thousands of commodity categories, including food, fresh produce, daily necessities, household appliances, etc., and the commodity information (such as price, promotion activities, inventory, specific location) is in continuous dynamic change. When shopping in such a complex environment, customers often face the dilemma of information asymmetry: it is difficult to quickly find the exact location of the desired commodity, it is not easy to obtain the latest price or participate in preferential activities in real time, and they lack understanding of the dazzling array of commodities, especially newly launched products. To improve the shopping experience of customers and the operation efficiency of the mall, the introduction of intelligent shopping guide services has become an industry trend. The intelligent shopping guide system based on voice interaction, through the interaction interface deployed on the service desk, self-service terminal or mobile robot, allows customers to ask questions in the most natural way, and theoretically can provide more convenient and efficient services than traditional touch-screen queries or limited manual shopping guides.

[0003] However, to achieve truly intelligent, accurate and fluent voice shopping guide, the core challenge lies in the system's deep understanding ability of customers' natural language queries. The voice queries in the hypermarket environment have remarkable characteristics: First, they involve a vast amount of continuously updated domain-specific vocabulary such as commodity names, brand names, specifications, promotion terms, etc., and general speech recognition and natural language processing models often have serious out-of-vocabulary (OOV) problems and semantic understanding biases; Second, customers' questioning methods are flexible and diverse, with colloquial expressions, ellipsis, reference, and even vague descriptions being common; Third, the query intentions are complex and may involve multiple aspects such as price, location, attributes, evaluation, recommendation, etc. Therefore, the system not only needs to accurately convert speech to text (ASR), but more importantly, it needs to precisely understand the intention behind the text and extract all key information entities (NLU). Directly training a complex NLU model from scratch for each hypermarket to address the above challenges requires a vast amount of high-quality domain annotation data, which is costly and unrealistic. Therefore, using a large language model pre-trained on a vast amount of general text (such as BERT, etc.), and through the method of transfer learning, using a relatively small amount of domain annotation data for fine-tuning to quickly adapt it to the specific field of hypermarket shopping guide has become the most promising technical path.

[0004] Currently, fine-tuning techniques based on transfer learning have been widely applied to various NLP tasks. In the field of intelligent shopping guides, there have also been attempts to use pre-trained models for NLU optimization. However, during the fine-tuning process, the setting of the learning rate is a crucial and tricky part. In existing technologies, the commonly adopted methods include: (1) using a fixed and very small learning rate for global fine-tuning; (2) adopting a preset learning rate decay strategy, such as linear decay, exponential decay, or decay by steps. The main drawback of these methods is that their adjustment methods fail to fully consider the dynamics and heterogeneity of the training data in the supermarket shopping guide scenario.

[0005] Specifically, when new product names or rare promotion terms appear in the training data, the model needs to quickly learn this new knowledge with a relatively large step size; while when dealing with very common and simple queries (such as "How much is an apple"), it needs to perform stable fine-tuning with a smaller step size to avoid disrupting the already learned patterns. A fixed learning rate cannot meet these two requirements simultaneously, which may lead to slow learning of new knowledge or over-adjustment on simple data, causing oscillations. The preset decay strategy has nothing to do with the content of the training data and cannot respond in real time to the specific challenges encountered during the training process. Although there are some general adaptive learning rate algorithms (such as Adam, RMSProp, etc.), they mainly adjust the learning rate based on historical gradient information and do not explicitly combine the specific characteristics of the domain task. Therefore, when the existing learning rate adjustment strategies are applied to the fine-tuning of the supermarket shopping guide NLU model, it is often difficult to achieve the optimal convergence speed, stability, and final model performance, which limits the further improvement of the understanding ability of the intelligent shopping guide system. Summary of the Invention

[0006] To address the problem that the above fixed learning rate cannot meet the two requirements simultaneously, in the first aspect, the present invention proposes an intelligent shopping guide speech recognition method based on transfer learning, including: obtaining the customer's inquiry speech for automatic speech recognition to obtain the initial recognition text; performing natural language understanding on the initial recognition text based on a pre-trained language model to identify the customer's query intention and the entities in the query text; constructing a query instruction according to the query intention and entities and retrieving in the supermarket commodity information database; generating a reply content based on the retrieved relevant information and outputting it through voice or / and interface; using manually labeled data to adjust the parameters of the pre-trained language model through backpropagation and parameter update; the parameter update includes an adaptive learning rate , where: ; where represents the base learning rate; represents the batch loss value; represents the batch Previous loss average; Represents the minimum integer; tanh represents the hyperbolic tangent function; Represents the batch Comprehensive adaptation index; Represents the set scaling factor; the comprehensive adaptation index is positively correlated with the entity novelty score and negatively correlated with the prediction confidence; the reciprocal mean of the occurrence frequencies of all entities in the current training batch constitutes the entity novelty score; the arithmetic mean of the maximum prediction probability of all query intents in the current training batch and the label probability of the entity constitutes the prediction confidence.

[0007] The method of the present invention realizes a complete intelligent voice shopping guide process by combining automatic speech recognition, natural language understanding based on a pre-trained model, database query and response generation. Especially in the fine-tuning stage of the natural language understanding model, a specific adaptive learning rate is innovatively adopted. This learning rate not only considers the conventional training loss changes, but also incorporates a comprehensive adaptation index customized for the supermarket shopping guide scenario, which directly correlates with entity novelty and model prediction confidence. Compared with the existing technologies that use a fixed learning rate, a preset decay strategy or a general adaptive algorithm based only on the gradient history, this method can dynamically and finely adjust the learning step according to the actual novelty degree of the current batch of data and the model's understanding of it. When encountering new products or situations where the model is uncertain, it can learn and adapt faster, and when dealing with common and easily understood queries, it can be stably optimized, significantly improving the model's understanding accuracy, robustness and convergence efficiency for various queries in the supermarket environment with dynamic changes in commodity information, and overcoming the defect that the learning rate adjustment strategy in the existing technologies lacks scene awareness ability.

[0008] Further, the calculation method of the comprehensive adaptation index is specifically: ; Wherein Represents the batch Comprehensive adaptation index; Represents the batch Entity novelty score; Represents the batch Prediction confidence.

[0009] Further, the calculation method of the entity novelty score is specifically: ; Wherein Represents the batch Entity novelty score; Represents the batch Total number of entities included; Represents the entity The frequency that appears during the training process; Represents the natural logarithm function.

[0010] The present invention specifically defines the calculation method of the entity novelty score as the average of the entity inverse logarithm frequency correlation values within a batch. Compared with simple inverse frequency or binary new / old judgment, this method can more smoothly and effectively quantify the novelty degree of entities, give the highest novelty weight to extremely rare entities, and at the same time can also distinguish the differences between common entities and very common entities, providing a finer-grained signal that is non-linearly related to the entity occurrence frequency for guiding the learning rate adaptation adjustment.

[0011] Furthermore, the specific calculation method of the prediction confidence is as follows: ; where represents the prediction confidence of batch ; represents the number of samples in batch ; represents the average value of the token numbers corresponding to the samples in batch ; represents the maximum value of the intent prediction probability of sample ; represents the entity label probability of the rd token in sample

[0012] The present invention specifically stipulates that the prediction confidence is calculated by comprehensively averaging the maximum intent prediction probability of the samples within a batch and the entity label probabilities of all tokens. This method comprehensively reflects the overall prediction confidence level of the model on the two subtasks of intent classification and sequence labeling in the current batch. Compared with a single index that only considers intent confidence or only considers the lowest / average entity confidence, it provides a more global and robust confidence evaluation, enabling the learning rate adjustment to be more accurately based on the comprehensive confidence level of the model for the entire prediction task.

[0013] Furthermore, the BERT-base-chinese model is used as the basic natural language understanding model, and the pre-trained language model is obtained by collecting the manually annotated data and performing end-to-end supervised training based on the basic natural language understanding model.

[0014] In the present invention, by explicitly selecting the widely verified BERT-base-chinese model as the base model and adopting an end-to-end supervised training method for fine-tuning, it is ensured that the NLU module can effectively utilize the powerful general language representation ability and fully adapt to the specific language patterns and task requirements in the supermarket shopping guide field through the adjustment of overall parameters. Compared with selecting a weaker base model or only adjusting some parameters for fine-tuning methods, it is expected to obtain better domain adaptation performance and higher understanding accuracy.

[0015] Further, the artificially written simulated queries, supermarket commodity catalogs, promotional activity descriptions, and floor shelf layout information are used as the original text, and the artificial annotation data is obtained by using the BIO annotation system for the original text.

[0016] In the present invention, by specifically indicating that the artificial annotation data for fine-tuning comes from the original text highly relevant to simulated queries, commodity catalogs, promotional descriptions, and shelf layouts, and adopting the standard BIO annotation system, the domain pertinence and annotation standardization of the training data are ensured, enabling the model to directly learn the real language expressions and entity structures in the supermarket scenario. Compared with training using general corpus or data with non-standard annotations, it can significantly improve the accuracy and reliability of the NLU model in practical applications.

[0017] Further, it also includes using a DNN-based denoising model to preprocess the customer's inquiry voice.

[0018] In the second aspect, the present invention provides an intelligent shopping guide speech recognition system based on transfer learning, including: a voice input unit: used to receive the customer's voice command; a speech recognition unit: connected to the voice input unit and used to convert the voice command into text; a natural language understanding unit: connected to the speech recognition unit and used to process the text based on a pre-trained language model to identify the customer's query intention and key entities; the pre-trained language model included in the natural language understanding unit is trained and configured according to the fine-tuning step in the intelligent shopping guide speech recognition method based on transfer learning; an information processing and query unit: connected to the natural language understanding unit and the commodity information database, used to construct a database query, execute the query, and retrieve information; a commodity information database: storing detailed information of all commodities in the supermarket, including real-time prices, promotional information, and accurate shelf position information; a reply generation unit: connected to the information processing and query unit and used to generate anthropomorphic reply content according to the retrieval result; an output unit: connected to the reply generation unit and used to output the reply content through voice or / and interface.

[0019] Further, the output unit includes at least one of the following: a speech synthesizer and a speaker, which are used to synthesize the reply content into speech and broadcast it; a display screen, which is used to display the text of the reply content, associated images, or a navigation path generated according to the commodity location information.

[0020] Further, the voice input unit includes at least one microphone array for receiving voice commands from customers.

[0021] The technical effects of the present invention are as follows: The core innovative content of the present invention is concentrated in a novel multi-stage adaptive learning rate adjustment mechanism adopted when fine-tuning a pre-trained language model for NLU tasks in the supermarket shopping guide field. Different from the existing technologies that adopt a fixed learning rate, a preset decay strategy, or a general adaptive algorithm that only relies on the gradient history, the mechanism proposed by the present invention deeply combines the characteristics of the supermarket shopping guide scenario. First, the relative novelty of entities within a batch is calculated to quantify the emergence degree of new knowledge; secondly, the overall confidence of the model's prediction for the current batch is evaluated to judge the model's mastery of the current input; then, these two scenario-based features are fused into a comprehensive adaptation index, which reflects the adaptation requirements of the current batch of data in the dimensions of "novelty" and "model certainty"; finally, this scenario adaptation signal is combined with a factor measuring the relative learning difficulty of the current batch, and the base learning rate is jointly modulated through a smooth bounded function. This mechanism enables the learning rate to simultaneously respond to the immediate difficulty during the training process and the domain characteristics of the data itself (such as new and old knowledge, prediction confidence), realizing a more intelligent and detailed control of the learning step size, thereby significantly improving the learning efficiency, stability, and final performance of the model in a dynamic supermarket environment. Description of the Drawings

[0022] By reading the following detailed description with reference to the accompanying drawings, the above and other purposes, features, and advantages of the exemplary embodiments of the present invention will become easily understandable. In the drawings, several embodiments of the present invention are shown in an exemplary rather than restrictive manner, and the same or corresponding reference numerals represent the same or corresponding parts, where: Figure 1 It is a flowchart schematically showing the intelligent shopping guide speech recognition method based on transfer learning in the embodiments of the present invention; Figure 2 It is a block diagram schematically showing the structure of the intelligent shopping guide speech recognition system based on transfer learning in the embodiments of the present invention. Detailed Embodiments

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

[0024] The following will describe in detail the specific implementation manners of the present invention with reference to the accompanying drawings.

[0025] An embodiment of an intelligent shopping guide speech recognition method based on transfer learning: As Figure 1 shown, the intelligent shopping guide speech recognition method based on transfer learning of the present invention includes: S1. Implement the complete process of the intelligent voice shopping guide method for large-scale shopping malls, including speech processing, core recognition, database query, intelligent response, and continuous optimization of the model.

[0026] In modern large-scale shopping malls, there are a dazzling array of goods, a wide variety of categories, a complex layout, and frequent changes in promotional activities. During the shopping process, customers often face pain points such as time-consuming to find specific goods, inability to obtain accurate price or preferential information in a timely manner, and lack of understanding of new products. Although traditional human shopping guides can provide help, they are limited by labor costs, service coverage, and response capabilities during peak hours, and it is difficult to meet the immediate and accurate information needs of all customers. Therefore, in this embodiment, intelligent shopping guide services can be provided through natural human-machine voice interaction, accurately understanding the intentions of customers, and combining real-time dynamic information of the shopping mall.

[0027] In one embodiment, the complete intelligent shopping guide process may include: customer voice collection and preprocessing, speech recognition and optimization, information extraction and database, feedback and interaction, and recording interaction data to optimize the model.

[0028] Among them, customer voice collection and preprocessing means that first, natural voice commands issued by customers near the shopping guide device are captured by the microphone array of the shopping guide device; then, the built-in voice processing system of the shopping guide device enhances the voice of the target speaker; finally, a denoising model based on DNN can be used to suppress environmental noise and eliminate reverberation, and output a digital audio stream of the clear voice of the customer. It should be noted that: in this embodiment, the shopping guide device can be an interactive intelligent shopping guide counter or an intelligent shopping guide robot.

[0029] The voice recognition and optimization process is the key process of this embodiment. The recognition effect of the digital audio stream after preprocessing in this process directly determines the output accuracy of the final shopping guide device and the human-computer interaction experience. In this embodiment, this process can first convert the audio stream into initial text through an automatic speech recognition (ASR) engine; then deeply analyze the recognized text through a natural language understanding (NLU) module, and finally accurately recognize the customer's query intention (such as querying price, location, promotion, inventory, product details, seeking recommendations, etc.) and extract the key information entities in the query (such as product name, brand, specification, attribute words, etc.). In subsequent steps S2 and S3, the specific implementation of the voice recognition and optimization process of this embodiment will be emphasized. This process can output a structured semantic representation, for example: {intent: "query_price_location", entities: [{"entity_type": "brand", "value": "XX brand"}, {"entity_type": "product", "value": "cucumber"}]}.

[0030] After obtaining the structured semantic representation in the information extraction and database query process, the system parses this structured information and automatically constructs a query instruction that conforms to the background database query language (such as SQL). This instruction will be used to query the real-time updated supermarket product information database. In this embodiment, the database should at least maintain the product category, brand, origin, specification, real-time selling price, current promotion activities, inventory status, associated product information, and the precise physical location in the mall (for example: floor - area - shelf number - tier number). This process can output the detailed information of one or more product records that match the query conditions.

[0031] In the feedback and interaction process, the database first queries and returns the detailed product information. Based on the retrieved information, the system can construct a user-friendly, clear, and accurate reply through natural language generation technology (NLG); then the reply content will prioritize highlighting the core information directly asked by the user (such as price and location); further, the system can integrate a recommendation algorithm to recommend relevant products or promotions according to the user's query. It should be noted that: in this embodiment, the presentation method of the reply of the shopping guide device is diverse. In one embodiment, it can generate a voice broadcast through a text-to-speech (TTS) engine, and in another embodiment, it can display text, pictures, price tags, promotion details on the display screen of the shopping guide device. If the hardware of the shopping guide device supports it, it can also dynamically render an indoor navigation path map from the current device location to the target product location.

[0032] Finally, in the process of recording and optimizing the interaction data model, the back-end of the shopping guide device can store the real data of human-computer interaction (after user consent and anonymization) for offline analysis and periodic iterative optimization of the model. In particular, for the NLU model, incremental training or re-fine-tuning can be performed with these new data to continuously improve its performance in real scenarios and adapt to changes in new products, new promotional statements, and customer expression habits. The training and application processes of the models used in the above processes are well-known technologies and will not be elaborated here.

[0033] S2. Detailed Explanation of Speech Recognition and Optimization Core: It involves the whole process of ASR speech-to-text conversion, selection of pre-trained NLP models, construction of domain annotation data, key model fine-tuning and transfer learning, performance verification and iterative optimization, and finally deployment and online launch.

[0034] In the whole process, the performance of "speech recognition and optimization" directly determines the effectiveness of subsequent links and is the key to improving the user experience. Specifically, in this embodiment, this process includes the following 6 steps, namely: application of basic acoustic and language models, selection of basic natural language understanding models, collection and preparation of domain-specific data, model fine-tuning - transfer learning, model verification and optimization, and model deployment and (optional) online learning.

[0035] In this embodiment, the step of applying basic acoustic and language models uses an end-to-end ASR system. Exemplarily, in one embodiment, an acoustic model based on the Transformer architecture can be used to directly map the acoustic feature sequence to a subsequence. In another embodiment, a hybrid system combining a deep neural network acoustic model (such as TDNN, LSTM) and a traditional language model (such as N-gram or RNN-LM) can be adopted. The output of this step is an initially recognized text string. For example: "I want to find that XX brand of milk".

[0036] In this embodiment, the BERT-base-chinese model can be selected in the step of selecting the basic natural language understanding model. This model has learned rich vocabulary, grammar, and semantic knowledge through unsupervised pre-training (such as Masked Language Model, Next Sentence Prediction tasks) on a large amount of text (such as encyclopedias, news, web pages). Subsequently, domain-specific data is collected. In one embodiment, if there are some traditional text-based shopping guide counters with non-intelligent voice interaction in the supermarket before, the original user query logs can be collected as the original text. In another embodiment, artificially written simulated queries, supermarket product catalogs (including categories, brands, names, specifications, aliases), promotional activity descriptions, floor and shelf layout information, etc. can be used as the original text.

[0037] After obtaining the original text, it is necessary to clean, deduplicate, and standardize the original text, and then perform manual annotation, specifically including: assigning predefined intent categories to each query text, such as: query price, query location, query inventory, query promotion, query product attributes, request recommendation and chat, etc. Perform entity annotation for each query text. In this embodiment, the BIO annotation system can be used to accurately annotate the key information segments and their types involved in the text, such as: B - product name, I - product name, B - brand, I - brand, B - specification, I - specification, B - attribute word (such as "price", "location", "discount"), etc. Preferably, special attention should be paid to annotating various aliases, abbreviations, and colloquial expressions of products and brands. Exemplary description: Sample 1: Text: "What's the price of cucumbers of XX brand today?"; Intent: query_price (query price); Entities - using BIO annotation method: Today: O (Outside); Day: O; X: B - brand (Begin - Brand); X: I - brand (Inside - Brand); Brand: I - brand; Of: O; Yellow: B - product (Begin - Product); Melon: I - product (Inside - Product); What: O; About: O; Price: B - attribute (Begin - Attribute, referring to price); Rate: I - attribute (Inside - Attribute); ?: O.

[0038] Sample 2: Text: "Which shelf is the tomato sauce on?"; Intent: query_location (query location); Entities: Tomato: B - product; To: I - product; Sauce: I - product; On: O; Which: B - location indicator (Begin - Location Indicator); One: I - location indicator; Cargo: I-position indication; Rack: I-position indication; ? :O.

[0039] Finally, this step can output a structured, high-quality, labeled dataset covering the main query types and entities in supermarket shopping guide scenarios. ,have: ; in Indicates The text of the labeled samples; Indicates The intention of the labeled samples; Indicates Entities of labeled samples; Represents the total number of labeled samples.

[0040] After selecting a basic natural language understanding model and obtaining a domain-specific annotated dataset After that, you can first add task-specific output layers (for example, a linear layer + Softmax for intent classification and a linear layer + Softmax for sequence labeling (entity recognition)) on top of the pre-trained model, i.e. the base natural language understanding model; then use The dataset performs end-to-end supervised training on the entire model (including the pre-training part and the newly added task layer). This step can output a natural language understanding model specifically for supermarket shopping guide tasks. .

[0041] Further obtain the natural language understanding model After that, the performance of the model can be comprehensively evaluated. The indicators used in this embodiment include the accuracy (Accuracy), precision (Precision), recall (Recall), F1 score of intent recognition, and the accuracy, recall, and F1 score of entity recognition (calculated separately by entity type or macro / micro averaged here). Subsequently, the advantages and disadvantages of the model are analyzed according to the verification results (for example, which types of intent / entity recognition are poor), the model architecture is adjusted, the hyperparameters are modified, the training data is increased or improved, different pre-trained models are tried, etc., and then the fine-tuning and verification process is repeated until the model performance reaches the predetermined goal; after debugging, this step can output an optimal NLU model version for deployment. The above-mentioned evaluation of model performance and adjustment of model parameters are well-known technologies and will not be repeated here.

[0042] Finally, the optimized and finalized NLU model can be deployed to actual shopping guide devices or cloud servers to provide real-time NLU service interfaces. At the same time, for the continuous evolution of the model, an online learning or regular update mechanism can also be designed. For example, in the process of optimizing the model by recording interaction data in step S1, user feedback signals (such as whether the user is satisfied with the result and whether the recognition error has been corrected) and new interaction data are collected, and after screening and annotation, they are used for incremental update (online fine-tuning) or periodic full retraining of the model.

[0043] S3. Combine the batch entity novelty score and prediction confidence to obtain a comprehensive adaptation index, and dynamically adjust the learning step size in combination with the batch loss to improve the domain adaptability of transfer learning.

[0044] When fine-tuning the shopping mall guide NLU model in transfer learning, some actual situations may be faced, such as: emergence of new entities: new products, brands, and promotional activities keep emerging, and the model needs to learn quickly; different query complexities: simple and direct queries ("Where is the milk?") coexist with complex and ambiguous queries ("Is there any promotion on that imported yogurt that is not too sweet and suitable for the elderly?"); changes in model confidence: the model has different prediction confidences for different inputs, and low confidence may mean more cautious or stronger adjustments. A single learning rate strategy is difficult to balance the three requirements of quickly adapting to new knowledge, stably learning known patterns, and adjusting the learning intensity according to the model's own state.

[0045] Therefore, in the model fine-tuning - transfer learning step of S2 above, the setting of the learning rate is crucial for the convergence speed, stability, and final performance of model training. The traditional fixed or pre-determined decay learning rate strategy cannot well adapt to the challenges of data diversity and dynamics in the shopping mall guide scenario. So in this embodiment, an adaptive learning rate can be set to improve the effect of model training.

[0046] Since new products are the norm in the shopping mall scenario, the model should be able to pay more attention to samples containing new entities and accelerate learning. Therefore, an index based on the dynamic update of entity frequencies during the training process can be constructed to quantify the "novelty" degree of named entities (products, brands, etc.) that appear in the current training batch relative to the training data that the model has seen.

[0047] Specifically, before or during the training, a counter can be maintained first to record the number of occurrences (or frequencies) of each entity encountered in the training data; then for each recognized entity in the current batch , which can also be denoted as , calculate its novelty score ; where adding 1 to the logarithmic frequency is to avoid zero-sum processing of the first-occurring entity, that is, at this time , the novelty is the highest, and adding 1 to the denominator ensures that the score is in the range (0, 1]; finally, calculate the average novelty score of all entities within the batch, and we have: ; where represents the entity novelty score of batch , with a value range in (0, 1]. The higher its value, the more novel or rare the entities included in the batch; represents the total number of entities included in batch ; represents the entity appearance frequency during the training process; represents the natural logarithm function.

[0048] When batch contains a large number of first-occurring or rarely-occurring entities, is generally small. At this time, is closer to 0, while is closer to 1, that is, is closer to 1. At this time, this batch has high novelty; conversely, when most of the entities in the batch are common entities, that is, is large, then is also large. At this time, is smaller and closer to 0, indicating that this batch has low novelty.

[0049] Further obtain the confidence of the model prediction to measure the overall "confidence" degree of the model in intent classification and entity annotation for the samples in the current batch . The confidence of the model prediction is a direct reflection of its understanding of the current input. Low confidence may mean that the input is ambiguous, the model is insufficient in ability, or it encounters boundary situations and needs to be learned and adjusted; high confidence indicates that the model is more familiar with the current pattern.

[0050] First, for the intent classification task, obtain the highest probability output by the model; then, for the entity recognition task, obtain the probability of the predicted label for each token; finally, calculate the arithmetic mean of the average intent confidence and the average entity label confidence for all samples within the batch, and we have: ; where represents the prediction confidence of batch ; represents the number of samples in batch ; Indicates the batch The average sequence length (number of tokens) of the samples in; Indicates the sample The maximum value of the intention prediction probability; Indicates the sample In the Entity label probability of the th token.

[0051] In this embodiment, high novelty ( ) or low confidence ( ) may both indicate the need for a stronger learning signal, but the combination of the two better reflects the specific situation. Exemplary illustration: For novel but highly confident samples, only moderate adjustment may be needed; for common but low-confidence samples, the reasons need to be identified and learning strengthened. Therefore, a comprehensive adaptation index can be calculated in this embodiment, specifically: ; Where Indicates the comprehensive adaptation index of the batch , and the value range is [0, 2]; Indicates the entity novelty score of the batch ; Indicates the prediction confidence of the batch . When the batch is very novel, that is, is close to 1, and the model prediction is very unconfident, that is, is close to 0, is close to 2, indicating that the batch of data requires more model adaptation; conversely, when the entity is common, that is, is close to 0, and the model is very confident, that is, is close to 1, is closer to 0, indicating that the batch of data does not require significant model adaptation. Finally, the adaptive learning rate calculation method is obtained as: ; Where Indicates the adaptive learning rate for fine-tuning the model during the transfer learning process for the batch ; Indicates the base learning rate; Indicates the loss value of the batch ; Indicates the average loss of all batches during the model learning process; Indicates a very small integer to avoid the denominator being 0, and can be set to 1e-8 in this embodiment; Indicates the batch Comprehensive adaptation index; Indicates the scaling factor used to control Regarding the impact of the learning rate, in this embodiment, it can be set to the empirical value 2 to make it on the same order of magnitude as the loss ratio term to avoid completely dominating the adjustment; tanh represents the hyperbolic tangent function, which is used to smoothly map the combined signal inside the parentheses to (-1, 1), so that the factor (1 + tanh( )) fluctuates within the range of (0, 2).

[0052] The above formula allows the learning rate to respond to information in two dimensions simultaneously, namely the relative difficulty and the scene adaptation requirement. Among them, the relative difficulty is represented by the term, which reflects the ease or difficulty of the current batch compared with the recent average level. And the scene adaptation requirement: is represented by the term, which incorporates the adaptive adjustment signal for the supermarket scene refined from entity novelty and prediction confidence. After adding the two, they are smoothed and bounded scaled by tanh, and finally modulate the base learning rate , and the introduction of is to balance the contributions of the two signal sources.

[0053] When the batch is both difficult, that is, , and has a strong scene adaptation requirement, that is, is large, the value inside the parentheses of the tanh function will be significantly positive. At this time, tanh approaches 1, will be significantly greater than (close to ), achieving fast learning and adaptation; and when the batch is both simple, that is, , and has a weak scene adaptation requirement, that is, is small, the value inside the parentheses of the tanh function will be significantly negative. At this time, tanh approaches -1, will be significantly less than (close to 0), achieving fine-tuning and stable learning; at the same time, in other combined situations (such as difficult but weak adaptation requirement, or simple but strong adaptation requirement), the adjustment factor between the two will be obtained according to the relative strength of the two signals, so that the learning rate can respond more precisely to complex data characteristics.

[0054] The adaptive learning rate calculated can enable the model to more intelligently allocate learning resources during the fine-tuning process, be more sensitive to new knowledge and difficult samples, and be more robust in processing known and simple patterns, so as to achieve better overall performance in the complex supermarket shopping guide scene.

[0055] S4. Review of the complete process of the intelligent shopping guide integrating the adaptive learning rate, the overall workflow and examples after integrating the multi-stage dynamic learning rate adjustment strategy.

[0056] In summary, this embodiment provides an optimized intelligent shopping guide method. This method follows a complete process from voice collection, preprocessing, core recognition and optimization, information retrieval, to feedback interaction and data record optimization. During the model fine-tuning process in the "recognition and optimization" stage, it adopts the adaptive learning rate mechanism detailed in step S3. This mechanism enables the NLU model based on transfer learning to more intelligently and efficiently adapt to the characteristics of fast commodity updates and diverse customer queries in a large supermarket environment. The following are two application examples: Suppose the system has been deployed on a shopping guide robot at the entrance of the second floor of a large supermarket, and its NLU model has been fine-tuned using the adaptive learning rate strategy proposed in this embodiment.

[0057] Scenario 1: Common query: Customer's voice: "Excuse me, which floor is the pure milk on?" First, the robot microphone captures the voice and removes background noise to obtain clear audio; then the ASR system outputs the text: "Excuse me, which floor is the pure milk on"; further, the NLU model ( ) processes the text: Learning for such samples during fine-tuning: "Pure milk" is a common entity, that is low, the query is simple and direct, and the model prediction confidence is high, so the value is relatively low; at the same time, the loss of such samples may also be lower than the recent average . According to the formula in step S3, its comprehensive signal is negative, outputs a negative value, and finally will be slightly less than , which enables the model to make stable and fine adjustments when processing such familiar samples.

[0058] The current inference output is: intent: query_location, entities: [{"entity_type": "product", "value": "pure milk"}]; Then construct a query statement to find the location information of "pure milk"; subsequently, the database returns: "Pure milk is mainly distributed on the shelves B01 - B05 in the refrigerated area on the second floor"; finally, the robot voice broadcasts: "Pure milk is mainly in the refrigerated area on the second floor, and it is available on shelves B01 to B05." Optionally, at the same time, text information is displayed on the screen, and the map navigation path of area B in the refrigerated area on the second floor is highlighted.

[0059] Scenario 2: Involving new products or more complex queries: Customer voice: "I'm looking for the newly launched 'Zero Sensation Blueberry Flavor' sparkling water. Is there any discount?" First, perform voice collection and preprocessing in the same real-time manner as described above; then the ASR outputs the text: "I'm looking for the newly launched Zero Sensation Blueberry Flavor sparkling water. Is there any discount?"; further, the NLU model ( ) processes the text; during fine-tuning, learn from such samples: "Zero Sensation Blueberry Flavor sparkling water" may be a newly launched product. At the initial stage of training is low, resulting in being high. At the same time, due to it being a new product or a combined name, the initial prediction confidence of the model may not be high. The query also involves the "discount" attribute. Therefore the value will be high. The loss of such samples may also be higher than . According to the formula in step S3, the comprehensive signal is positive, output a positive value, and finally will be significantly greater than . This enables the model to learn this new entity and its associated query pattern at a faster speed.

[0060] The current inference output is: intent: query_promotion_availability, entities: [{"entity_type": "product", "value": "Zero Sensation Blueberry Flavor sparkling water"}, {"entity_type": "attribute", "value": "discount"}] Next, construct a query to find the inventory, location, and promotion information of "Zero Sensation Blueberry Flavor sparkling water". The database returns: "Zero Sensation Blueberry Flavor sparkling water is in stock, located on shelf D12 in the beverage area on the third floor. It is a new product promotion this week, and you can enjoy a 50% discount on the second item."; finally, the robot voice broadcasts: "Found it! The 'Zero Sensation Blueberry Flavor' sparkling water you want is on shelf D12 in the beverage area on the third floor. There is an activity now, 50% off on the second item!" At the same time, the shopping guide device screen displays: product pictures, prices, promotion details, and the navigation path to shelf D12 on the third floor.

[0061] Optionally, record this successful interaction of the new product query in the background, which helps with subsequent consolidation learning.

[0062] From the above two comparison examples, it can be seen that the intelligent shopping guide method proposed in this embodiment, which includes a multi-stage adaptive learning rate fine-tuning strategy, can dynamically adjust the learning process according to the commonness, novelty of the query content, and the prediction confidence of the model. Thus, in practical applications, it can not only stably process regular requests but also quickly adapt to new product information and complex queries, providing a more accurate, efficient, and intelligent shopping guide service.

[0063] An embodiment of an intelligent shopping guide speech recognition system based on transfer learning: On the other hand, the present invention also provides an intelligent shopping guide speech recognition system based on transfer learning.

[0064] As Figure 2 shown, the intelligent shopping guide speech recognition method of the present invention includes the following units that are interconnected and work together: Speech input unit 101: Responsible for receiving the natural speech commands issued by customers. This unit can be configured with at least one microphone array to enhance the speech pickup effect and suppress environmental noise, ensuring the capture of clear speech signals.

[0065] Speech recognition unit (ASR) 102: Connected to the speech input unit 101, receiving the collected speech signals. This unit integrates advanced acoustic models and language models internally, and is used to convert the input speech stream into a text string in real time.

[0066] Natural language understanding unit (NLU) 103: Connected to the speech recognition unit 102, receiving the text output by it. This unit is based on a powerful pre-trained language model (such as BERT, etc.). The key is that this pre-trained language model is trained and configured according to the steps defined in the aforementioned intelligent shopping guide speech recognition method of the present invention based on transfer learning, which includes a specific adaptive learning rate fine-tuning strategy.

[0067] Therefore, this NLU unit can accurately understand the language in the shopping mall shopping guide scenario, and identify the customer's core query intent (such as querying the price) and key entities (such as the commodity name "milk").

[0068] Information processing and query unit 104: Connected to the natural language understanding unit 103, receiving the structured intent and entity information output by it. This unit is responsible for converting this information into query instructions that can be executed in the background database (for example, generating SQL statements). At the same time, it connects and manages the interaction with the commodity information database 105, and can realize querying and returning results through remote API calls.

[0069] Product Information Database 105: Stores detailed and real-time updated information of all products in the supermarket. This information includes at least the real-time price of the product, details of current promotional activities, and the precise physical location information of the product on the shelf.

[0070] Reply Generation Unit 106: Connected to the Information Processing and Query Unit 104, receives the query results retrieved from the database. This unit is responsible for organizing the retrieved data (such as price, location, promotional description) into one or more paragraphs of natural, fluent, and anthropomorphic reply text. For example, generate the text "This milk is currently sold at 10 yuan per bottle and is on Shelf A02 in the refrigerated area on the second floor." Output Unit 107: Connected to the Reply Generation Unit 106, receives the generated reply content. This unit is responsible for presenting the information to the customer. This unit may include: Text-to-Speech (TTS) Synthesizer and Speaker: Converts the reply text into speech for broadcasting.

[0071] Display Screen: Used to display the reply text, product pictures, price tags, promotional posters, or dynamically generate and display a navigation path map according to the retrieved location information.

[0072] Example of Workflow: When a customer asks the system (such as a shopping guide robot) "Where is the XX brand yogurt?", The Voice Input Unit 101 (including the microphone array) captures the voice; The Speech Recognition Unit 102 converts it into the text "Where is the XX brand yogurt"; The Natural Language Understanding Unit 103 (using a model fine-tuned by a specific method) analyzes the text, identifies the intention as "query location", and the entities as "brand: XX brand", "product: yogurt"; The Information Processing and Query Unit 104 constructs a query based on the NLU result and searches for the location of the XX brand yogurt in the Product Information Database 105; The Product Information Database 105 returns the location information (such as "Shelf C05 in the dairy section on the third floor"); The Reply Generation Unit 106 generates the reply text; The Output Unit 107 broadcasts "The XX brand yogurt is on Shelf C05 in the dairy section on the third floor" through the speaker, and may also display a map to the location on the screen at the same time.

[0073] In the present invention, the foregoing memory may be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random-access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store the required information and can be accessed by an application program, module, or both. Any such computer storage medium may be part of the device or accessible or connectable to the device. Any application or module described in the present invention may be implemented using computer-readable / executable instructions stored or otherwise maintained by such a computer-readable medium.

Claims

1. An intelligent shopping guide speech recognition method based on transfer learning, characterized in that: The method comprises: Acquire the customer's inquiry voice and perform automatic voice recognition to obtain an initial recognition text; perform natural language understanding on the initial recognition text based on a pre-trained language model to identify the customer's query intent and entities in the query text; construct a query instruction based on the query intent and entities and search in the supermarket product information database; generate reply content based on the retrieved relevant information and output it through voice and / or interface; Using manually annotated data, the parameters of the pre-trained language model are adjusted through back propagation and parameter update; the parameter update includes an adaptive learning rate ,have: ； in represents the basic learning rate; Indicates batch The loss value of Indicates batch The average of previous losses; represents a very small integer; tanh represents the hyperbolic tangent function; Indicates batch Comprehensive adaptation indicators; Indicates the set scaling factor; the comprehensive adaptation index is positively correlated with the entity novelty score and negatively correlated with the prediction confidence; the reciprocal mean of the occurrence frequency of all entities in the current training batch constitutes the entity novelty score; the arithmetic mean of the maximum predicted probability of all query intents in the current training batch and the label probability of the entity constitutes the prediction confidence.

2. According to the method of intelligent shopping guide speech recognition based on transfer learning in claim 1, it is characterized in that: The calculation method of the comprehensive adaptation index is specifically as follows: ； in Indicates batch Comprehensive adaptation indicators; Indicates batch Entity novelty score of ; Indicates batch prediction confidence.

3. According to claim 2, the intelligent shopping guide speech recognition method based on transfer learning is characterized in that: The calculation method of the entity novelty score is specifically as follows: ； in Indicates batch Entity novelty score of ; Indicates batch The total number of entities contained in ; Representing Entities frequency of occurrence during training; Represents the natural logarithm function.

4. According to the method of intelligent shopping guide speech recognition based on transfer learning in claim 2, it is characterized in that: The calculation method of the prediction confidence is specifically as follows: ； in Indicates batch Confidence of predictions; Indicates batch The number of samples in ; Indicates batch The average number of tokens corresponding to the samples in the Representation sample The maximum value of the intention prediction probability; Representation sample Middle The entity label probability of a token.

5. According to the method of intelligent shopping guide speech recognition based on transfer learning in claim 1, it is characterized in that: The BERT-base-chinese model is used as the basic natural language understanding model, the manually annotated data is collected, and end-to-end supervised training is performed based on the basic natural language understanding model to obtain the pre-trained language model.

6. The intelligent shopping guide speech recognition method based on transfer learning according to claim 5 is characterized in that: The manually written simulation query, supermarket product catalog, promotion activity description and floor shelf layout information are taken as original text, and the BIO annotation system is used to obtain the manually annotated data.

7. The intelligent shopping guide speech recognition method based on transfer learning according to claim 1 is characterized in that: It also includes pre-processing the customer's inquiry voice using a DNN-based denoising model.

8. An intelligent shopping guide speech recognition system based on transfer learning, characterized in that: The system comprises: Voice input unit: used to receive voice commands from customers; Speech recognition unit: connected to the speech input unit, used to convert the speech instruction into text; A natural language understanding unit: connected to the speech recognition unit to process the text based on a pre-trained language model to identify the customer's query intention and key entities; the pre-trained language model included in the natural language understanding unit is trained and configured according to the fine-tuning step in the intelligent shopping guide speech recognition method based on transfer learning according to any one of claims 1 to 7; Information processing and query unit: connected to the natural language understanding unit and the product information database, used to construct database queries, execute queries and retrieve information; product information database: stores detailed information of all products in the supermarket, including real-time prices, promotional information and precise shelf location information; reply generation unit: connected to the information processing and query unit, used to generate anthropomorphic reply content according to the retrieval results; output unit: connected to the reply generation unit, used to output the reply content through voice and / or interface.

9. The intelligent shopping guide speech recognition system based on transfer learning according to claim 8, characterized in that: The output unit includes at least one of the following: A speech synthesizer and a speaker, used to synthesize the reply content into speech and broadcast it; A display screen is used to display the text of the reply content, associated images, or a navigation path generated according to the product location information.

10. The intelligent shopping guide speech recognition system based on transfer learning according to claim 8, characterized in that: The voice input unit includes at least one microphone array for receiving voice instructions from customers.

Images