Business hall customer consultation system based on AI glasses

Abstract

The invention relates to the technical field of intelligent services, in particular to a business hall customer consultation system based on AI glasses, which comprises an AI glasses terminal, and the AI glasses terminal comprises a display module used for realizing virtual image projection; the sensor tracking module is used for tracking external information and collecting the information; the data processing module is used for receiving and processing data and generating result feedback; the knowledge base management module comprises a knowledge base, related knowledge service data information is stored in the knowledge base, and data support is provided for feedback of a generated result; when the business hall customer consultation system based on the AI glasses operates, voice and / or posture information of a user is collected, meaning analysis is carried out, and corresponding feedback information is generated and visually displayed and interacted through AR. Through deep integration of technical innovation and scenes, the efficiency bottleneck of traditional business hall services is broken through, and the method has remarkable advantages in the aspects of improving customer satisfaction, reducing operation cost, strengthening data security and the like.

Description

Technical Field

[0001] The present invention relates to the field of intelligent service technology, and in particular to a business hall customer consultation system based on AI glasses. Background Art

[0002] In traditional business hall service scenarios, staff face problems such as diverse customer consultation needs, low information retrieval efficiency, and insufficient standardization of service processes. These problems are manifested in the following ways:

[0003] Delayed information acquisition: Staff members need to frequently consult paper documents or look down at computers to inquire about business processes, product information, etc., which results in longer waiting times for customers and a poor interactive experience.

[0004] Insufficient service standardization: Different staff members have varying degrees of mastery of the knowledge base, which may lead to inconsistent answers to similar questions and affect customer trust.

[0005] Lack of personalized customer services: The lack of real-time customer historical data support makes it difficult to quickly understand customer background needs and provide targeted services.

[0006] Single interaction method: Relying on traditional keyboard, mouse or touch screen operation, the operation efficiency is low when both hands are busy or when customer communication needs to be taken care of.

[0007] In existing technologies, some business halls have introduced intelligent customer service robots or mobile apps to assist in consultation, but the following limitations still exist:

[0008] Robots lack flexible interaction: they are unable to capture non-verbal signals such as customer expressions and gestures in real time, and have difficulty adapting to complex consulting scenarios.

[0009] Mobile devices distract attention: When staff use mobile phones or tablets to query information, they need to interrupt face-to-face communication with customers, affecting service continuity.

[0010] Insufficient real-time data: Traditional system knowledge bases are updated with lags and cannot synchronize with the latest business policies, resulting in decreased accuracy of answers.

[0011] Therefore, there is an urgent need for a new type of equipment that can integrate real-time information retrieval, intelligent interaction and augmented reality display to improve the efficiency and quality of business hall services. Summary of the Invention

[0012] The technical problem to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a customer consultation system for business halls based on AI glasses.

[0013] The technical solution adopted by this application to solve the technical problem is: a business hall customer consultation system based on AI glasses includes an AI glasses terminal, and the AI ​​glasses terminal includes:

[0014] Display module: used to realize virtual image projection;

[0015] Sensor tracking module: used to track external information and collect information;

[0016] Data processing module: used to receive and process data and generate result feedback;

[0017] Knowledge base management module: including the knowledge base, which stores relevant knowledge and business data information and provides data support for generating result feedback;

[0018] During operation, the customer consultation system in the business hall based on AI glasses collects user voice and / or gesture information, parses the meaning, generates result feedback, and performs AR visualization display and interaction.

[0019] The display module projects a virtual image using a transparent display technology.

[0020] The sensor tracking module includes:

[0021] Head tracking sensor, used to capture the movement information of the user's head;

[0022] Eye tracking sensor, used to capture changes in the user's gaze, providing a more accurate user experience;

[0023] Environmental perception sensors are used to capture surrounding environment information and user posture information, providing support for the precise superposition of real-world and virtual information.

[0024] The data processing module includes:

[0025] Computing and processing unit: Equipped with speech recognition and natural language processing models, the computing and processing unit is used to run algorithm models and process data to ensure efficient system response;

[0026] Augmented reality image processing unit: used for image processing, synthesizing virtual images and making real-time adjustments, accurately superimposing virtual information with the real environment, and displaying it.

[0027] The data K in the knowledge base is expressed as: K = {k1, k2, ..., km}, where km is a knowledge item, and km includes question and answer pairs, handling procedures and related regulations.

[0028] To ensure the timeliness and accuracy of information, the knowledge base management module regularly updates and manages the data in the knowledge base. This application uses an automated data crawling and update mechanism to ensure the timeliness and accuracy of answers. For data storage and query, the knowledge base of this application can be stored in a relational database (such as MySQL) or a graph database (such as Neo4j).

[0029] The operation process of the business hall customer consultation system based on AI glasses includes the following steps:

[0030] Speech recognition feedback: Collects user speech information, converts it into processable text, performs relevance queries based on the semantics of the text, and generates result feedback;

[0031] Result visualization: The generated result feedback information is integrated with the real scene to generate a display image. The core task of the augmented reality unit in AR glasses is to overlay virtual information on the real environment, providing users with an enhanced visual experience. This process requires the coordinated work of multiple technical modules.

[0032] Posture interaction: Capture user posture information and realize posture interaction.

[0033] The speech recognition feedback includes the following sub-steps:

[0034] Speech signal processing: collects the user's speech information and generates the corresponding text sequence W;

[0035] Semantic understanding: The text sequence W is passed into the natural language understanding model to understand the semantics and obtain the user's intention;

[0036] Information retrieval: Based on the user's intention, relevant information is retrieved from the knowledge base and result feedback information is generated.

[0037] The user posture information captured in the posture interaction includes user hand gestures.

[0038] The AI ​​glasses terminal is also equipped with a communication module, which is used to implement data exchange with external devices (such as mobile phones, cloud servers, etc.). The communication module supports wireless communication protocols such as Wi-Fi, Bluetooth, and 5G.

[0039] It also includes a power supply (such as a battery) for achieving power supply.

[0040] In addition, during the implementation process, the system's response speed and interactive experience are very important. This application uses the following processing technologies to optimize performance:

[0041] Asynchronous processing: Use multi-threading or asynchronous processing technology to enable tasks such as speech recognition, information retrieval, and AR display to be performed in parallel, improving response speed.

[0042] Caching technology: For common query results, use caching technology (such as Redis) to store them, reduce repeated calculations, and improve query efficiency.

[0043] Security and Privacy Protection: Since the system involves customer data, privacy protection is also of paramount importance. This application uses encryption protocols (such as HTTPS and TLS) for encryption during data transmission and uses encryption storage technology (such as AES) when storing data.

[0044] The AI-powered customer service system for business halls provides efficient and accurate real-time consulting and assistance services. Leveraging technologies such as speech recognition, natural language processing, and augmented reality, combined with efficient information retrieval and interactive models, the system significantly improves staff efficiency and customer service quality. Data security and privacy protection are also key to ensuring the system's sustainable development.

[0045] Compared with the existing technology, this application has the following beneficial effects: This application breaks through the efficiency bottleneck of traditional business hall services through deep integration of technological innovation and scenarios, and has significant advantages in improving customer satisfaction, reducing operating costs, and strengthening data security. It provides a new paradigm for the application of smart service terminals in vertical fields.

[0046] This application enables seamless real-time information presentation, intelligent semantic analysis, and rapid retrieval, improving service efficiency and accuracy. By overlaying business processes, FAQs, and other information on the display module in real time, staff can access information without interrupting customer interactions, reducing head-down time and improving consultation response speed. Automatically analyzing customer question intent based on speech recognition and natural language processing models, combined with deep learning models for semantic matching, the knowledge base retrieval has a high accuracy rate, helping staff better answer customer questions.

[0047] This application enables multimodal contactless interaction and an immersive service experience, optimizing the interactive experience and ease of operation. It supports voice commands and gesture control, freeing your hands and making it particularly suitable for carrying documents or operating devices. Based on AR technology, it integrates virtual information with real-world scenes, using eye tracking and head motion capture to accurately locate information and reduce visual interference.

[0048] This application realizes service standardization and traceability. The system uniformly calls the answers in the knowledge base to avoid differences in manual answers. The system automatically records the consultation history to facilitate service quality traceability and optimization.

[0049] The database is updated regularly to help staff obtain accurate and real-time updated information during the service process. DETAILED DESCRIPTION

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0051] The business hall customer consultation system based on AI glasses includes an AI glasses terminal, which includes a display module, a sensor tracking module, a data processing module and a knowledge base management module.

[0052] Among them, the display module is used to realize virtual image projection; the display module uses transparent display technology (such as optical waveguide, micro projection, etc.) to project the virtual image.

[0053] Sensor tracking module: used to track external information and collect information; the sensor tracking module includes:

[0054] A head tracking sensor is used to capture the movement information of the user's head. The head tracking sensor can use sensors such as accelerometers, gyroscopes, and magnetometers;

[0055] An eye tracking sensor is used to capture changes in a user's gaze and provide a more accurate user experience. The eye tracking sensor may use an eye movement sensor;

[0056] Environmental perception sensors are used to capture information about the surrounding environment and user posture, supporting the precise overlay of real-world and virtual information. Environmental perception sensors include depth sensors and cameras.

[0057] Data processing module: used to receive and process data and generate result feedback; the data processing module includes:

[0058] Computing processing unit: Equipped with speech recognition and natural language processing models, the computing processing unit is used to run algorithm models and data processing to ensure efficient response of the system; the computing processing unit uses an embedded computing unit (such as CPU or GPU, etc.), which is responsible for running AI algorithms, deep learning models, image processing, etc. to ensure efficient response of the system.

[0059] Augmented reality image processing unit: used for image processing, synthesizing virtual images and making real-time adjustments, accurately superimposing virtual information with the real environment, and displaying it.

[0060] Knowledge base management module: including a knowledge base, which stores relevant knowledge business data information and provides data support for generating result feedback; the data K in the knowledge base is expressed as: K = {k1, k2, ..., km}, where km is a knowledge entry, km contains question and answer pairs, handling procedures and related regulations, etc. In order to ensure the timeliness and accuracy of information, the knowledge base management module regularly updates and manages the data in the knowledge base. This application adopts an automated data crawling and updating mechanism to ensure the timeliness and accuracy of answers. For data storage and query, the storage of the knowledge base of this application can use a relational database (such as MySQL) or a graph database (such as Neo4j), etc.

[0061] During operation, the customer consultation system in the business hall based on AI glasses collects user voice and / or gesture information, parses the meaning, generates result feedback, and performs AR visualization display and interaction.

[0062] When in use, the user wears AI glasses and asks questions. The operation process of the customer consultation system based on AI glasses includes the following steps:

[0063] Speech recognition feedback: Collects user speech information, converts it into processable text, performs relevance queries based on the semantics of the text, and generates result feedback;

[0064] The speech recognition feedback includes the following sub-steps:

[0065] Speech signal processing: The user's speech information is collected and a corresponding text sequence W is generated. Specifically, this application defines the user's speech information as a time series S(t), where t is time. Using Fourier transform, the time series is converted from the time domain to the frequency domain, resulting in spectrum information: Sfreq(f) = F(S(t)). A convolutional neural network (CNN) or recurrent neural network (RNN) is then used to extract features from the spectrum, ultimately outputting a corresponding text sequence W. The text sequence W can be represented as W = {w1, w2, ..., wn}, where wn represents a single word or token in the user's input text.

[0066] Semantic Understanding: The text sequence W is passed to the natural language understanding model to understand the semantics and obtain the user's intent, using Intent to represent the user's intent. Furthermore, this application passes the recognized text sequence W to the natural language understanding model to understand the user's intent. Semantic analysis of text is performed based on deep learning models such as Transformers (e.g., BERT, GPT), which is represented as follows:

[0067] P(Intent∣W)=P(W∣Intent)P(Intent) / P(W);

[0068] Among them, P(Intent|W) is the posterior probability that the user's intention is Intent given the text W. The model outputs the intention of the user's query, P(Intent) is the prior probability that the user's intention is Intent, and P(W) is the total probability of the text sequence W.

[0069] Information retrieval: Based on the user's intent, relevant information is retrieved from the knowledge base and feedback is generated. Furthermore, this application uses an information retrieval model (such as an embedded retrieval model) to calculate the relevance score (score(Q,km)) between the query Q and each knowledge item. This application utilizes deep learning-based embedded retrieval, using the context-embedded BERT model to calculate the relevance between the query and the knowledge item.

[0070] The calculation process is as follows:

[0071] Preprocessing: First, perform text preprocessing on the query Q and all knowledge items km, removing stop words, stemming, and punctuation marks.

[0072] Embedding representation: The query Q and knowledge item km are converted into vector representations; through the deep learning model, the query Q and knowledge item km are converted into high-dimensional vectors (embedding). Then, the relevance is evaluated by calculating the cosine similarity between the query vector and the item vector, which is calculated as follows:

[0073]

[0074] Among them, Q and km are the transformed embedding vectors of query Q and knowledge item, respectively, and ||·|| represents the modulus of the vector.

[0075] Calculating Relevance: Based on the deep learning model, the relevance score between the query Q and each knowledge item km is calculated. The BERT model understands the context and generates context-sensitive embeddings to further improve the matching between the query Q and the knowledge item km. For example, BERT can be used to extract vector representations of the query and item, and then perform similarity calculations.

[0076] Sorting: Sort the knowledge items according to the similarity score and return the most relevant items as answers as result feedback.

[0077] Result visualization: The generated result feedback information is integrated with the real scene to generate a display image. The core task of the augmented reality unit in AR glasses is to overlay virtual information on the real environment, providing users with an enhanced visual experience. This process requires the coordinated work of multiple technical modules.

[0078] The display module is responsible for accurately projecting the virtual image onto the user's retina. Common technologies include waveguide and micro-projection. Waveguide technology uses specially designed transparent optical elements to guide light from the display source to the eye. The image the user sees through the glasses is displayed through the waveguide while maintaining a visual integration with the real world.

[0079] Micro-projection technology: using a small projection device to project images onto a transparent lens to generate virtual information.

[0080] The sensor tracking module captures the user's head and eye position, ensuring seamless integration of virtual information with the real world. This module uses accelerometers and gyroscopes to detect head movement, tracking tilt, rotation, and acceleration. Depth sensors (such as ToF sensors) and cameras capture the surrounding environment, ensuring alignment of virtual objects with the real world. This module ensures accurate positioning of virtual objects through environmental mapping and motion tracking.

[0081] The data processing module is mainly responsible for processing the data obtained from sensors and cameras, and generating suitable virtual images according to real-time environmental changes.

[0082] Among other things, depth sensors provide three-dimensional data of the environment, which the system uses to accurately position virtual objects in real scenes.

[0083] The core of image synthesis is to fuse virtual information with real scenes through transparency adjustment and image blending algorithms. Specifically, virtual information is usually weighted and blended with images in the real environment using transparency parameters and weight coefficients to generate the final display image. The display image can be expressed as:

[0084] Idisplay(x′,y′)=αIscene(x′,y′)+(1-α)Ivirtual(x′,y′);

[0085] Among them, Idisplay(x′,y′) is the final display image, Iscene(x′,y′) is the image of the real world, Ivirtual(x′,y′) is the image of the virtual object, and α is the transparency coefficient that controls the display intensity of the virtual information.

[0086] In order to map the virtual image in the three-dimensional space to the two-dimensional display screen of the glasses, a projection matrix is ​​required for coordinate transformation. The projection matrix formula is:

[0087]

[0088] Where M is the projection matrix, which contains the intrinsic and extrinsic parameters of the camera and defines how the virtual image is mapped to the display screen. [x′, y′] is the two-dimensional coordinate on the display screen; [x, y, z] is the object coordinate in three-dimensional space.

[0089] Through this projection matrix, environmental data and virtual information can be accurately superimposed on the real-world image, ensuring that the spatial relationship between virtual objects and real objects seen by the user is correct.

[0090] Gesture interaction: Capturing user gesture information to achieve gesture interaction. The user gesture information captured in the gesture interaction includes user hand gestures.

[0091] This application uses gesture recognition technology to enable users to control AR glasses through gestures without touching the device. The gesture recognition process is as follows:

[0092] Gesture acquisition and sensor data processing:

[0093] Gesture recognition typically relies on sensors on glasses, such as accelerometers, gyroscopes, depth sensors, and even external cameras. These sensors capture hand or finger movements and generate a time series of data G = {g1, g2, …, gn}, where gn represents the gesture data captured at time tn.

[0094] Accelerometer: Detects acceleration and movement of the hand.

[0095] Gyroscope: Detects the rotation angle and direction of the hand.

[0096] Depth sensor: captures the spatial position and movement of fingers.

[0097] Gesture recognition and classification processing:

[0098] The captured gesture data is processed by a machine learning model (such as a convolutional neural network (CNN) or a recurrent neural network (RNN)) and classified into different gesture categories.

[0099] Assume that G is the captured gesture data, gesture category L = {l1,l2,…,lk}, lk is the k-th gesture category, then the gesture category can be obtained by the following formula:

[0100] lk=argmax l∈L P(l|G);

[0101] Among them, P(l|G) is the predicted probability of gesture category lk given gesture data G.

[0102] Gesture and operation mapping:

[0103] Each gesture category is mapped to a specific action. For example:

[0104] Tap: Mapped to the "Select" or "Confirm" operation.

[0105] Swipe: Mapped to "page switching" or "scrolling" operations.

[0106] Pinch: Mapped to "zoom in" or "zoom out" interface elements.

[0107] Rotate: Mapped to the "rotate virtual object" or "adjust perspective" operation.

[0108] It can be defined specifically according to operating habits. By defining the mapping relationship between gestures and operations, the system can respond to user actions and perform corresponding operations.

[0109] Mathematical mapping formula for gesture operation:

[0110] The triggering of the operation can be achieved through the mapping function f between gesture categories and operations. The mapping formula is:

[0111] Ok=f(lk);

[0112] Where: Ok is a specific action triggered (such as "switch page" or "enlarge view").

[0113] For example, if l1 = slide, then O1 = page switching; if l2 = pinch, then O2 = zoomed in view.

[0114] Real-time feedback and confirmation:

[0115] The system confirms the success of gestures through real-time feedback. For example, when a gesture is executed, the AR glasses will immediately display interface changes or trigger specific functions. The system may also notify the user of the successful operation through vibration or voice feedback.

[0116] The AI ​​glasses terminal is also equipped with a communication module, which is used to implement data exchange with external devices (such as mobile phones, cloud servers, etc.). The communication module supports wireless communication protocols such as Wi-Fi, Bluetooth, and 5G.

[0117] It also includes a power supply (such as a battery) for achieving power supply.

[0118] In addition, during the implementation process, the system's response speed and interactive experience are very important. This application uses the following processing technologies to optimize performance:

[0119] Asynchronous processing: Use multi-threading or asynchronous processing technology to enable tasks such as speech recognition, information retrieval, and AR display to be performed in parallel, improving response speed.

[0120] Caching technology: For common query results, use caching technology (such as Redis) to store them, reduce repeated calculations, and improve query efficiency.

[0121] Security and Privacy Protection: Since the system involves customer data, privacy protection is also of paramount importance. This application uses encryption protocols (such as HTTPS and TLS) for encryption during data transmission and uses encryption storage technology (such as AES) when storing data.

[0122] The AI-powered customer service system for business halls provides efficient and accurate real-time consulting and assistance services. Leveraging technologies such as speech recognition, natural language processing, and augmented reality, combined with efficient information retrieval and interactive models, the system significantly improves staff efficiency and customer service quality. Data security and privacy protection are also key to ensuring the system's sustainable development.

[0123] The above descriptions are merely optional embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention specification under the concept of the present invention, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A customer consultation system for business halls based on AI glasses, including an AI glasses terminal, characterized in that: The AI ​​glasses terminal includes: Display module: used to realize virtual image projection; Sensor tracking module: used to track external information and collect information; Data processing module: used to receive and process data and generate result feedback; Knowledge base management module: including the knowledge base, which stores relevant knowledge and business data information and provides data support for generating result feedback; When the customer consultation system in the business hall based on AI glasses is running, it collects the user's voice and / or gesture information, parses the meaning, and generates corresponding feedback information through AR visualization and interaction.

2. The business hall customer consultation system based on AI glasses according to claim 1 is characterized in that: The display module projects a virtual image using a transparent display technology.

3. The business hall customer consultation system based on AI glasses according to claim 2 is characterized in that: The sensor tracking module includes: Head tracking sensor, used to capture the movement information of the user's head; Eye tracking sensor, used to capture changes in the user's gaze, providing a more accurate user experience; Environmental perception sensors are used to capture surrounding environment information and user posture information, providing support for the overlay of real-world and virtual information.

4. The AI ​​glasses-based business hall customer consultation system according to claim 3 is characterized in that: The data processing module includes: Computing and processing unit: Equipped with speech recognition and natural language processing models, the computing and processing unit is used to run algorithm models and process data to ensure system responsiveness; Augmented reality image processing unit: used for image processing, synthesizing virtual images and making real-time adjustments, accurately superimposing virtual information with the real environment, and displaying it.

5. The business hall customer consultation system based on AI glasses according to claim 1 is characterized in that: The data K in the knowledge base is expressed as: K = {k1, k2, ..., km}, where km is a knowledge item, and km includes question and answer pairs, handling procedures and related regulations.

6. The business hall customer consultation system based on AI glasses according to claim 5 is characterized in that: The knowledge base management module regularly updates and manages the data in the knowledge base.

7. The AI ​​glasses-based business hall customer consultation system according to any one of claims 1 to 6, characterized in that: The operation process of the business hall customer consultation system based on AI glasses includes the following steps: Speech recognition feedback: Collects user speech information, converts it into processable text, performs relevance queries based on the semantics of the text, and generates result feedback; Result visualization: Fuse the generated result feedback information with the real scene to generate a display image; Posture interaction: Capture user posture information and realize posture interaction.

8. The AI ​​glasses-based business hall customer consultation system according to claim 7 is characterized in that: The speech recognition feedback includes the following sub-steps: Speech signal processing: collects the user's speech information and generates the corresponding text sequence W; Semantic understanding: The text sequence W is passed into the natural language understanding model to understand the semantics and obtain the user's intention; Information retrieval: Based on the user's intention, relevant information is retrieved from the knowledge base and result feedback information is generated.

9. The AI ​​glasses-based business hall customer consultation system according to claim 8, characterized in that: The user posture information captured in the posture interaction includes user hand gestures.

10. The business hall customer consultation system based on AI glasses according to claim 1 is characterized in that: The AI ​​glasses terminal is also provided with a communication module, which is used to realize data interaction with external devices.

Images