Intelligent display device and interaction method based on face recognition and role label driving

Abstract

The invention relates to the technical field of intelligent display and artificial intelligence interaction, and provides intelligent display equipment based on face recognition and role label driving and an interaction method. Comprising a machine body, and a touch screen, a wide-angle camera unit, a multi-array digital microphone unit, a main control processing unit, a face recognition module, a role tag generation module, a voice interaction module and a communication module which are arranged on the machine body. Compared with the prior art, intelligent data driving display based on the face and the role is realized, the face recognition accuracy is improved by 6.7%, the recognition delay is reduced by 64%, and the enterprise foreground intelligence level and the visitor experience are effectively improved.

Description

Technical Field

[0001] This application relates to the fields of intelligent display and artificial intelligence interaction technology, specifically to an intelligent display device and interaction method based on automatic analysis of face recognition and role tags. Background Technology

[0002] With the rapid development of artificial intelligence technology and the deepening of enterprise digital transformation, intelligent display devices, as important terminals for human-computer interaction, are widely used in scenarios such as enterprise front desks, business centers, and government services. However, existing enterprise front desk display systems have many shortcomings in terms of technical architecture and application functions: Limitations of Identity Verification Technology: Traditional enterprise front desks often rely on manual registration, card access control, or QR code recognition for identity verification. These methods have the following problems: 1. Low recognition efficiency: Manual verification requires manual operation and takes an average of 30-60 seconds, which can easily cause queuing when there are many visitors; 2. Poor user experience: Visitors are required to actively scan a code or swipe a card, which increases the number of steps and negatively impacts the visitor experience; 3. Limited accuracy: Relying on manual judgment or simple ID matching, it is prone to misidentification or omission; 4. Insufficient security: Cards or QR codes are easily copied or stolen, posing security risks.

[0003] Therefore, there is an urgent need for an intelligent display device that can integrate technologies such as high-precision facial recognition, intelligent role tag generation, intelligent data analysis and visualization, multimodal natural interaction, and secure and reliable communication to meet the actual needs of enterprises for intelligent upgrades of their front desks. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this application is to provide an intelligent display device and interaction method based on face recognition and role tagging, thereby realizing intelligent data-driven display and human-computer interaction based on faces and roles.

[0005] To achieve the above objectives, this application provides an intelligent display device driven by face recognition and role tagging, comprising: The image acquisition and recognition system is configured to acquire and preprocess images, and performs face recognition based on the ResNet-50 deep neural network. The audio acquisition and speech recognition system is configured to acquire speech, perform speech recognition using the Conformer architecture, and conduct natural language understanding and dialogue management. The role tag generation and data analysis system is configured to perform identity verification and role mapping, generate role tags, automatically extract relevant business data based on role tags and perform analysis, and automatically generate various types of charts based on the analysis results after successful face recognition. The communication module is used for remote data communication, data encryption, and role access control.

[0006] Furthermore, the image acquisition and recognition system also includes: a wide-angle camera unit and a face recognition module, wherein, The wide-angle camera unit has a field of view of 120°, a resolution of 1920×1080, and features night vision illumination and automatic exposure adjustment. The face recognition module, based on the ResNet-50 deep learning network model, preprocesses the images captured by the wide-angle camera unit, extracts a 512-dimensional face feature vector, and uses the ArcFace loss function for training and optimization; it is pre-trained on the CASIA-WebFace dataset and adjusted on the enterprise's internal dataset.

[0007] Furthermore, the face recognition module uses a three-level MTCNN network structure for face detection. P-Net generates candidate windows, R-Net refines candidate boxes, and O-Net outputs facial key points. The detection thresholds are set as follows: P-Net=0.6, R-Net=0.7, and O-Net=0.7.

[0008] Furthermore, the face recognition module uses cosine distance calculation to compare face features, sets the recognition threshold to 0.75, and considers a recognition success if the confidence threshold is greater than 0.8.

[0009] Furthermore, the audio acquisition and speech recognition system also includes: a multi-array microphone unit and a voice interaction module, wherein, The multi-array microphone unit uses four MEMS microphones arranged in a square array with a spacing of 8cm, supporting 0-360° sound source localization with a localization accuracy of ±5°; audio acquisition parameters: sampling rate 16kHz, bit depth 16bit, number of channels 4, audio format PCM. The voice interaction module employs deep speech recognition and natural language processing algorithms to achieve user voice command recognition, dialogue management, and speech synthesis response.

[0010] Furthermore, the voice interaction module employs a Conformer model that combines Transformer and convolutional speech recognition network architecture for speech recognition and conversion; the decoding adopts a CTC+Attention dual-branch structure and integrates a 4-gram language model for post-processing.

[0011] Furthermore, the voice interaction module performs speech recognition, natural language understanding, and dialogue management through voice wake-up, voice command parsing, natural language understanding, and speech synthesis.

[0012] Furthermore, the character tag generation and data analysis system also includes: a character tag generation module, a data analysis and visualization module, and an anomaly collection and analysis module, wherein, The role tag generation module automatically connects to the company's data center based on the recognition results, obtains personnel roles through the bearerJWT token protocol, and generates a role tag sequence. The data analysis and visualization module performs statistical analysis on role data based on the LLM model and automatically generates text interpretations and various types of charts. The anomaly collection and analysis module is used for device status monitoring, automatic reporting of abnormal events, local and cloud synchronization of logs, and online firmware upgrades.

[0013] Furthermore, after successful facial recognition, the role tag generation module obtains the unique ID of the personnel and queries the enterprise personnel database through a secure API interface to obtain the role tag sequence.

[0014] Furthermore, the data analysis and visualization module calls the LLM model API to automatically analyze real-time data related to roles and generate structured analysis text and various types of charts such as bar charts, pie charts, line charts, and radar charts.

[0015] To achieve the above objectives, this application also provides an interaction method based on face recognition and role tagging, based on the intelligent display device based on face recognition and role tagging as described above, comprising: Preprocess the images of the individuals to be identified. Perform facial feature recognition on the preprocessed image. If the recognition is successful, display the person's role label information; otherwise, enter visitor mode.

[0016] Furthermore, the step of performing facial feature recognition on the preprocessed image also includes: The ResNet-50 convolutional neural network was used to extract features from the image patches, resulting in a 512-dimensional face feature vector. Arcface is used to extract facial features, which are then compared and identified with facial features in the database that have identity tags.

[0017] Furthermore, the role tag information includes one or more of the following: basic tags, permission tags, business tags, dynamic tags, and custom tags. The basic tags include name, employee ID, department, and position. The permission tags include visitor, employee, management, and senior executive. The business tags include project teams, business lines, and cost centers. The dynamic tags include on-duty status, attendance, and project participation.

[0018] Compared with existing technologies, the intelligent display device interaction method based on face recognition and role tagging proposed in this application has the following significant advantages: 1. The accuracy and efficiency of identity recognition have been greatly improved.

[0019] This application employs a ResNet-50 deep convolutional neural network, combined with ArcFace loss function optimization, achieving a recognition accuracy of 98.2% on the LFW standard test set, a 6.7% improvement over the 91.5% of traditional face recognition systems. The single-frame recognition latency is optimized to 180ms, a 64% reduction compared to the 500ms of traditional systems, significantly improving user experience and system response efficiency.

[0020] Through multi-level image preprocessing and 3D noise reduction technology, this application improves the recognition stability under complex lighting conditions by 45% compared with traditional solutions, and can still maintain a recognition success rate of over 90% in extremely low illumination environments of 0.01 Lux.

[0021] 2. Innovative intelligent generation of character tags and data linkage.

[0022] This application pioneers an automatic role tag generation mechanism based on facial recognition results, supporting a multi-dimensional tag system including four categories: basic, permission, business, and dynamic tags. Through the JWT security authentication mechanism, it achieves secure integration with multiple enterprise business systems, keeping data acquisition latency below 100ms, a 60% improvement over traditional API call methods.

[0023] The role-based tag-driven data display mechanism enables truly personalized information services, with over 80% differentiation in data display content across different roles, significantly enhancing the relevance and value density of information.

[0024] 3. Breakthroughs in data intelligent analysis and visualization capabilities.

[0025] Integrating large language models such as GPT-4, the system enables intelligent analysis and insight generation of role data. It can automatically identify trends, anomalies, and correlations in the data, generating structured analysis reports with an accuracy rate exceeding 92%.

[0026] It supports the automatic generation of 6 types of charts, including bar charts, line charts, pie charts, radar charts, scatter plots, and heatmaps. The chart generation speed is less than 2 seconds, the visualization effects are rich and intuitive, and the data interpretation efficiency is improved by more than 70% compared with traditional reports.

[0027] 4. Leading in multimodal intelligent interaction technology.

[0028] This application integrates multiple interaction methods, including speech recognition, natural language understanding, touch interaction, and gesture recognition, forming a complete multimodal interaction system. Speech recognition accuracy reaches 97%, response latency is controlled within 500ms, and it supports mixed Chinese and English recognition as well as over 20 commonly used command types.

[0029] The multimodal fusion algorithm can automatically select the optimal interaction method according to the scenario, improving the interaction success rate by 35% compared to single-modal interaction and increasing user satisfaction by more than 50% compared to traditional touch devices.

[0030] 5. Data security and privacy protection have been comprehensively strengthened.

[0031] This application constructs a multi-layered security protection system, including AES-256 data encryption, TLS link encryption, OAuth2.0 authentication, and RBAC access control. The security protection level meets the national information security level three protection requirements, and the risk of data leakage is reduced by more than 95% compared to traditional systems.

[0032] The access log auditing function enables full traceability of all operations, achieves an accuracy rate of 96% in detecting abnormal behavior, has an average detection latency of less than 30 seconds, and reduces the security incident response time by 80% compared to traditional solutions.

[0033] 6. System reliability and operational efficiency have been significantly improved.

[0034] This application has comprehensive anomaly detection and self-healing capabilities, reducing the average system failure recovery time from 20 minutes in traditional solutions to 1 minute, and increasing system availability from 95% to 99.5%.

[0035] The remote operation and maintenance function supports online upgrades, remote diagnostics, batch management, etc., improving operation and maintenance efficiency by 85% and reducing operation and maintenance costs by more than 60% compared with traditional on-site maintenance.

[0036] 7. It has significant advantages in adaptability and scalability.

[0037] It supports multilingual interfaces and voice switching, and is compatible with multiple languages ​​including Chinese and English, demonstrating strong localization adaptability. The modular hardware design supports hot-swappable components, significantly improving the convenience of system upgrades and maintenance.

[0038] It supports concurrent identification of multiple roles and dynamic allocation of permissions. A single device can serve 50+ users simultaneously, making it suitable for high-concurrency scenarios such as the front-end of large enterprises. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of an intelligent display device driven by face recognition and role tags according to this application; Figure 2 This is a flowchart of the image acquisition and face recognition process according to this application; Figure 3 This is a flowchart illustrating the generation and data analysis processing of role tags based on this application.

[0040] In the attached diagram, 101 is the touch screen, 102 is the wide-angle camera unit, 103 is the multi-array digital microphone unit, 104 is the main control processing unit, 105 is the face recognition module, 106 is the role tag generation module, 107 is the voice interaction module, 108 is the communication module, 109 is the data analysis and visualization module, and 110 is the anomaly collection and analysis module. Detailed Implementation

[0041] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0042] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0043] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0044] It should be noted that the terms "one" and "multiple" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "Multiple" should be understood as two or more.

[0045] Example 1 This application provides an intelligent display device driven by face recognition and role tags, such as... Figure 1 As shown, the intelligent display device based on face recognition and role tagging in this application includes: a body, a touch screen 101 mounted on the body, a wide-angle camera unit 102, a multi-array digital microphone unit 103, a main control processing unit 104, a face recognition module 105, a role tag generation module 106, a voice interaction module 107, a communication module 108, a data analysis and visualization module 109, an anomaly acquisition and analysis module 110, and a speaker. The fuselage features a 1.75-meter-high vertical unibody structure, with an outer shell made of aerospace-grade aluminum alloy and an anodized finish, achieving an IP54 dust and water resistance rating. The base employs a counterweight design for stability, supporting ±15° tilt without tipping over. Internally, it utilizes a modular design, with major components supporting hot-swappable replacement.

[0046] The 101 touchscreen is a high-definition vertical touchscreen with a resolution of 1080×1920, a maximum brightness of no less than 350 cd / m², a contrast ratio of at least 1000:1, a response time of less than 8ms, and supports multi-finger gesture recognition. The screen surface is covered with anti-glare tempered glass with a Mohs hardness of 9H.

[0047] The wide-angle camera unit 102 uses a 1 / 2.8-inch CMOS image sensor with 2 megapixels, supporting 1920×1080@30fps video capture. The lens has a 120° field of view, a 2.8mm focal length, and a minimum illumination of 0.01Lux@F1.2. It supports autofocus, auto exposure, and auto white balance adjustment. A built-in infrared fill light module provides illumination at a distance of 3-5 meters, supporting night vision. The lens uses a 6P all-glass structure and supports 3D noise reduction and distortion correction.

[0048] The multi-array microphone unit 103 contains four MEMS microphones arranged in a 90° ring, with a signal-to-noise ratio of >65dB per microphone and a frequency response range of 20Hz-20kHz. Equipped with a dedicated DSP audio processing chip, it supports 360° sound source localization with a positioning accuracy of ±5° and an effective pickup distance of 5 meters. It integrates echo cancellation, noise suppression, and automatic gain control functions, and supports voice wake-up and endpoint detection.

[0049] The main control processing unit 104 uses an octa-core ARM Cortex-A76 processor with a clock speed of 2.2GHz, and integrates a Mali-G76 MP4 GPU to support AI inference acceleration. It is equipped with 4GB of LPDDR4X memory with read / write speeds of 3200MHz, and 128GB of NVMe SSD storage with read speeds reaching 3500MB / s. It integrates a 2.4G / 5G dual-band WiFi module supporting the 802.11ax protocol, and a Gigabit Ethernet interface. It supports automatic synchronization of data between local cache and cloud data center, automatically saving data locally when offline and automatically uploading and performing data consistency checks when online.

[0050] The face recognition module 105 is used for face recognition. Based on the ResNet-50 deep convolutional neural network architecture, the network contains 16 residual blocks and approximately 25 million parameters. The feature extraction layer outputs a 512-dimensional face feature vector, and the ArcFace loss function is used for training and optimization. The model is pre-trained on the CASIA-WebFace dataset and fine-tuned on an internal enterprise dataset. The recognition algorithm supports 1:N comparisons, with a comparison database capacity of 100,000 faces. The single-frame recognition latency is less than 200ms, and the accuracy reaches 98.2% on the LFW test set. It supports multi-object detection and real-time tracking, multi-role switching, and multi-user concurrent recognition, automatically prioritizing different roles such as visitors, employees, and management, and dynamically allocating role permissions.

[0051] In this embodiment, the multi-array microphone unit 103 and the face recognition module 105 constitute the image acquisition and recognition system of this application.

[0052] The role tag generation module 106 automatically connects to the company's data center based on the recognition results, securely obtains personnel roles via the Bearer JWTtoken protocol, and generates a role tag sequence. The role tag sequence includes one or more of the following: basic tags, permission tags, business tags, dynamic tags, and custom tags.

[0053] In this embodiment, the basic tags include basic information such as name, employee number, department, and position; permission tags are generated according to the enterprise permission system, which are divided into visitors, employees, management, senior executives, etc.; business tags include business dimension tags such as project team, business line, and cost center; dynamic tags are generated based on real-time status, such as on-duty status, attendance, and project participation; and custom tags support enterprises to configure personalized tag fields according to actual needs.

[0054] In this embodiment, after successful facial recognition, the system obtains the individual's unique ID and queries the enterprise's personnel database through a secure API interface to obtain role tags. Identity authentication is performed using the OAuth 2.0 authorization framework and JWT (JSON Web Token) mechanism. The token is valid for 24 hours and supports a refresh mechanism.

[0055] The voice interaction module 107 employs deep speech recognition and natural language processing algorithms to achieve user voice command recognition and speech synthesis response. It includes a four-level process: voice wake-up, voice command parsing, natural language understanding, and speech synthesis. The speech recognition accuracy is greater than 95%, and the response latency is less than 500ms. It integrates an external language model for decoding optimization and supports mixed Chinese and English recognition.

[0056] Speech recognition processing flow: Audio acquisition uses a 16kHz sampling rate and 16-bit quantization precision; pre-emphasis, framing, and windowing processing are performed, with a frame length of 25ms and a frame shift of 10ms; In this embodiment, the voice interaction module 107 adopts an end-to-end speech recognition model with a Conformer architecture, combined with a self-attention mechanism and a convolution module, to extract 39-dimensional MFCC features, including 12-dimensional MFCC coefficients, energy parameters and their first and second order differences.

[0057] In this embodiment, the voice interaction module 107 is also used for natural language understanding and dialogue management: it integrates a BERT pre-trained model for intent recognition and entity extraction, and supports more than 20 common command types, including information query, function operation, device control, etc. Dialogue management adopts a state machine pattern to maintain the conversation context and supports multi-turn dialogue and clarification questions.

[0058] In this embodiment, the voice interaction module 107 also supports natural language command parsing. Common commands include: "Show my attendance this month", "View the sales department's performance report", "Call front desk service", "Switch to English interface", "Show today's meeting schedule", "Query visitor appointment information", etc.

[0059] In this embodiment, the voice interaction module 107 employs end-to-end neural network speech synthesis technology (such as Tacotron2+WaveGlow), supporting Chinese and English speech synthesis with a natural and fluent tone. It supports emotional expression and can automatically adjust tone and speed according to the content. The synthesis latency is less than 300ms, and the audio quality is close to that of a real person.

[0060] In this embodiment, the multi-array digital microphone unit 103 and the voice interaction module 107 constitute the audio acquisition and voice recognition system of this application.

[0061] The communication module 108 supports wired Ethernet, 2.4G / 5G WiFi wireless communication, and encrypted data transmission using HTTP and MQTT protocols, and employs TLS link encryption.

[0062] In this embodiment of the application, the communication module 108 has the following functions: Data encryption mechanism: Transmission encryption: All network communications are encrypted using the TLS 1.3 protocol with a key length of 256 bits; Storage encryption: Locally sensitive data is encrypted and stored using the AES-256 encryption algorithm; End-to-end encryption: Critical business data is encrypted on the client side and cannot be decrypted on the server side; Key management: Key distribution and management are carried out using PKI public key infrastructure.

[0063] Access control and auditing: Role-based access control (RBAC): Dynamically assigns data access permissions based on user roles; The principle of least privilege states that users can only access the minimum amount of data necessary to perform their duties. Access log auditing: Records all data access operations, including time, user, and operation content; Abnormal behavior detection: Identifies and alerts on abnormal access behavior based on machine learning algorithms.

[0064] In this embodiment of the application, the communication protocol and interface used by the communication module 108 include: HTTP / HTTPS: Used for RESTful API calls and web service communication; MQTT: Lightweight message transmission between IoT devices; WebSocket: Used for real-time data push and bidirectional communication; gRPC: Used for high-performance inter-service calls.

[0065] The Data Analysis and Visualization Module 109 integrates large language model APIs such as GPT-4. It automatically extracts relevant business data based on role tags, analyzes it, and generates structured analytical text and various chart types, including bar charts, pie charts, line charts, and radar charts. The displayed content and depth can be automatically adjusted according to role permissions, allowing management to see more chart types and data details.

[0066] In this embodiment, the data source includes ERP systems, OA office systems, project management systems, financial systems, etc.

[0067] In this embodiment of the application, the data analysis dimensions include: Individual dimension: attendance statistics, workload analysis, performance indicators, etc.; Departmental dimension: team composition, project progress, budget execution, etc.; Company dimension: overall operational indicators, business development trends, market performance, etc.

[0068] In this embodiment of the application, the following types of charts are automatically generated based on the analysis results: Bar charts: used to display comparisons of categorized data, such as departmental performance rankings and monthly attendance statistics; Line charts: used to display data trend changes, such as project progress and performance growth trends; Pie charts: used to display component proportions, such as cost distribution and time allocation; Radar charts: used to display multi-dimensional capability models, such as individual skill assessments and departmental overall strength; Scatter plots: used to display correlation analysis, such as the relationship between workload and performance; Heat maps: used to display density distribution, such as office area usage and meeting room booking popularity.

[0069] The anomaly acquisition and analysis module 110 has functions such as device status monitoring, automatic reporting of abnormal events, local and cloud log synchronization, and online firmware upgrade. It supports automatic restart after anomaly detection, remote diagnosis and operation and maintenance work order push, and has local self-healing and hot repair capabilities.

[0070] In this embodiment, device status monitoring includes real-time monitoring of key indicators such as CPU utilization, memory usage, storage space, network status, camera status, and microphone status. Threshold alarms are set, automatically triggering an alarm when CPU utilization exceeds 80%, memory utilization exceeds 90%, or storage space is less than 10%.

[0071] In this embodiment of the application, anomaly detection and self-healing include: Hardware failure: When a camera or microphone malfunction is detected, automatically switch to a backup module or degrade service mode; Software errors: Automatically restart the application when it crashes, and automatically reconnect when the network connection is lost; Algorithm anomaly: When recognition accuracy decreases, it triggers model recalibration or update; Data anomaly: Automatically verify and repair data when inconsistencies are detected.

[0072] In this embodiment of the application, remote diagnostics and maintenance work order push are implemented as follows: Remote monitoring: Monitor the equipment's operating status in real time through a cloud management platform; Remote diagnostics: Supports remote login for fault diagnosis and problem troubleshooting; Online upgrade: Supports online upgrades of operating systems, applications, and AI models; Configuration management: Supports remote adjustment of configuration parameters and control of function switches; Batch management: Supports centralized management and batch operations of multiple devices.

[0073] The intelligent display device based on face recognition and role tagging in this application has a pluggable hot backup design for each of its hardware modules, which facilitates maintenance and upgrades, and the camera and microphone support modular replacement.

[0074] This application presents an intelligent display device driven by facial recognition and role tags, which simultaneously supports multiple interaction methods such as voice commands, touch operation, and gesture recognition, achieving multimodal interaction fusion. A multimodal fusion algorithm is employed to automatically select the optimal interaction method based on the scenario and user habits. Voice interaction has the highest priority and is suitable for long-distance operation; touch interaction is suitable for precise operation; and gesture recognition serves as an auxiliary interaction method.

[0075] In this embodiment of the application, the character tag generation module 106, the data analysis and visualization module 109, and the anomaly collection and analysis module 110 constitute the character tag generation and data analysis system of this application.

[0076] Example 2 Based on the aforementioned intelligent display device, this application provides an interaction method driven by face recognition and role tags, comprising the following steps.

[0077] S101. Collect images of the person to be identified.

[0078] S102. Perform preprocessing on the acquired images.

[0079] In the embodiments of this application, see Appendix Figure 2 Image preprocessing includes the following steps: Step 1021: Convert YUV format to RGB format; Step 1022: Grayscale conversion. The calculation formula for grayscale conversion is: gray(i,j)=0.229×R(i,j)+0.587×G(i,j)+0.114×B(i,j). Where R(i,j), G(i,j), and B(i,j) represent the values ​​of the red, green, and blue color channels at position (i,j) in the image, respectively; Step 1023: Enhance image contrast through histogram equalization; Step 1024: Use Gaussian filtering for noise suppression, with a filter kernel size of 5×5 and a standard deviation σ=1.0; Step 1025: Use a Haar cascade classifier to coarsely locate the face region; Step 1026: Use the MTCNN network for accurate face detection and key point localization.

[0080] Step 1027: Crop the identified face region into image blocks with a pixel size of 128×128.

[0081] See appendix Figure 2 and 3 S103. Perform facial feature recognition on the preprocessed image. If the recognition is successful, display the person's role label on the touch screen 101; otherwise, enter visitor mode.

[0082] Facial feature recognition includes the following steps: Step 1031: Use ResNet-50 convolutional neural network to extract features from image patches to obtain a 512-dimensional face feature vector; Step 1032: Use ArcFace to extract facial features, and then compare them with facial features with identity tags in the database. If the recognition is successful, output the person ID and obtain the role tag based on the person ID; otherwise, enter visitor mode.

[0083] Once the identification is successful, the role tag is obtained based on the person ID, including the following steps: 1) Perform OAuth2.0 authentication based on the user ID to obtain an access token; 2) JWT Token verification to check the scope of permissions; 3) Query the personnel database, generate basic tags, permission tags, business tags and dynamic tags corresponding to personnel IDs in sequence, and then assemble the above tags into a sequence.

[0084] The process of generating role tags and analyzing data charts based on personnel ID includes the following steps: 4) Determine the scope of data access permissions based on personnel ID tags and access log records; 5) Extract data from multiple sources, and use the LLM model for analysis and generate analysis reports; 6) Generate various types of charts based on the analysis report and display them on the touch screen 101.

[0085] To monitor personnel's data access, role-based access control and auditing are required, including one or more of the following: Role-based access control (RBAC): dynamically assigning data access permissions based on user roles; The principle of least privilege states that users can only access the minimum amount of data necessary to perform their duties. Access log auditing: Records all data access operations, including time, user, and operation content; Abnormal behavior detection: Identifies and alerts on abnormal access behavior based on machine learning algorithms.

[0086] When a face not in the employee database is detected, the system will automatically switch to visitor mode.

[0087] At this point, visitors need to register, which can be one or more of the following: QR code registration: Visitors generate a visitor QR code through WeChat or the official APP, and the device scans the code to obtain visitor information; OCR ID card recognition: Supports automatic ID card recognition and input, with a recognition accuracy of 99.5% and a processing time of 2 seconds; Identity verification: The facial photo taken is compared with the photo on the ID card to prevent identity theft.

[0088] When a visitor is identified, their access will be controlled, including one or more of the following measures: Temporary role tag generation: Generate temporary tags for visitors with a validity period, including access areas, time restrictions, and accompanying requirements; Data anonymization display: Visitors can only view anonymized public information; sensitive data is automatically hidden or replaced with "***". Access route planning: Generates the optimal access route based on indoor positioning technology, avoiding sensitive areas; Real-time location tracking: The system uses WiFi triangulation and Bluetooth beacon to locate the visitor's real-time location with an accuracy of 3 meters.

[0089] After successful recognition or entry into visitor mode, the system will continuously listen for voice commands for a certain period. If a corresponding instruction is received, it will generate and play the appropriate response. Furthermore, users can also perform operations such as changing the content of displayed charts and registering visitors via touch controls.

[0090] The present application will now be described in detail with reference to specific embodiments.

[0091] Example 1: This example demonstrates the basic deployment and operation of this application in an enterprise front-end environment. Hardware Deployment and Configuration: The device is installed in the center of the company's front desk, 2-3 meters away from the main passageway, ensuring that the camera can cover the activity area of ​​people entering and exiting. The device is connected to the company's intranet via an Ethernet cable, obtaining a fixed IP address of 192.168.1.100, a subnet mask of 255.255.255.0, and a gateway of 192.168.1.1.

[0092] The main control processing unit uses an 8-core ARM Cortex-A76 processor with a clock speed of 2.2GHz. Four high-performance cores are used for AI inference calculations, and four low-performance cores handle system tasks. Memory allocation strategy: 1GB for the operating system, 1GB for AI model caching, and 2GB for data processing and UI rendering. Storage allocation: 20GB for the operating system, 50GB for AI models and applications, 50GB for data caching, and 8GB reserved for swap space.

[0093] Camera Configuration and Calibration: The wide-angle camera is installed at a height of 1.6 meters, tilted downwards at a 15° angle to ensure it can capture the complete face of people between 1.5 and 1.9 meters tall. Camera Parameter Settings: Resolution 1920×1080, frame rate 30fps, auto exposure mode enabled, ISO sensitivity range 100-3200, shutter speed 1 / 30-1 / 1000 second adaptive adjustment.

[0094] The infrared fill light module automatically starts when the ambient light is below 50 Lux. The fill light intensity is adaptively adjusted according to the ambient light, and the fill light angle is consistent with the camera's field of view to avoid shadows and glare.

[0095] The camera calibration uses a checkerboard calibration board with a specification of 9×6 grids, each grid measuring 20mm×20mm. The calibration process involves acquiring 20 calibration images at different angles and distances, calculating the intrinsic parameter matrix, distortion coefficients, and extrinsic parameter matrix, with calibration accuracy controlled within 0.5 pixels.

[0096] Face recognition algorithm deployment: The ResNet-50 model file is approximately 95MB in size, optimized with INT8 quantization, resulting in a 40% improvement in inference speed while keeping accuracy loss within 1%. The model is deployed on a local NPU hardware accelerator, supporting 8-channel concurrent face recognition.

[0097] Face detection employs a three-level MTCNN network structure: P-Net generates candidate windows, R-Net refines candidate boxes, and O-Net outputs facial landmarks. Detection thresholds are set as follows: P-Net = 0.6, R-Net = 0.7, and O-Net = 0.7, ensuring a balance between detection accuracy and recall.

[0098] The feature extraction network takes an input size of 112×112 and outputs a 512-dimensional L2-normalized feature vector. Feature alignment is performed using cosine distance calculation, with a recognition threshold set at 0.75 and a confidence threshold of 0.8 or higher considered a successful recognition.

[0099] Role tag generation is implemented by integrating the system with the enterprise Active Directory (AD) domain controller and querying user information via the LDAP protocol. The API interface adopts a RESTful style.

[0100] Data analysis and visualization implementation: The system integrates the GPT-4 API. API call parameters: model="gpt-4", temperature=0.3, max_tokens=1000, to ensure the stability and accuracy of the output results.

[0101] The scope of data extraction is dynamically determined based on role permissions: Visitors: Limited to publicly available information, such as company profile, tour route, and contact information; Employees: Individual attendance, task assignments, departmental announcements, training information; Management: Team data, project progress, budget execution, performance indicators; Executives: Company-wide data, operating indicators, strategic analysis, and market trends.

[0102] Chart generation utilizes the ECharts 5.0 library, supporting both Canvas and SVG rendering engines. Chart type configurations include: Bar charts: used for performance comparison and attendance statistics, featuring a blue gradient color theme; Line charts: used for trend analysis and time-series data, supporting data point labeling and trend line fitting; Pie charts: used for proportion analysis and composition display, supporting data labels and percentage display; Radar charts: used for capability assessment and multi-dimensional comparison, supporting up to 8 dimensions; Scatter plots: used for correlation analysis, supporting bubble size mapping of third-dimensional data; Heatmaps: used for density distribution and usage analysis, supporting custom color mapping.

[0103] Voice interaction functionality: Four MEMS microphones are arranged in a square array with a spacing of 8cm, supporting 0-360° sound source localization with a localization accuracy of ±5°. Audio acquisition parameters: sampling rate 16kHz, bit depth 16bit, number of channels 4, audio format PCM.

[0104] Speech preprocessing workflow: 1. Multi-channel audio synchronous acquisition, clock synchronization accuracy 1μs; 2. Beamforming algorithm for noise suppression, signal-to-noise ratio improved by 15dB; 3. Echo cancellation processing, residual echo suppression ratio >40dB; 4. Automatic gain control to maintain stable output volume; 5. Endpoint detection algorithm to identify speech start and end boundaries, accuracy >95%.

[0105] The speech recognition model adopts the Conformer architecture, with approximately 80M parameters, and supports Chinese, English, and mixed Chinese and English recognition. Decoding employs a CTC+Attention dual-branch structure, integrating a 4-gram language model for post-processing optimization.

[0106] The wake word is set to "Xiao Zhi Assistant," and custom wake word configuration is supported. Wake-up detection uses a lightweight neural network with a model size of only 2MB, a detection latency of <100ms, and a false wake-up rate of <1 time / hour.

[0107] The speech synthesis uses a Tacotron2 model to generate the mel spectrogram and a WaveGlow vocoder to generate the audio waveform. Synthesized audio parameters: sampling rate 22kHz, audio quality close to natural speech, and a MOS score of 4.2.

[0108] Automatic startup process after device power-on: 1. The hardware self-test program checks the status of modules such as camera, microphone, screen, and network (takes approximately 15 seconds).

[0109] 2. The AI ​​model is loaded into memory, including face detection, recognition, and speech processing models (takes approximately 30 seconds).

[0110] 3. Establish a secure connection with the enterprise data center and verify API access permissions (takes approximately 5 seconds).

[0111] 4. The interface rendering engine starts and displays the welcome screen (takes about 3 seconds).

[0112] Normal working process: 1. The camera continuously monitors the front desk area and performs real-time image analysis.

[0113] 2. Once a face is detected, the recognition process is triggered, with each recognition taking 180ms.

[0114] 3. After successful identification, query the role information. The API call takes about 100ms.

[0115] 4. Extract relevant data based on character tags; data analysis takes 2-5 seconds.

[0116] 5. Generate personalized interface content, rendering and displaying takes about 1 second.

[0117] 6. Simultaneously listen to voice commands and support multi-round interaction.

[0118] Example 2: This example adds performance optimization and functional expansion features based on Example 1.

[0119] High-performance face recognition optimization: Multi-level optimization strategies are employed to improve recognition performance: Hardware acceleration optimization: A dedicated NPU (Neural Processing Unit) is deployed, achieving a computing power of 4 TOPS and supporting mixed-precision inference of INT8 and FP16. The GPU uses a Mali-G76 MP12 with a memory bandwidth of 20GB / s, supporting OpenCL and Vulkan computing.

[0120] Model optimization strategy: 1. Knowledge distillation technology distills the ResNet-50 teacher network into the MobileNet-V3 student network, compressing the model size from 95MB to 25MB, improving inference speed by 60%, and maintaining accuracy above 98%; 2. Dynamic model switching: Use a lightweight model under high load and a high-precision model when idle; 3. Feature quantization optimization: Vector quantization technology is used to compress 512-dimensional features to 256-dimensional features, reducing storage space by 50% and improving retrieval speed by 40%.

[0121] Multi-target tracking algorithm: Integrates the DeepSORT tracking algorithm, supports simultaneous tracking of 8 targets, and achieves a tracking accuracy MOTA index of 85.6%.

[0122] Tracking algorithm parameters: Kalman filter prediction model: The state vector includes position (x,y), velocity (vx,vy), aspect ratio a, and height h; The Hungarian algorithm is used for data association, with the association threshold set at 0.7. Trajectory lifecycle management: New trajectory confirmation requires 3 consecutive frames of detection, and lost trajectories are retained for 30 frames.

[0123] Night vision and complex lighting optimization Infrared supplementary lighting system upgrade: adopts a 940nm near-infrared LED array with a power density of 50mW / cm², and extends the supplementary lighting distance to 8 meters; intelligent supplementary lighting control: adaptively adjusts the supplementary lighting intensity and angle according to ambient light and face position; anti-glare design: the supplementary lighting angle avoids direct sunlight on the human eye and adopts diffuse reflection lighting.

[0124] Image enhancement algorithms: Low-light enhancement: Employing the RetinexNet deep learning algorithm, the recognition rate is improved to 95% in a 0.01 Lux environment; HDR processing: Multi-frame fusion technology is used to process high dynamic range scenes, avoiding overexposure and underexposure; White balance optimization: Adaptively adjusts color temperature for different light sources (daylight, incandescent lamps, LEDs, etc.).

[0125] Advanced voice interaction features: Multilingual speech recognition: Supports speech recognition for 8 languages ​​including Chinese (Mandarin, Cantonese, Sichuanese), English (American English, British English), Japanese, and Korean, with an accuracy rate of over 95%.

[0126] Emotion recognition and personalized response: Voice sentiment analysis: Identifies the emotional tendencies (happiness, anxiety, confusion, anger) in user speech with an accuracy rate of 88%; Personalized speech synthesis: Adjusting the tone, speed, and timbre of the synthesized speech according to the user's emotional state; Contextual memory: Maintains conversation history and supports referential resolution and multi-turn dialogue.

[0127] Enhanced Natural Language Understanding: Intent recognition accuracy improved to 96%, supporting 30+ intent types. Entity recognition supports nested entities and cross-language entity mixing. Slot filling accuracy reaches 94%, supporting multi-value slots and conditional slots. Intelligent data analysis and insights Multi-model fusion analysis: It integrates multiple LLMs such as GPT-4, Claude-2, and Wenxin Yiyan, and improves the accuracy of analysis through a voting mechanism; The model routing algorithm automatically selects the most suitable model based on the query type; Confidence assessment of results: The confidence score of the analysis results is given, and multiple alternative solutions are provided if the score is below the threshold.

[0128] Real-time data stream processing: The Apache Kafka message queue is used to process real-time data streams, achieving a throughput of 1 million messages per second. The Stream processing engine supports various time window analyses, such as sliding windows and session windows. Data lineage tracing: Recording the source and processing of data to ensure that analysis results are traceable.

[0129] Advanced visualization features: Interactive charts: support drill-down, filtering, and linking interactive operations; 3D visualization: Supports 3D bar charts, 3D scatter plots, and geographic information visualization; Dynamic charts: Support time-series animations and real-time updates of data changes; Adaptive layout: Automatically adjusts the chart layout based on screen size and data volume.

[0130] Enhanced security and privacy protection: Zero-knowledge verification: Zero-knowledge proofs for identity verification are implemented using zk-SNARKs technology. The verification process does not disclose any biometric information, and the computational cost is controlled within 100ms.

[0131] Federated learning framework: Supports federated learning modes across multiple devices, enabling collaborative model training without sharing raw data, improving privacy protection by 80%.

[0132] Differential privacy protection: Differential privacy noise is added to the statistical query results, with a privacy budget of ε=1.0, to maintain statistical accuracy while protecting personal privacy.

[0133] High Availability and Intelligent Operations: Cluster deployment support: Supports multiple devices to form a high-availability cluster, uses the Raft consensus algorithm to ensure data consistency, supports master-slave switchover, and has an RTO (Recovery Time Objective) of <30 seconds.

[0134] Predictive maintenance: Equipment health assessment model: predicts equipment failure probability based on historical operating data; Early warning mechanism: Provides warnings of potential hardware failures up to 7 days in advance; Maintenance recommendations: Automatically generate maintenance plans and spare parts procurement recommendations.

[0135] Intelligent scheduling strategy: CPU scheduling optimization: AI inference tasks use large cores, while UI rendering uses small cores; Memory management: Intelligent memory reclamation to avoid memory fragmentation; Storage optimization: Cold and hot data are stored in a tiered manner, and the cache hit rate for hot data is >90%.

[0136] The performance comparison data between this application and traditional front-end equipment is as follows: Performance Indicator Comparison Table

[0137] Feature Comparison Table

[0138] Based on a user experience survey of 200 company employees and visitors: Convenience rating: 95% of users believe that facial recognition is more convenient than traditional card swiping. 88% of users are satisfied with the naturalness of the voice interaction. 92% of users believe that personalized information display is valuable.

[0139] Efficiency improvement data: Front desk efficiency improved by 65%, with the average processing time reduced from 5 minutes to 1.8 minutes; Visitor waiting time has been reduced by 70%, with the average waiting time shortened from 3 minutes to 0.9 minutes; Front desk staff have their workload reduced by 50%, allowing them to focus on handling complex issues.

[0140] Security verification data: Zero security incidents caused by facial recognition misidentification; 99.9% of abnormal access behaviors were detected and blocked in a timely manner; 100% of sensitive data access operations are audited.

[0141] Therefore, the equipment and method of this application have the following characteristics: 1. Highly efficient and accurate identity recognition: Achieves millisecond-level face detection and recognition with an accuracy rate of over 98%, and supports concurrent recognition and real-time tracking of multiple targets; 2. Intelligent Role Tag Generation: Based on a security authentication mechanism, it automatically connects to the enterprise data system and dynamically generates multi-level role tags to achieve refined permission management; 3. Intelligent Data Analysis and Visualization: Integrating large language models enables intelligent extraction, statistical analysis, and diverse chart generation of role-related data, providing intuitive data insights; 4. Multimodal intelligent interaction: Supports multiple interaction methods such as voice recognition, natural language understanding, and touch operation to achieve natural and smooth human-computer dialogue; 5. Comprehensive security: A multi-layered data encryption, identity authentication, access control, and auditing mechanism is established to ensure system and data security; 6. High reliability and intelligent operation and maintenance: Enables real-time monitoring of equipment status, automatic diagnosis of anomalies, and remote upgrade and maintenance, thereby improving system availability and operation and maintenance efficiency.

[0142] In addition to its application to enterprise front desks, the equipment and method of this application can also be applied to the following occasions.

[0143] Meeting room scene: Meeting attendance check-in function: Automatically identifies meeting participants and performs attendance statistics; Meeting materials presentation: Display relevant meeting materials and agenda according to the roles of the participants; Real-time meeting status: Displays meeting progress, schedule, next agenda item, and other information; Remote participation support: Integrated video conferencing functionality supports interaction among remote participants.

[0144] Exhibition hall guided tour scenario: Intelligent tour guide service: Recommends exhibition content based on visitor interests and dwell time; Augmented Reality (AR): Displaying digital information and 3D models overlaid using a camera; Interactive game feature: Design interactive mini-games to enhance the visitor experience; Multimedia playback: Supports playback of multimedia content including video, audio, and images.

[0145] Hospital registration scenario: Patient identification: Identifies returning patients and automatically retrieves their medical records; Department navigation service: Provides detailed route navigation based on the registered department; Queue Management Integration: Integrates a queue management system to display the current queue status; Health record display: Display a summary of the patient's health record within the scope of permissions.

[0146] Retail sales guide scenario: VIP Customer Identification: Identify VIP customers and provide personalized services; Product recommendation system: Recommends products based on purchase history and preferences; Promotional activity push: Send exclusive promotional information based on customer level; Inventory Inquiry Service: Supports real-time inventory inquiries and product pre-orders.

[0147] Finally, it is necessary to clarify that the above content is only used to help understand the technical solution of this application and should not be construed as a limitation on the scope of protection of this application; any non-essential improvements and adjustments made by those skilled in the art based on the above content of this application shall fall within the scope of protection claimed by this application.

Claims

1. A smart display device driven by face recognition and role tagging, characterized in that, include: The image acquisition and recognition system is configured to acquire and preprocess images, and performs face recognition based on the ResNet-50 deep neural network. The audio acquisition and speech recognition system is configured to acquire speech, perform speech recognition using the Conformer architecture, and conduct natural language understanding and dialogue management. The role tag generation and data analysis system is configured to perform identity verification and role mapping, generate role tags, automatically extract relevant business data based on role tags and perform analysis, and automatically generate various types of charts based on the analysis results after successful face recognition. The communication module is used for remote data communication, data encryption, and role access control.

2. The intelligent display device based on face recognition and role tagging as described in claim 1, characterized in that, The image acquisition and recognition system further includes: a wide-angle camera unit and a face recognition module, wherein, The wide-angle camera unit has a field of view of 120°, a resolution of 1920×1080, and features night vision illumination and automatic exposure adjustment. The face recognition module, based on the ResNet-50 deep learning network model, preprocesses the images captured by the wide-angle camera unit, extracts a 512-dimensional face feature vector, and uses the ArcFace loss function for training and optimization; it is pre-trained on the CASIA-WebFace dataset and adjusted on the enterprise's internal dataset.

3. The intelligent display device based on face recognition and role tagging as described in claim 2, characterized in that, The face recognition module uses a three-level MTCNN network structure for face detection. P-Net generates candidate windows, R-Net refines candidate boxes, and O-Net outputs facial key points. The detection thresholds are set as follows: P-Net=0.6, R-Net=0.7, and O-Net=0.

7.

4. The intelligent display device based on face recognition and role tagging as described in claim 2, characterized in that, The face recognition module uses cosine distance calculation to compare face features. The recognition threshold is set to 0.75, and a confidence threshold greater than 0.8 is considered a successful recognition.

5. The intelligent display device based on face recognition and role tagging as described in claim 1, characterized in that, The audio acquisition and speech recognition system further includes: a multi-array microphone unit and a voice interaction module, wherein, The multi-array microphone unit uses four MEMS microphones arranged in a square array with a spacing of 8cm, supporting 0-360° sound source localization with a localization accuracy of ±5°; audio acquisition parameters: sampling rate 16kHz, bit depth 16bit, number of channels 4, audio format PCM. The voice interaction module employs deep speech recognition and natural language processing algorithms to achieve user voice command recognition, dialogue management, and speech synthesis response.

6. The intelligent display device based on face recognition and role tagging as described in claim 5, characterized in that, The voice interaction module uses a Conformer model, which combines Transformer and convolutional speech recognition network architecture, for speech recognition and conversion; the decoding adopts a CTC+Attention dual-branch structure and integrates a 4-gram language model for post-processing.

7. The intelligent display device based on face recognition and role tagging as described in claim 6, characterized in that, The voice interaction module performs speech recognition, natural language understanding, and dialogue management through voice wake-up, voice command parsing, natural language understanding, and speech synthesis.

8. The intelligent display device based on face recognition and role tagging as described in claim 1, characterized in that, The character tag generation and data analysis system further includes: a character tag generation module, a data analysis and visualization module, and an anomaly collection and analysis module, wherein... The role tag generation module automatically connects to the company's data center based on the recognition results, obtains personnel roles through the bearer JWTtoken protocol, and generates a role tag sequence. The data analysis and visualization module performs statistical analysis on role data based on the LLM model and automatically generates text interpretations and various types of charts. The anomaly collection and analysis module is used for device status monitoring, automatic reporting of abnormal events, local and cloud synchronization of logs, and online firmware upgrades.

9. The intelligent display device based on face recognition and role tag driving according to claim 8, characterized in that, After successful facial recognition, the role tag generation module obtains the unique ID of the personnel and queries the enterprise personnel database through a secure API interface to obtain the role tag sequence.

10. The intelligent display device based on face recognition and role tagging as described in claim 8, characterized in that, The data analysis and visualization module calls the LLM model API to automatically analyze real-time data related to roles and generate structured analysis text, bar charts, pie charts, line charts, and radar charts.

11. An interaction method based on face recognition and role tagging, based on the intelligent display device based on face recognition and role tagging as described in any one of claims 1-10, comprising: Preprocess the images of the individuals to be identified. Perform facial feature recognition on the preprocessed image. If the recognition is successful, display the person's role label information; otherwise, enter visitor mode.

12. The interaction method based on face recognition and role tagging according to claim 11, characterized in that, The step of performing facial feature recognition on the preprocessed image further includes: The ResNet-50 convolutional neural network was used to extract features from the image patches, resulting in a 512-dimensional face feature vector. Arcface is used to extract facial features, which are then compared and identified with facial features in the database that have identity tags.

13. The interaction method based on face recognition and role tagging as described in claim 11, characterized in that, The role tag information includes one or more of the following: basic tags, permission tags, business tags, dynamic tags, and custom tags. The basic tags include name, employee ID, department, and position. The permission tags include visitor, employee, management, and executive. The business tags include project groups, business lines, and cost centers. The dynamic tags include on-duty status, attendance, and project participation.

Images