AI-Based Customer Re-Identification System for Integrated Environment Control and Personalized Interaction

Abstract

The present invention relates to an integrated environment control and customized interaction system through AI-based individual customer recognition, and is characterized by comprising: an edge layer installed at a store site to receive a customer's video stream, extract an appearance feature vector and a face feature vector from the video stream, and transmit the extracted vector to a core server; a core server layer that receives the feature vectors from the edge layer, integrates and analyzes a plurality of customer features including the appearance feature vector and the face feature vector using a probabilistic fusion model to identify the customer based on an anonymized ID (anon_user_id), and calculates a customized service decision based on the anonymized ID; a device control layer that controls at least one of a shooting mode, content display, and lighting settings in real time according to the decision of the core server layer; and a data management layer that stores customer profile data and product content data linked to the anonymized ID and records all access history of the system as a security log. According to the present invention as described above, the accuracy of individual customer recognition and re-identification is innovatively improved compared to existing technologies through the fusion of multiple evidence such as Re-ID, facial recognition, and BLE, and by meticulously analyzing the external characteristics of customers (fashion style, logo recognition, etc.), it is possible to provide real-time appearance-based hyper-personalized services that go beyond simple purchase history-based recommendations.

Description

Technology Field

[0001] The present invention relates to a retail technology system that performs individual re-identification of customers in an offline store using artificial intelligence and computer vision technology, and provides real-time personalized interaction and customized product recommendation services by integrally controlling in-store environmental devices (IoT / devices) based on the analyzed external characteristics and behavioral patterns of customers. Background Technology

[0002] Existing in-store customer management and service delivery systems primarily rely on manual methods such as membership cards, QR codes, or mobile application logins. These existing technologies have the following clear limitations.

[0003] First of all, it is difficult to optimize services by recognizing individual characteristics in real time or automatically responding to changes in customer behavior from the moment a customer enters the store.

[0004] And since the existing method is based only on data voluntarily provided by customers (purchase history, etc.), it cannot provide immediate and in-depth personalized services that reflect customers' current external characteristics (fashion style, hairstyle, etc. of the day) or momentary interests.

[0005] Furthermore, when customers use experiential devices (photo booths, kiosks, digital displays, etc.) installed in the store, there is a lack of technical integration to automatically provide customized services by integrally controlling in-store equipment according to individual customers' preferences or conditions.

[0006] Therefore, there is an urgent need to develop an advanced AI system that utilizes artificial intelligence technology in a store environment to automatically recognize and re-identify customers individually, analyzes their characteristics and behaviors from various angles to provide personalized responses, and further integrates with in-store equipment to offer integrated environmental control and personalized interactions. Prior art literature

[0007] 1. Korean Published Patent No. 2023-0053270 (Method and system for product payment including customer behavior recognition) 2. Korean Registered Patent No. 2540745 (Device and method for operating a store based on vision recognition) 3. Korean Registered Patent No. 2446133 (AI product recommendation system that recommends products by linking voice information and purchase history of store visitors) 4. Korean Registered Patent No. 2060694 (Customer recognition system for providing personalized services) The problem to be solved

[0008] Therefore, the main objective of the present invention is to move away from existing passive membership-based systems and to individually recognize and continuously re-identify in-store customers in real time with high accuracy using AI-based multi-recognition evidence fusion technology (Re-ID, facial recognition, BLE, etc.), and to automatically generate a sophisticated personal profile containing the customer's tastes and preferences based on the analysis results by subdividing and analyzing external characteristics such as the customer's hairstyle, clothing style, and shoe brand logo using AI.

[0009] Furthermore, the present invention aims to automatically provide a customized experience service optimized for the customer's physical characteristics by having a conversational AI perform personalized conversations and product recommendations based on analyzed customer information, while simultaneously controlling various devices in the store, such as photo booths, kiosks, lighting, and digital signage, in real time.

[0010] Furthermore, an additional task of the present invention is to provide a data processing system with enhanced security, such as by applying anonymized ID and privacy leveling technologies to minimize the risk of privacy infringement when customers process appearance information, and by immediately discarding collected image data at the edge. means of solving the problem

[0011] According to one aspect of the present invention for achieving the above-mentioned purpose, a device is installed at a store site to receive a customer's video stream, and an external feature vector (F) is obtained from the video stream. visual ) and face feature vector(F face An integrated environment control and customized interaction system based on artificial intelligence-based customer individual recognition is provided, comprising: an edge layer that extracts a vector and transmits the extracted vector to a core server; a core server layer that receives the feature vectors from the edge layer, integrates and analyzes a plurality of customer features including the appearance feature vector and face feature vector using a probabilistic fusion model to identify a customer based on an anonymous ID (anon_user_id), and calculates a customized service decision based on the anonymous ID; a device control layer that controls at least one of a shooting mode, content display, and lighting settings in real time according to the decision of the core server layer; and a data management layer that stores customer profile data and product content data linked to the anonymous ID and records all access history of the system as a security log.

[0012] Here, the edge layer comprises a segmentation network that receives the image stream as input and separates the human body region into pixel units; an attribute analysis module that predicts clothing style, color palette, and whether accessories are worn using the separated human body region as input; a logo / character recognition module that detects logos within clothing and shoe regions and classifies brand names using the separated human body region as input; and a feature integration module that generates the appearance feature vector by combining the results of the attribute analysis module and the results of the logo / character recognition module through a feature fusion unit.

[0013] In addition, the core server layer comprises a profile management module that stores and manages the customer's external characteristics, behavioral patterns, and preference tags based on the anonymized ID; a conversational AI module that generates customized response sentences according to the customer's utterances and behaviors based on the data of the profile management module; and a recommendation and control decision module that recommends services and products suitable for the customer based on the anonymized ID and transmits control commands to the device control layer.

[0014] Here, it is preferable that the recommendation and control decision module calculates the control command using an AI conversational state machine that determines the service flow according to a plurality of predefined states and transition conditions based on the customer's interaction state and events.

[0015] The above device control layer may include a photo booth device that changes the shooting mode, frame theme, and lighting settings according to the command of the recommendation / control decision module, and digital signage that displays customer-customized content and AI recommendation results in real time.

[0016] In addition, the system may be configured to further include an anonymization and encryption module that assigns a high-risk grade to original video and audio data collected from a customer, and performs at least one de-identification process among face region masking, original discarding after embedding vector extraction, and encrypted storage on the high-risk grade data.

[0017] In addition, the system may further include a data lifecycle management module that applies a policy of automatic deletion or immediate disposal within 24 hours to the high-risk grade data, and performs deletion verification based on encryption key disposal and hash check upon deletion.

[0018] It is preferable that the edge layer further receives a proximity signal through at least one of a BLE receiver, a Wi-Fi signal analyzer, or an IR sensor, and that the core server layer inputs the proximity signal along with the appearance feature vector and the face feature vector into the probabilistic fusion model to perform customer identification.

[0019] According to one aspect of the present invention for achieving the above-mentioned purpose, the method comprises the steps of: receiving a video stream of a customer from an RGB camera and a depth sensor installed at a store site; detecting and tracking a human body object within the video stream and assigning a unique track ID (track_id); and, from the image of the tracked object, obtaining an external feature vector (F visual ) and face feature vector(F face A step of extracting ) respectively and immediately discarding the original image of the face feature vector, and the shape feature vector (F visual ) and face feature vector(F faceA method for providing customized interaction through AI-based individual customer recognition is provided, comprising: a step of calculating a final recognition probability p of a customer by weighting multiple recognition grounds, including ) using a probabilistic fusion model; a step of assigning an anonymous ID (anon_user_id) to the customer and loading a customer profile associated with the ID when the final recognition probability p is greater than or equal to a reference value; a step of recommending products and services suitable for the customer and generating a command to control a device in a store in real time according to an AI conversational state machine based on the loaded profile; and a step of transmitting the generated control command to the device via the MQTT protocol and receiving the execution result of the device as feedback.

[0020] Here, the step of generating the control command preferably involves transitioning to an alternative proposal state upon the occurrence of a rejection event () for the customer's initial recommendation, analyzing the cause of the customer's rejection, and generating a next-ranked candidate product having characteristics different from the initial recommendation as a new control command. Effects of the invention

[0021] According to the present invention, the accuracy of individual customer recognition and re-identification is innovatively improved compared to existing technologies through the fusion of multiple evidence such as Re-ID, facial recognition, and BLE, and by meticulously analyzing the external characteristics of customers (fashion style, logo recognition, etc.), it is possible to provide real-time appearance-based hyper-personalized services that go beyond simple purchase history-based recommendations.

[0022] Furthermore, by having AI automatically and in real-time integrate and control all in-store devices—such as photo booths, kiosks, lighting, and displays—based on customer recognition results, it provides a customized customer experience without the need for operational personnel intervention, significantly increasing the automation and efficiency of store operations. Additionally, service quality is progressively improved through a continuous learning structure where services adaptively change in response to changes in customer appearance and behavior.

[0023] Furthermore, customers can experience product recommendations and interactions (such as conversational AI) optimized for their style and preferences simply by visiting the store without any separate registration process, thereby improving customer satisfaction and enhancing the quality of in-store experiential services, which strengthens positive perception and loyalty toward the brand and contributes to increased sales. Brief explanation of the drawing

[0024] FIG. 1 is a block diagram showing the overall configuration of an AI-based in-store customer individual recognition and customized interaction provision system according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating the configuration of an in-store customer recognition pipeline according to one embodiment of the present invention. FIG. 3 is a block diagram illustrating the configuration of a customer external feature analysis network according to one embodiment of the present invention. FIG. 4 is a block diagram illustrating a flowchart of an AI conversational state machine and an event transition according to an embodiment of the present invention. FIG. 5 is a block diagram illustrating a two-stage recommendation structure and explainability module of an AI recommendation engine according to an embodiment of the present invention. FIG. 6 is a block diagram illustrating the command transmission and feedback flow structure of an AI-based store device control system according to one embodiment of the present invention. FIG. 7 is a block diagram illustrating the privacy protection step and lifecycle management procedure of customer data according to one embodiment of the present invention. Specific details for implementing the invention

[0025] The embodiments described in the present invention and the configurations illustrated in the drawings are merely preferred embodiments of the present invention and do not represent all of the technical concept of the present invention; therefore, the scope of the rights of the present invention should not be interpreted as being limited by the embodiments and drawings described in the text. That is, since the embodiments are subject to various modifications and may take various forms, the scope of the rights of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific embodiment must include all of them or only such effects; therefore, the scope of the rights of the present invention should not be understood as being limited by them.

[0026] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning not explicitly defined in this invention.

[0027] It is the core mechanism.

[0028] FIG. 1 is a block diagram showing the overall configuration of an AI-based in-store customer individual recognition and customized interaction provision system according to one embodiment of the present invention.

[0029] The system is largely composed of an edge layer (100), a core server layer (200), a device control layer (300), and a data management layer (400), and they communicate with each other through a wired / wireless network (500).

[0030] The edge layer (100) is composed of devices installed at the store site to directly recognize customers and perform real-time inference and primary data preprocessing, and includes an image recognition module (110), a feature extraction module (120), a conversation interface module (130), and an edge controller (140).

[0031] The image recognition module (110) collects RGB images, depth information, and body shape information from a camera (111) installed in the store, and detects and tracks visiting customers through an artificial intelligence-based object detection model.

[0032] The image recognition module (110) collects input data for customer recognition and detects / tracks the location and movement of the customer in real time. It collects not only RGB images but also depth information and body shape information from the camera (111) to recognize the customer in a three-dimensional space rather than a two-dimensional plane, thereby increasing detection accuracy.

[0033] The feature extraction module (120) can infer shopping preferences and style, such as a customer's hairstyle, clothing color, and shoe brand, through external features (F). visual ) is analyzed intensively, and these features are converted into a feature vector form, and this vector is transmitted so that the AI ​​recommendation engine of the core server (200) can utilize it.

[0034] The conversation interface module (130) serves as an input / output channel that enables voice-based interaction with a customer, receives voice input from the customer through a microphone (131), and performs TTS (Text-to-Speech) based voice output through a speaker (132). If necessary, the response speed is increased by utilizing a small LLM or NLU model inside the edge device to primarily determine the customer's basic intent.

[0035] The edge controller (140) is a local manager responsible for the efficient operation and data management of all modules within the edge layer, and controls the operation of the modules (110, 120, 130).

[0036] The edge controller (140) performs data compression and security encryption before transmission to the core server, and ensures service continuity by enabling basic inference and services to continue locally without the help of the core server in the event of network delay or failure.

[0037] The core server layer (200) integrates and analyzes data collected from multiple edge devices and performs central processing such as customer identification, customized interaction control, and recommendation logic execution, and includes a customer identification engine (210), a profile management module (220), a conversational AI module (230), and a recommendation control decision module (240).

[0038] The customer identification engine (210) fuses various forms of customer feature data transmitted from the edge layer (100) to accurately and individually identify customers within the store.

[0039] The customer identification engine (210) combines complex data such as Re-ID vectors (Re-Identification, same customer re-identification vectors), facial features, and BLE proximity signals using Bayesian Fusion or a deep learning-based multimodal recognition model, and through this fusion process, lowers the misrecognition rate and accurately maps returning customers to existing anonymized IDs, thereby enabling continuous personalized services.

[0040] The profile management module (220) is a data hub that comprehensively manages the anonymized profile of a customer confirmed by the customer identification engine (210), and stores the customer's physical characteristics (current style), behavior logs (stay time, click history), past usage history, and preference patterns by linking them to an anonymized ID (anon_user_id). When a returning customer is recognized, past data is immediately loaded and provided as base data for personalized interactions and customized recommendations.

[0041] The conversational AI module (230) leads natural real-time conversation and interaction with customers and includes a conversation generation model based on an LLM (Large Language Model).

[0042] The conversational AI module (230) generates contextually appropriate customized voice responses and visual guidance sentences to be displayed on digital signage (320), etc., based on events such as customer speech, gestures, and in-store behavior. The generated final sentence is transmitted to the conversational interface module (130) at the edge layer and output as voice (TTS).

[0043] The recommendation control decision module (240) is the final decision-maker of the core server layer and calculates actual device control commands by combining the results of the AI ​​recommendation engine and store environment information. At this time, the recommendation control decision module (240) comprehensively considers the customer's profile (style, preference), current location, time zone, event information, etc., as decision criteria.

[0044] The recommendation control decision module (240), for example, when the customer's style is analyzed as "street fashion," sends a control command such as CMD_FRAME_SET to the photo booth (310) to display the relevant image and frame. This command is transmitted to the device layer through the core server unit (6100) to induce actual physical changes.

[0045] The device control layer (300) is composed of various digital devices distributed within the store, and these devices communicate with the core server (200) to provide a customer-customized experience in real time, and include a photo booth device (310), digital signage (320), a lighting and sound system (330), and a payment and interaction terminal (340).

[0046] The photo booth device (310) is a core device responsible for the customer's personalized experience (taking photos, AI image conversion). The photo booth device (310) receives a control command such as CMD_FRAME_SET from the recommendation / control decision module (240) and automatically changes the shooting mode, frame theme, and lighting settings to match the customer's external characteristics (fashion style). This means that instead of providing a uniform experience to all customers, a 1:1 customized service is realized.

[0047] Digital signage (320) is a primary display device that provides visual information and customized content to customers, and updates advertisements, product images, event information, etc. displayed on the screen in real time according to the AI's recommendation results (Top N products, event information). By showing the most relevant content based on the customer's current location and profile, it increases the purchase and experience conversion rate.

[0048] The lighting and sound system (330) is an environment control device that creates the atmosphere (emotion) of the store and induces a customer's reaction, and controls the lighting scene using professional lighting control protocols such as Art-Net or DMX. The lighting and sound system (330) changes the color, brightness, and background music of the lighting in real time according to the state of the conversational AI module (230) or changes in the customer's emotion. This demonstrates that the AI ​​maximizes the customer experience by creating emotional reactions beyond simple information provision.

[0049] The payment and interaction terminal (340) is an interface that allows a customer to directly operate (touch) it to make a final decision on purchasing a product, making a reservation, or selecting a service. It receives a payment request command generated from the AI ​​result guidance state (4700), etc., and provides the customer with functions such as purchasing a product, processing rewards, and making an experience reservation. The customer's touch or selection history is a behavior log (F behavior It is collected again and fed back to the profile management module (220).

[0050] The data management layer (400) is responsible for storing the central processing results of the core server layer (200), continuously improving the performance of the AI ​​model based on this data, and ensuring transparency by recording all activities of the system, and includes a customer profile DB (410), a product content DB (420), a model training server (430), and a security log server (440).

[0051] The customer profile DB (410) is a core information repository for personalization services, and is based on an anonymized customer identifier (anon_user_id) and appearance features (F) extracted from the edge layer. visual ), in-store usage history and behavior logs (F behavior By storing preference tags inferred by AI, etc., it enables continuous personalized service by immediately loading past data when a returning customer is recognized.

[0052] The product content DB (420) provides static content and metadata for AI recommendation logic and device control, and stores a list and attributes (metadata) of all content that the system can present to or control to customers, such as information on products sold in the store, theme frames that can be used in the photo booth, lighting presets (Scene), and event information, and is used as reference data for the AI ​​recommendation engine to generate candidate products or determine rankings.

[0053] The model training server (430) is a training center that continuously improves the intelligence of the system and periodically retrains and updates the following AI models using customer actual response (click, purchase, response) feedback and activity log data.

[0054] The recognition model improves customer identification and feature extraction accuracy, the recommendation model improves the click-through rate / conversion rate of Top N product recommendations, and the conversational model improves the ability to generate natural responses and understand intent. The model training server (430) enables the system to adapt to changes in the store environment and customer trends and to develop performance on its own.

[0055] The security log server (440) is an immutable record storage responsible for the security and audit functions of the system. It stores records of all major activities in chronological order, such as records of attempts to access personal information, records of device control commands issued by the core server to the device, and records of API calls between systems. By linking with the data utilization and disposal log (7600), it enables internal control and security audits. This information is used as evidence to clarify the transparency of system operations and the location of legal liability.

[0056] The network configuration (500) is a local communication network between the edge device (100) inside the store and the device control device (300), mainly using Wi-Fi 6E or wired LAN, and includes an internal communication network (5010) and a cloud network (502).

[0057] The internal communication network (501) is a local communication network between the edge device (100) and the device control device (300) inside the store, and mainly uses Wi-Fi 6E or wired LAN.

[0058] The cloud network (502) connects the core server (200) and the data layer (400), enables bidirectional communication based on HTTPS / gRPC / MQTT protocols, and maintains security by applying TLS 1.3 encryption.

[0059] The overview of the data flow is explained through Figure 1 as follows.

[0060] When a customer enters the store, the camera (111) generates detection data and the edge device (100) preprocesses it.

[0061] The preprocessed feature vector is transmitted to the customer identification engine (210) of the core server (200) via the network (500).

[0062] The core server identifies the customer profile, and the conversational AI module (230) generates personalized greetings and suggestions.

[0063] The recommendation and control decision module (240) generates device control commands based on store conditions and customer data and transmits them to the device control layer (300).

[0064] Each device of the device control layer (300) executes the corresponding command and stores the result and log in the data management layer (400).

[0065] The data management layer (400) uses this data to perform continuous learning and system advancement.

[0066] FIG. 2 is a block diagram illustrating the configuration of an in-store customer recognition pipeline according to one embodiment of the present invention.

[0067] FIG. 2 illustrates a process of recognizing a customer through an image recognition module (110) and a feature extraction module (120). More specifically, it includes a series of processing steps that use image data input from multiple cameras installed in a store to automatically detect a customer, track the same object in time and space, identify whether it is the same customer (Re-Identification / Face Recognition) using the customer's appearance information and face information, and finally fuse multiple recognition results to uniquely recognize an individual customer.

[0068] The video input unit (2100) may be configured to include an RGB camera (2110) and a depth sensor (2120) installed on the ceiling or entrance of a store, an experience zone, etc.

[0069] The RGB camera (2110) is a standard camera that collects general visual information such as the customer's color, appearance, and clothing style, and the depth sensor (2120) works together with the RGB camera (2110) to additionally collect depth information regarding the customer's position, movement, and body shape in three-dimensional space.

[0070] This depth information dramatically improves the accuracy of extracting customer body features and tracking motion compared to using only simple 2D images.

[0071] Each camera collects data in the form of a continuous frame-by-frame video stream, which is transmitted to the edge controller (140) in real time without delay.

[0072] The edge controller (140) receives this original image data and transmits it to the image recognition module (110) and the feature extraction module (120) so that they can process it immediately.

[0073] The video input unit (2100) can optionally be linked with other types of sensors, such as a microphone, a BLE receiver, or an IR sensor. These sensors serve to detect events where a customer approaches a specific area or emits a sound. These events are used as triggers to start the operation of the video recognition module (110) or to activate specific AI logic, thereby reducing unnecessary computational load on the system and increasing response speed.

[0074] The object detection module (2200) receives a frame-unit video stream from the video input unit (2100) and accurately identifies the area corresponding to a person within the screen.

[0075] The object detection module (2200) does not simply store image information, but limits the area to be analyzed so that the AI ​​can perform the next steps of feature extraction (120) and tracking (2300), thereby increasing computational efficiency.

[0076] The object detection module (2200) uses an artificial intelligence-based object detection network (2210) and detects people by utilizing deep learning models with excellent real-time performance and high accuracy, such as YOLOv8 DETR and Faster R-CNN.

[0077] The output of the object detection module (2200) is provided in a standardized data format so that a subsequent module can perform analysis. The location coordinates (Bounding Box) calculate the coordinates (x, y, w, h) of a rectangular area surrounding the detected human body, and the confidence score presents the probability that the area is actually a person as a numerical value (conf: 0.92), allowing inaccurate detection results to be filtered out. The classification label (Class) assigns a label indicating that the detected object is a person. Since individual customer identification (Re-ID) has not yet been performed at this stage, the initial value is null. This ID is assigned in the next stage, the tracking module (2300).

[0078] The detection results are transmitted to the subsequent tracking and feature extraction steps in the form of a JSON data structure as follows.

[0079] bbox: [x, y, w, h], "conf": 0.92, "id": nul

[0080] The object tracking module (2300) connects the frame-by-frame detection results (bounding boxes) continuously received from the object detection module (2200) to track the movement trajectory of how the same person moves within the store. The object tracking module (2300) separates multiple people within the store into individual customer sessions and tracks behavior logs (F), such as the customer's dwell time, movement path, and areas of interest. behavior Prepare basic data to extract ).

[0081] The Multi-Object Tracking (MOT) algorithm (2310) is an algorithm that uniquely identifies and tracks each object when multiple objects (people) move simultaneously within the screen.

[0082] Available algorithms may include DeepSORT, ByteTrack, OC-SORT, etc.

[0083] From the moment tracking begins, a unique track ID (track_id) is assigned to each customer and is maintained for the entire session the customer spends in the store; based on this ID, the system accumulates and records logs of all the customer's behaviors and interactions. (Subsequently mapped to an anonymous ID (anon_user_id) by the core server.)

[0084] The tracking results, including dynamic customer information over time, are transmitted to subsequent modules.

[0085] The tracking result is stored in the form {"track_id": 5, "bbox": [...], "velocity": v, "direction": θ}. Here, v is the customer's speed of movement and θ is the customer's direction of movement. This dynamic information is used to determine the customer's state (e.g., departure detection, entry into area of ​​interest) and to determine the transition condition of the interaction state (4600).

[0086] The appearance feature extraction and re-identification module (2400) extracts style features necessary for AI recommendation from the image of a customer with the track_id assigned by the object tracking module (2300), and determines that the person is the same person even with multiple cameras or over time, thereby providing grounds for mapping to an anonymized ID. The appearance feature extraction and re-identification module (2400) selects a representative image of the tracked object, discards unnecessary frames, and increases analysis efficiency.

[0087] The re-identification neural network (2410) is an AI model that has deep learned visual attributes such as clothing patterns, colors, and body shape, and enables consistent recognition that a customer is the same person even if the customer is captured by other cameras in the store, changes their posture, or is briefly obscured and then reappears. This is essential for integrating customer movement and sessions in environments where multiple cameras are installed, such as a store.

[0088] The output value is represented as an embedding vector of 512 dimensions or more. The embedding vector compresses the customer's complex style and external features into hundreds of numerical coordinates, and this vector is transmitted to the profile management module (220) of the core server F visual It is used as.

[0089] The system calculates the cosine similarity between the stored existing customer vector and the newly extracted vector, and if this similarity is greater than a predefined threshold, it determines that the two vectors represent the characteristics of the same person.

[0090] This module extracts customers' static style features through high-dimensional embedding vectors and guarantees the accuracy of personalized services by utilizing ReID technology to ensure the continuity of in-store customer tracking.

[0091] The face recognition module (2500) converts the customer's face features into unique vectors and is used to determine whether the customer is the same person as a previously recognized customer. This is done simultaneously with the full body and clothing pattern analysis of the ReID module (2400) to verify the accuracy of customer identification from various angles.

[0092] The face detector (2510) precisely detects only the face region within the image frame and extracts the optimal region to be input to the subsequent module, the embedding generator.

[0093] The face embedding generator (2520) uses deep learning-based models such as FaceNet and ArcFace to generate face feature vectors representing the unique features of an individual from detected face regions.

[0094] The face recognition result is immediately hashed to protect privacy, and the original face image, which is the most sensitive information, is discarded from edge memory as soon as the embedding vector is generated. This preemptively implements the Level C data processing principle of the data lifecycle management module (7500) from the edge stage.

[0095] If face embedding similarity ≥ τ_face, it is registered as the same customer candidate.

[0096] The face recognition module (2500) is a module that is fused with ReID to increase the accuracy of customer identification, while also having a security design applied that drastically reduces the possibility of leakage of personal identification information through the discarding of original images and hash processing.

[0097] The fusion determination module (2600) integrates all recognition grounds, such as appearance-based re-identification results, face recognition results, BLE signals (optional use), and movement path patterns, to determine whether the customer is finally identified. The purpose of this is to reduce the possibility of false positives that may occur in individual modules and to probabilistically confirm that the person is the same through multifaceted evidence.

[0098] The probabilistic fusion model (2610) is a model that combines the reliability (similarity score) of signals from different sensors and AI modules by assigning weights (w), and a Bayesian Fusion or neural network-based fusion model may be used.

[0099] The fusion determination module (2600) calculates the final recognition probability p(identity) by weighting the reliability of each signal.

[0100] Example: p = w1 × sim ReID + w2× sim Face + w3× sim BLE

[0101] The weight (w) represents the relative importance of each signal to the final recognition result. For example, if Face is more trusted than ReID, w2 can be set larger than w1.

[0102] If the calculated final recognition probability p ≥ 0.7 or higher, it is finally confirmed as the same customer, and the customer identification engine (210) transmits a newly generated or previously stored anonymous ID (anon_user_id) based on this final confirmation result to the profile management module (220). This anon_user_id is transmitted to the profile management module (220) along with the customer's appearance feature vector and behavior log, and the first step for personalized service begins.

[0103] The fusion determination module (2600) is an essential step that probabilistically integrates information from multiple sensors to raise the accuracy of customer identification to the highest level and finally determines an anonymous ID that allows the system to safely start a personalized service.

[0104] The result transmission unit (2700) plays the role of safely transmitting the final results of all customer recognition and identification processes performed at the edge layer (100) to the core server (200), which is a central control system. It packages and transmits the customer recognition results (anonymized ID, reliability) finally confirmed by the fusion determination module (2600) into a form that the core server (200) can process. This is intended to transmit the identification information and session information of individual customers generated at the edge to the central server in real time, so that the core server can immediately perform subsequent central processing tasks, such as profile loading, recommendation logic execution, and conversational AI initiation, based on this information.

[0105] The recognition results are packaged in JSON (JavaScript Object Notation) format, which is efficient for data exchange between systems.

[0106] TLS-based encryption is applied during transmission, and the data includes "session_id", "anon_user_id", "confidence", and "timestamp".

[0107] "session_id" is a unique session identifier maintained while the customer stays in the store, and "anon_user_id" is an anonymized customer ID finally confirmed by the fusion determination module (2600). Through this ID, the core server loads past history from the profile management module (220).

[0108] "Confidence" is the final recognition probability p value calculated in the fusion judgment model (2610), and the core server determines the stability of the subsequent service based on this confidence.

[0109] "Timestamp" is time information transmitted when the recognition result is finally confirmed, serving as a time / spatial context (F context It is used as part of ).

[0110] TLS (Transport Layer Security)-based encryption is applied during the transmission process. This follows the communication protocol of the cloud network (502) of the network configuration (500), ensuring the confidentiality and integrity of the data while it is being transmitted from the edge device to the core server, thereby preventing interception or tampering.

[0111] FIG. 3 is a block diagram illustrating the configuration of a customer external feature analysis network according to one embodiment of the present invention.

[0112] This network is structured to recognize a full-body or upper-body image of a customer from video input and to multi-layeredly extract the customer's external features through i) segmentation by body part, ii) classification of clothing and fashion attributes, and iii) brand logo and character recognition (OCR).

[0113] The input preprocessing unit (3100) receives the customer's bounding box image (object tracking result) and performs the role of removing noise and standardizing the data so that subsequent segmentation and style analysis can be performed accurately, and includes an image receiving module (3110) and a data normalization module (3120).

[0114] The image receiving module (3110) receives bounding box images of each customer tracked in the customer recognition pipeline and performs resolution adjustment (e.g., 512×512), background removal, and noise correction filtering.

[0115] The data normalization module (3120) converts the preprocessed image into a unified numerical range and form preferred by the deep learning model, and performs color uniformization to correct white balance or color tone different for each camera so that the AI ​​model can consistently recognize the actual color of the object regardless of changes in the lighting environment, illumination correction to improve the readability of the image information by correcting uneven lighting problems such as shadows or excessive brightness, and finally standardization to scale pixel values ​​into a certain range (e.g., between 0 and 1) and convert them into a tensor form, after which the optimized tensor-form data is transmitted to the next stage, the segmentation network (3200), and used for full-scale external feature analysis.

[0116] The human body segmentation module (3200) receives a normalized customer image from the input preprocessing unit (3100) and performs image segmentation to separate human body regions within the image into pixel units. This is intended to completely exclude interference from the background or other body parts when analyzing the color, pattern, brand, etc. of clothing, and to extract only the pixel information of the corresponding part (e.g., top).

[0117] The segmentation network (3210) uses an artificial intelligence-based deep learning model to assign a specific class (part) label to each pixel of an image, and SCHP, DeepLabV3+, HRNet, etc. are models specialized for high-resolution image segmentation and are used to accurately separate complex boundaries and shapes of clothing.

[0118] The output result is configured in the form of a mask map for each part and includes multiple detailed classes (e.g., 18 to 25) such as upper body, lower body, shoes, bags, hats, and hair. Each segment divided in this way is passed to the next step, the attribute analysis module (3300), and used as core input data for analyzing characteristics of each part. For example, the color of the shoes and the brand logo can be analyzed using only the shoe mask map.

[0119] Conditional Random Field (CRF) post-processing can be performed on boundary regions to improve segmentation accuracy. The output of a segmentation network can often contain errors in boundary regions. CRF post-processing maximizes the quality of the final mask map by sharpening and accurately correcting boundary regions by considering the relationships between pixels.

[0120] The attribute analysis module (3300) converts pixel information of clothing into meaningful style tags based on the results of the human body segmentation module (3200) and synthesizes them to generate a comprehensive fashion profile of the customer, and is configured to include an attribute extractor (3310) and an attribute fusion unit (3320).

[0121] The attribute extractor (3310) receives a region-specific image patch (e.g., an image with only the upper region cut out) separated by the segmentation module (3200) and predicts specific style attributes of the corresponding region.

[0122] Multi-head classification models based on CNNs (Convolutional Neural Networks) or Vision Transformers (ViTs) are used for attribute extraction. Multi-head models enhance efficiency by simultaneously predicting multiple independent attributes (color, style, material, etc.) from a single input (image patch). The key predicted attributes are as follows.

[0123] o Hair Attributes: Identifying the customer's aesthetic characteristics by analyzing hair length, color, shape, etc.

[0124] o Clothing Color: Extract color palette for tops / bottoms

[0125] o Clothing Style: Classification of abstract fashion style tags defined by AI, such as casual, formal, street, and sports.

[0126] o Accessories: Determine whether items such as hats, bags, and glasses are being worn

[0127] Each attribute is calculated as a probability vector and stored in JSON format such as "style_tags", "color_palette", and "accessory_flag".

[0128] The attribute fusion unit (3320) collects the attribute results by part predicted by the attribute extractor (3310) and generates a comprehensive fashion profile (feature profile) that defines the customer's overall fashion concept. The attribute fusion unit (3320) goes beyond simple listing of attributes and analyzes the relationships between attributes to infer a new style.

[0129] For example, if the color tones of the top and shoes are determined to be very similar, they are automatically classified into a high-level concept called "tone-on-tone style."

[0130] A comprehensive fashion profile (F) intricately fused in this way visual ) is transmitted to the core server (200) and becomes a key feature vector used by the AI ​​recommendation engine to recommend products that are visually similar to or match the customer's current style.

[0131] The logo / character recognition module (3400) is a precision analysis step in the cognitive pipeline of the AI ​​recommendation engine that identifies brand preference from the customer's appearance, and plays the role of inferring 'what brand the customer consumes' by interpreting visual features (logos, characters) appearing on the clothing or shoes worn by the customer, and includes a logo detector (3410), an OCR recognizer (3420), and a logo reliability evaluator (3430).

[0132] The logo detector (3410) is responsible for detecting small areas where a brand logo is likely to be found in areas such as upper body, lower body, and shoes separated as a result of human body segmentation (3200).

[0133] We use deep learning-based object detection models such as YOLOv8 and RetinaNet. These models are specialized in quickly and accurately detecting small objects (logos) within images, and the input data is an image region limited by a segment mask (e.g., a shoe patch image).

[0134] The OCR recognizer (3420) receives the candidate region found by the logo detector (3410) and actually reads and interprets the text (characters) included in the logo.

[0135] Text in an image is converted into digital text using an OCR (Optical Character Recognition) model, such as a CNN-RNN-based character recognition model (CRNN, TrOCR, etc.), and the recognized text is matched with a brand DB stored in a product / content DB (420). For example, when the text 'NIKE' is recognized, it is ultimately classified as the brand name "Nike".

[0136] The logo reliability evaluator (3430) filters out errors (False Positives) or low-quality results that may occur during the logo detection and OCR recognition process, thereby ensuring the reliability of the final brand judgment.

[0137] At this point, the detection reliability (the probability that the logo actually exists), the OCR match rate (how well the recognized text matches the brand name in the database), and the positional stability (whether the logo is in a typical location on the corresponding clothing / shoe) are comprehensively evaluated to calculate the final reliability score.

[0138] Example: "Shoe logo recognition result: Only the top candidates with high probability, such as 'Nike(0.81), Puma(0.12)', are transmitted to the core server (200)."

[0139] This module contributes to enhancing the level of recommendation personalization by identifying customers' clear brand preferences through the combination of logo detection (location) and OCR recognition (character) and the refinement of results via reliability evaluation.

[0140] The feature integration module (3500) is a final aggregation step in which the customer's visual feature analysis is completed within the cognitive pipeline of the AI ​​recommendation engine, and the distributed analysis results are combined into a single unified feature vector and transmitted to the central server (200), wherein the customer's external features (F) visual It is the core process of creating the most comprehensive data package that defines ).

[0141] The feature integration module (3500) includes a feature fusion unit (3510) and a normalization and storage module (3520).

[0142] The feature fusion unit (3510) combines data received from the segmentation module (3200), the attribute analysis module (3300), and the logo / character recognition module (3400) to generate a final customer appearance feature vector as follows. The final customer appearance feature vector is generated by connecting each feature vector.

[0143]

[0144] Here means vector concatenation, and F seg is human body segmentation information, F attr is the style attribute analysis result, F logo is the result of brand logo / character recognition.

[0145] F connected in this way total It is a multidimensional vector containing all information about the customer's appearance (pixel location, style tag, brand preference) and serves as a key input feature that the core server's multimodal embedding network uses to understand the customer and recommend products.

[0146] The normalization and storage module (3520) normalizes the generated feature vector and stores it in the customer profile DB (410).

[0147] Each attribute is linked to an anonymous ID (anon_user_id) and utilized in the customer re-identification and recommendation stages (Figure 4 onwards). This vector is used not only to re-identify returning customers but also as a key criterion for real-time recommendations that reflect the customer's current style during the recommendation stage.

[0148] The feature integration module (3500) acts as a bridge to integrate distributed visual information into a single strong appearance feature vector through a feature fusion unit (3510), and provides it as core input data for an AI recommendation engine after normalization and encryption.

[0149] FIG. 4 is a block diagram illustrating the flow of an AI conversational state machine and event transitions according to an embodiment of the present invention.

[0150] (1) Initial state (4100)

[0151] The in-store AI device is in a default standby state and remains in detection mode until a customer approaches or is recognized within its line of sight.

[0152] When trigger events such as a customer detection event (E1: PersonDetected), a voice recognition event (E2: VoiceWake), or a BLE proximity signal (E3: BLEDetected) occur, the system transitions to a personnel state (4200).

[0153] (2) Personnel status (4200)

[0154] The greeting state is the stage of recognizing the customer's presence and attempting the first interaction.

[0155] The AI ​​determines whether the customer will revisit and generates different greeting messages as in the following example.

[0156] - Example 1: "It's your first time here! What would you like to try?"

[0157] - Example 2: "You're back. Did you enjoy the shoot last time?"

[0158] In the greeting state, output actions can consist of text-to-speech (TTS), display animations, lighting responses, etc.

[0159] The transition conditions that can occur in a personnel state can be represented by the following two types.

[0160] First, when a customer responds positively to a greeting or asks a question, the system continues the conversation and the profile enters a state (4300) to provide customer-tailored information (profile-based) (E4).

[0161] When a customer does not respond to the system's greeting and passes by or does not stop, the system returns to the initial waiting state (4100) to prevent unnecessary conversation from continuing and to avoid burdening the customer, and waits for the next customer or new action (E5).

[0162] (3) Profiling status (4300)

[0163] The profiling stage is the step of constructing a real-time personalized profile (based on anonymous ID) by analyzing the customer's appearance, facial expressions, clothing style, shoe brand, etc.

[0164] Conversational AI generates customized questions or suggestion phrases based on customer characteristics.

[0165] Example: "Your outfit looks really great today! Would you like to take a picture with a street frame?"

[0166] By utilizing customer visual characteristics (fashion, logo, style) and profile information extracted from the edge layer and finally analyzed in the core layer, products or in-store experiential services (frames / backgrounds) deemed most relevant to the customer are proposed.

[0167] In this state, when the customer shows one of three possible responses to the system's suggestion, the conversation state transitions to the next state.

[0168] When a customer positively accepts a recommendation or suggestion from the system, the system immediately switches to a state (4400) that provides specific guidance to lead to an actual experience or purchase (E6).

[0169] If the customer rejects or ignores the current recommendation, the system does not give up on the interaction and attempts to regain the customer's interest by switching to a state (4500) to present a different style or alternative product (second best option) (E7).

[0170] When a customer moves to another place in the store or remains stationary for a certain period of time without any response (accept / reject / talk) after receiving a proposal from the system, the system returns to an initial waiting state (4100) to avoid coercion and wait for the next event (E8).

[0171] The profiling status (4300) is designed to branch into three paths based on the customer's response: successful guidance (E6), retry with alternative presentation (E7), or abandonment of interaction and waiting (E8), thereby enabling a customer-centric, flexible service.

[0172] (4) Experience Guide Status (4400)

[0173] If a customer selects a specific experience (photo booth, tarot, rhythm game, etc.), the AI ​​provides information on the location of the device, how to use it, and the waiting order.

[0174] It provides specific information to help with actual use of the experience service that the customer is recommended to or chooses themselves in the profiling state (4300). This step serves to maximize customer convenience and prevent service abandonment.

[0175] In this state, in addition to providing guidance to the customer, the system generates commands to control in-store devices, thereby inducing changes in the physical environment.

[0176] When a customer selects a photo booth, the core server's recommendation / control decision module sends a start command (CMD_PhotoStart) to the corresponding photo booth device, causing the photo booth to switch to a customer-customized mode (e.g., backgrounds / filters tailored to the customer's style) or become immediately available for use.

[0177] Guide the customer by displaying a guide video or map of the device on the nearest digital signage (320) according to the customer's current location or movement path.

[0178] The experience guide state (4400) transitions to the next state by two events.

[0179] When the customer receives the guidance and actually starts using the device (e.g., when entering a photo booth or starting to operate a kiosk), the system switches to a state (4600) that manages interactions occurring during the experience (E9).

[0180] This occurs when a customer receives guidance but changes their mind, cancels the experience, and moves to another location within the store, and the system stops the interaction and returns to the initial waiting state (4100) to wait for a new event to occur (E10).

[0181] The experience guidance state (4400) serves as an important bridge to convert the customer's intention into actual action through guidance and control, and actively controlling store equipment such as photo booths during this process is a core function of the present invention.

[0182] (5) Alternative proposal status (4500)

[0183] The alternative suggestion state (4500) is a key interaction step in which the AI ​​finds and presents an alternative when there is no product that directly matches the customer's request or preference, or when the customer shows a negative reaction to the initial recommendation.

[0184] When the recommendation and control decision (4400) is rejected by the customer (event E8), or when a new product or experience option is explored and presented in response to the customer's question (event E7),

[0185] The conversational AI module (230) and the AI ​​recommendation engine analyze the cause of failure of the initial recommendation (color, price range, style, etc.) and search for a next-ranked candidate group with characteristics different from the existing ones.

[0186] As a method of recommending similar products, you can make suggestions such as, "How about another product with the same material but a different color from the item you saw earlier?" or suggest changing the style / theme, such as, "Instead of the street style, would you like me to change it to a 'casual' theme frame that matches today's outfit?"

[0187] And by adjusting the price range, you can make suggestions such as, "This doesn't seem to be the price range you were looking for. I will show you similar products at a more reasonable price."

[0188] (6) Conversation and interaction status (4600)

[0189] This state is activated when the customer enters a core process of the service, such as actually entering a photo booth, operating a payment terminal, or creating a character through voice conversation, after passing through the experience guidance state (4400). The system fully supports this process to ensure it is completed smoothly.

[0190] In this stage, the role of the conversation interface module (130) is maximized to detect and analyze the customer's various and complex input methods (Multi-modal Input) in real time, so that the AI ​​can immediately provide appropriate feedback suitable for the situation.

[0191] The recommendation / control decision module (240) effectively controls the store equipment (300) in combination with the customer's external characteristics. For example, if the customer is using a photo booth, the AI ​​leads the progress of the service by controlling the lighting and sound system (330) to induce photography, and if the customer is in the product selection stage, it induces payment through the payment and interaction terminal (340).

[0192] As an example of AI action in this step, the customer's behavior at a specific time can be induced through a voice prompt such as "Please look at the camera" to support device operation (shooting), or additional information about the customer's preferences can be collected through a prompt such as "What style do you want?" to further customize the service (e.g., character costumes, filters).

[0193] The interaction state (4600) is terminated by two major events.

[0194] The first transition condition occurs when the customer successfully completes the current experience activity, such as taking a photo or creating a character, and the system moves to the next state (4700) to provide the final result of the experience and provide a final guide (E11).

[0195] The second transition condition occurs when normal service cannot proceed due to device malfunction, network problems, or repeated incorrect input by the customer, and the system switches to a recovery state (4800) to resolve the problem or restart the service (E12).

[0196] The interaction state (4600) is a stage that determines the quality of the personalized experience, and aims for a smooth and satisfying service completion (E11) by having the AI ​​communicate with the customer in real time and control the store equipment.

[0197] (6) Result guidance status (4700)

[0198] This state is a stage in which the results of the experience are provided to the customer visually and audibly, and the final result of the experience (photo shooting, character creation, etc.) completed in the interaction state (4600) is shown and explained to the customer through digital signage (320) or a payment terminal (340), etc.

[0199] AI induces additional suggestions or follow-up actions based on output results (photos, videos, tarot results, etc.).

[0200] At this stage, beyond simply displaying results, the aim is to encourage the utilization of the deliverables or related actions, which is intended to drive subsequent consumption (printing, sharing, purchasing related products) at moments of high customer satisfaction.

[0201] Example: "Would you like to print this result?", "Would you like to share this frame on social media?"

[0202] In this state, the system provides options to determine the customer's final action and outputs the necessary information.

[0203] The system displays the experience results (photos, videos, etc.) on the screen so that the customer can finally check them through a result preview, provides payment options to physically own the results (e.g., photo prints) or purchase them as digital products, and recommends additional products related to the results (e.g., goods that go well with photo frames).

[0204] The result guidance status (4700) transitions to two states depending on the customer's final decision or no response.

[0205] The first transition condition occurs when the customer accepts or confirms a proposed subsequent action (print, share, purchase additional products, etc.), and the system transitions to a subsequent recommendation state (4900) for additional commercial opportunities after the experience (E13).

[0206] The second transition condition occurs when the customer leaves the seat without any follow-up action (confirmation, rejection, etc.) for a certain period of time after checking the result, and the system terminates the session and returns to the initial waiting state (4100) to prepare to welcome the next customer (E14).

[0207] The result guidance state (4700) is a stage where the final result of the service is delivered and opportunities for additional value creation (subsequent recommendation) are sought, and when the customer makes a final decision, it proceeds to the next stage.

[0208] (7) Follow-up recommendation status (4900)

[0209] In this stage, customized products or content are suggested by analyzing the customer's usage history and current experience data.

[0210] In this state, the experience data (photo style, selected options, etc.) that the customer just completed and the past usage history stored in the customer profile DB (410) are comprehensively analyzed, and based on the analysis results, products or other experience areas within the store that are expected to have the highest purchase / usage conversion rate are suggested to induce the customer to repurchase or extend their time in the store.

[0211] For example, by using a comment such as, "I will prepare a keyring that matches this style," you can recommend actual products related to the theme or character the customer just selected in the photo booth to induce a purchase, or by using a comment such as, "Tarot reading is popular in the next experience zone," you can suggest a different type of experience (content) since the customer has finished one experience to encourage them to continue their activities in the store.

[0212] As an output action, recommendations are primarily delivered via voice conversation along with visual images through digital signage (320) or a payment terminal (340), and this suggestion is also linked to the customer's physical characteristics.

[0213] The subsequent recommendation status (4900) is clearly divided into two states depending on whether the customer is interested.

[0214] The first transition condition occurs when the customer shows a positive response (voice, gesture, touch) to the AI's subsequent recommendation (product, other experience), and the system returns to the experience guidance state (4400) to connect the customer's interest to actual action and provides specific location guidance or usage instructions (E15).

[0215] The second transition condition occurs when a customer does not respond to the recommendation and moves toward the store exit or a departure is detected, and the system recognizes that the attempt at interaction was unsuccessful and transitions to a session end state (4950) to finish the current customer session (E16).

[0216] The subsequent recommendation status (4900) is a step that performs the commercial purpose of the system, focusing on preventing customer churn after the experience is completed and securing re-engagement (E15) or purchase opportunities.

[0217] (8) Session exit status (4950)

[0218] This state is entered when customer churn is detected in the subsequent recommendation state (4900), or when the need for interaction disappears because the customer no longer responds to the system's suggestions. This signifies the point at which active service provision to individual customers is completed.

[0219] Output Action

[0220] As an output action, a farewell message is output via digital signage (320) or a voice system (330) to provide a positive final experience to the customer and leave a good impression of the brand.

[0221] Session logs are stored in the data layer (400), and ID mapping information is preserved so that it can be referenced when the customer revisits.

[0222] All records of current interactions with customers (what products were viewed, what experiences were had, conversation content with AI, etc.) are transmitted in the form of session logs to the model training server (430) or security log server (440) of the data management layer (400).

[0223] This log serves as valuable material for continuous learning, and the anonymous ID (anon_user_id) and related information are stored in the customer profile DB (410) to provide a basis for immediately identifying and resuming customized services when the customer revisits.

[0224] Afterwards, the system returns to a standby state (4100).

[0225] Once all termination procedures are completed, the system completely disconnects from the current customer and returns to the initial waiting state (4100). This is a state in which the system is prepared to efficiently wait for new events (entry of a new customer, action of the next customer, etc.).

[0226] The session termination state (4950) is responsible for the soft landing of the interaction and is the final management step of returning to the initial state for the operational efficiency of the system, while simultaneously completing a feedback loop to improve the quality of the next service by permanently recording data.

[0227] (9) Recovery status (4800)

[0228] This state is activated when an unexpected technical problem (E12: ErrorDetected) occurs during system operation, such as in an interaction state (4600). Exception situations include all problems that impede service continuity, such as hardware errors (camera, sensor), software errors (speech synthesis failure), and communication errors (network delay).

[0229] As an output action, the system reduces customer confusion by providing a standardized message that simply explains the situation and encourages retries, instead of complex technical error messages when an error occurs, and requests that the customer wait briefly while the system is restored.

[0230] This message can be output through digital signage (320) or a sound system (330).

[0231] While entering this state, the system performs internal routines in the background to diagnose and resolve errors (e.g., attempting network reconnection, restarting the error-causing module, clearing the input buffer, etc.).

[0232] The recovery state transitions to a single state depending on whether the system succeeded in resolving the problem.

[0233] This transition condition occurs when the system succeeds in resolving the technical problem, and once recovery is complete, the system returns to the state immediately prior to the error, allowing the service to resume from where it left off (e.g., if an error occurred in the interaction state (4600), it returns to the interaction state (4600) after recovery).

[0234] FIG. 5 is a block diagram illustrating a two-stage recommendation structure and explainability module of an AI recommendation engine according to an embodiment of the present invention.

[0235] This system receives customer appearance, behavior, and preference data as input, quickly selects potential candidate items in the first stage, and performs final recommendations through a reordering process that considers multiple factors in the second stage.

[0236] In addition, it enhances customer trust by providing explainability of recommendation results.

[0237] The input data collection unit (5100) collects the customer's current session and past usage data.

[0238] Current session data may include (actions the customer is currently performing within the store (real-time location, gestures, conversation content, etc.)).

[0239] Past usage data may include previous visit records, purchase history, preferred styles, etc., stored in the customer profile DB (410).

[0240] By integrating these two types of data, a comprehensive recommendation foundation can be established that reflects both the latest information and long-term preferences.

[0241] The input data collection unit (5100) collects multifaceted feature data, such as customer appearance features, behavior logs, and time / space context, to increase the accuracy of the recommendation engine.

[0242] Customer visual features (F_visual) are data analyzed by the feature extraction module (120) and reflect the latest appearance-based preferences, such as the customer's clothing style, color, logo / brand, etc.

[0243] The behavior log (F_behavior) is data collected from the edge layer (100) and the core layer (200), representing active interests within the store, such as customer dwell time and screen click / selection history.

[0244] The temporal / spatial context (F_context) consists of the system's own logs and sensor data, which corrects recommendations by reflecting external environmental factors such as the time and day of the week of the customer's visit and the event area within the store where they are currently located.

[0245] The data is processed based on an anonymous ID (anon_user_id) and, after preprocessing, is passed to the candidate generation module (5200).

[0246] To protect privacy, all data is mapped to anonymized IDs for processing, thus maintaining security by separating it from personally identifiable information.

[0247] The collected raw data undergoes preprocessing steps such as normalization and encoding so that the recommendation model can learn and infer efficiently, and is then immediately transferred to the candidate generation module (5200), which is the first stage of the two-stage recommendation pipeline, to start the operation of the recommendation algorithm.

[0248] The candidate generation module (5200) extracts the top K candidates from thousands to tens of thousands of total items based on input data and metadata of the product / content DB (420).

[0249] The purpose of candidate generation is to avoid performing complex calculations for all products and to drastically reduce the load of the subsequent reordering (5300) module without lowering recommendation accuracy.

[0250] The input data includes the customer's appearance, behavior, and contextual features (F) received from the input data collection unit (5100). visual , F behavior , F context Use metadata (text descriptions, images, etc. of products) from the product content DB (420) and output a list of K (hundreds) candidate products {i1, i2, … , iK} to be presented to the end user.

[0251] The input data collection unit (5100) includes a vector embedding generator (5110), an approximate nearest neighbor searcher (5120), and a candidate filter (5130).

[0252] The vector embedding generator (5110) converts customers and products into a digital coordinate system called the same common embedding space, so that items located in similar positions in this space can be determined to be close to each other.

[0253] The customer vector (u) is the customer's appearance (F visual ), behavior(F behavior ), contextual information(F context Combine )(u=f(F visual ,F behavior ,F context)) to express the customer's overall preference and current state as a single vector.

[0254] The item vector (i) represents the product's features as a high-dimensional vector using a multimodal encoder such as CLIP, BERT, or ViT that integrally encodes the product's image (visual) and description (text).

[0255] The approximate nearest searcher (5120) searches for item vectors at ultra-high speeds for the location (similar product) closest to the customer vector (u) just created in the large item vector database.

[0256] The approximate nearest neighbor searcher (5120) uses an Approximate Nearest Neighbor (ANN) library such as FAISS and ScaNN to search millions of items within milliseconds (ms) with speed as the top priority instead of perfect accuracy.

[0257] Based on similarity, the distance in vector space is measured using the angle (cosine similarity) or dot product between vectors, and a similarity score s is calculated. Then, the top K similar items and their similarity scores {s1, s2, … , sK} are output to the customer.

[0258] The candidate filter (5130) refines the final candidate group by removing items that are unnecessary or cannot be provided to the customer from the candidate list output by the ANN explorer. The candidate filter (5230) removes items that have already been purchased, out-of-stock items that are out of stock in the store, non-exposed items previously shown to the customer, or specific duplicate frames, and the refined final candidate list is passed to the reordering module (5300) to undergo a more sophisticated ranking process.

[0259] The reordering module (5200) receives K candidates from the candidate generation stage as input, recalculates customer-customized scores to derive a final ranking, and is configured to include a feature integrator (5210), a ranking model (5220), and a multi-objective optimizer (5230).

[0260] The feature integrator (5210) collects multidimensional features for each candidate item passed from the candidate generation module so that a precise ranking model (5320) can utilize them, and combines them into a single integrated feature vector.

[0261] While the candidate generation phase primarily used simple similarity (Sim_visual) for speed, this section adds complex features that reflect business objectives and real-time conditions.

[0262] The integration features may include the following features.

[0263] - Sim_visual: Visual similarity score between the customer's appearance style and the product, reflecting personalized preferences (accuracy)

[0264] - Match_brand: Match / association between the brand currently worn by the customer and the product brand, reflecting brand preference

[0265] - Recency: Weighting based on product newness or recent updates, driving exposure to trends and new products.

[0266] - Popularity: Preference for the product among all users; ensuring stability by recommending verified products.

[0267] - Margin or Inventory: In-store inventory quantity and the sales margin weighting of the corresponding products, reflecting business objectives (inventory management, profit maximization).

[0268] The ranking model (5220) receives the integrated feature vector generated by the feature integrator (5310) as input and predicts the final score (S) that predicts the probability that each candidate item will ultimately be selected by the customer. corefinalCalculate )

[0269] The ranking model (5220) uses a List-wise Learning-to-Rank (LTR) model to learn the order of multiple candidate items at once, providing a more accurate final ranking than the individual point-wise model.

[0270] For example, XGBoost LTR, LambdaMART, or a Transformer-based RankFormer that learns order information are used to sort precisely, and candidate products are sorted in descending order according to the predicted final scores to determine the top N items as the final recommendation list to present to the customer.

[0271] The multi-purpose optimizer (5230) derives optimal results by simultaneously considering various business objectives in addition to simple accuracy (clicks / purchases) during the learning process of the ranking model or the final ranking determination stage.

[0272] As an objective function (L), various objectives are weighted (w) n It uses a complex loss function combined with ).

[0273] L = w_1·CTR + w_2·CVR + w_3·StockBalance - w_4·DiversityPenalty

[0274] Here, w_1·CTR (click-through rate) and w_2·CVR (conversion rate) increase the accuracy and validity of the recommendation.

[0275] w_3·StockBalance assigns weights to products with high inventory levels to encourage the achievement of the business goal of inventory management, and w_4·DiversityPenalty imposes a ranking penalty to ensure diversity and prevent the repeated recommendation of only similar products.

[0276] Through this module, the recommendation system can provide an optimal recommendation list that simultaneously satisfies the complex goals of user satisfaction, profitability, and store operational efficiency.

[0277] The explainability module (5300) is configured to include a reason extractor (5310), an explanation generator (5320), and an interface conveyor (5330) to transparently disclose the recommendation process of the AI, which is a complex black box model, to the user to provide rationality to the recommendation and to increase the likelihood that the customer will trust and accept the recommendation, by generating the selection basis for each recommended item and explaining it to the user in natural language form.

[0278] The evidence extractor (5310) analyzes the input features that have the most decisive influence on the ranking model (5220) calculating the final score and quantifies their contribution. As an analysis method, XAI (eXplainable AI) techniques such as Shapley Value (fairly calculating the degree to which each feature contributes to the prediction), Attention Weight (the degree to which the model focuses on a specific feature), and Feature Importance (the importance of the corresponding feature in the entire model) may be used.

[0279] The results can be passed to the next step by quantifying the contribution of each feature, such as "Color Match 40%, Brand Match 30%, Trend Score 20%".

[0280] The explanation generator (5320) converts the technical analysis results from the evidence extractor (5310) into natural language explanation phrases that customers can easily understand, by inserting the analyzed evidence into a predefined sentence template or by using an LLM-based language model to generate a flexible and friendly explanation suitable for the situation.

[0281] The results may, for example, provide a statement such as "It is a neon-colored item in the same tone as today's outfit" by utilizing color and style matching grounds, or provide a statement such as "It is a street style product that has been frequently selected recently" by utilizing behavior logs and popularity / trend scores grounds.

[0282] The interface conveyor (5330) selects an appropriate output channel to deliver the finally generated description to the customer and transmits the message.

[0283] The transmission target may be a conversational AI module and digital signage (320).

[0284] The conversational AI module (230) converts the explanatory text into speech (TTS) and integrates it into a conversation with the customer, and the digital signage (320) displays the explanation in the form of text or icons next to the recommended products.

[0285] This ensures that explanations are delivered clearly and effectively to customers by utilizing both voice (dialogue) and visuals (screen).

[0286] The result output unit (5400) integrates the top N recommended product list confirmed by the reordering module (5300) and the selection basis (description) for each recommendation generated by the explainability module (5400) into a single package and outputs it.

[0287] This result is transmitted to the conversational AI module (230) or recommendation / control decision module (240) of the core server layer (200), and is displayed on a device (digital signage, etc.) of the device control layer (300) or guided by voice.

[0288] AI tracks customer responses (clicks, responses, gaze) and utilizes them to improve the accuracy of recommendation results and for online learning.

[0289] After the recommendation result is presented to the customer, data regarding which item the customer clicked, whether they responded with voice, and even how long they kept their gaze on it is tracked to be transmitted to the continuous learning module (5100).

[0290] This data is transmitted to an AI model training server (430) and used as the correct answer (label) for 'what the customer chose'. Through this, the recommendation model learns the customer's actual preferences and is used to improve the accuracy of recommendations in the next session (online training or offline retraining).

[0291] FIG. 6 is a block diagram illustrating the command transmission and feedback flow structure of an AI-based store device control system according to one embodiment of the present invention.

[0292] This system is based on customer recognition and recommendation results on the core server.

[0293] It has a structure that transmits commands to multiple IoT devices (Device Layer) within the store and collects the status of each device in real time to perform bidirectional control, and includes a core server unit (6100), an MQTT broker (6200), and a device layer (6300).

[0294] The core server section (6100) serves as a central control hub for all digital devices in the store and transmits the final command of the core server layer (200) to the device control layer (300) and checks the status thereof, and includes a command generator (6110), a topic router (6120), an RPC interface (6130), and a feedback listener (6140).

[0295] The command generator (6110) converts the service goal (e.g., recommending product A to a customer) received from the conversational AI module (230) and the recommendation / control decision module (240) into specific control commands that each device can understand.

[0296] The command generator (6110) gathers all situational information, such as the customer's status (e.g., waiting for a photo booth), recommendation results (e.g., neon style frames), and conversation events (e.g., the customer responding "yes"), and generates the optimal command.

[0297] For example, the command is as follows.

[0298] - CMD_LIGHT_ON: Lighting system (330) control command

[0299] - CMD_FRAME_SET:street_neon_v3: command requesting a specific frame setting from the photo booth device (310)

[0300] - CMD_AUDIO_PLAY:intro_sound: command requesting the sound system (330) to play a specific sound source

[0301] The topic router (6120) is a message transmission manager that transmits generated control commands to each device in real time and efficiently.

[0302] The topic router (6120) uses a lightweight communication protocol such as MQTT (Message Queuing Telemetry Transport) to publish command messages to topics that are unique identifiers for each device. Devices subscribe to the topics assigned to them and execute commands immediately upon arrival, thereby providing loose coupling between the devices and the server, which can increase the scalability and stability of the system.

[0303] The RPC interface (6130) handles communication with devices requiring high-performance, high-volume data exchange that are difficult to process all commands with MQTT (e.g., Photo Booth), and uses a high-speed communication method such as gRPC (Google Remote Procedure Call) to quickly and reliably transmit large amounts of data (e.g., high-definition photo files, complex 3D rendering commands).

[0304] The feedback listener (6140) is a state monitoring manager that not only issues commands but also traces back the state changes of the devices after the commands are executed and transmits them to the core server, and subscribes to and collects status messages (telemetry) sent by each device via MQTT or other channels.

[0305] The items collected by the feedback listener (6140) include the temperature of the device, network connection status, and the results of the success / failure of processing previous commands, and these are identified in real time to help determine whether to enter a recovery state (4800) or to ensure service continuity.

[0306] The MQTT broker (6200) receives commands posted by the topic router (6120) of the core server and transmits them to the device, and performs a relay function by receiving feedback (Telemetry) sent by the device and transmitting it to the feedback listener (6140).

[0307] The MQTT broker (6200) acts as a routing hub for command messages and relays communication between the core server and the device. MQTT is a lightweight messaging protocol optimized for low-bandwidth, unstable network environments and is suitable for efficiently managing and controlling numerous IoT devices in a store in real time.

[0308] The MQTT broker (6200) includes, in detail, a post queue (6210), a topic mapper (6220), a delivery engine (6230), and a feedback collector (6240).

[0309] The post queue (6210) stores control command messages published by the core server's command generator (6110) through the topic router (6120) in a temporary queue, thereby distributing the load that occurs when the core server's rate of issuing commands differs from the device's rate of processing commands, and safely storing commands even when the device is temporarily offline to prevent loss of commands.

[0310] The topic mapper (6220) manages and maps the address scheme (Topic) of the message to deliver the message to the correct receiving device.

[0311] The address structure is store / <storeid> / device / <deviceid>It uses the / cmd structure. This structure explicitly includes the store ID and device ID, enabling the unique identification of numerous stores and their devices, and the accurate routing of commands.

[0312] The delivery engine (6230) is an engine that effectively delivers commands stored in a queue to the end receiving device and transmits messages by guaranteeing the Quality of Service (QoS) level (1 or 2), which is a core function of MQTT.

[0313] QoS 1 (At Least Once) guarantees that a message is delivered at least once, and QoS 2 (Exactly Once) guarantees that a message is delivered exactly once, thereby preventing duplicate execution of important control messages such as payment commands and maximizing reliability.

[0314] The feedback collector (6240) collects and relays status messages (Telemetry) transmitted from each device to the core server in the opposite direction of command transmission.

[0315] The feedback collector (6240) safely transmits messages of changes in the status of the received device (e.g., error, out of stock, completion of command processing) to the feedback listener (6140) of the core server, thereby enabling the core server to monitor the store environment in real time and make decisions such as entering a recovery state (4800).

[0316] The device layer (6300) consists of multiple IoT devices distributed within the store. Each device has a unique Device ID and Topic, and communicates with the MQTT broker (6200) to receive central control commands.

[0317] The photo booth device (6310) is a core device that provides a personalized experience for customers, receives a start / stop command for the experience from the core server, and controls the camera, lighting, AI conversion module, etc. mounted inside.

[0318] It enables shooting with customized frames based on the customer's physical characteristics through commands such as "CMD_CAPTURE_START" and "CMD_PRINT_RESULT".

[0319] Digital signage (6320) is a display device that provides visual information and guidance to customers. It displays video or image content in real time according to the command of the core server, and when a customer approaches, it broadcasts a customized welcome message, product recommendation image, guidance video, etc., that matches the customer's profile.

[0320] As an example of the command, you can play a promotional video for the Tarot experience via "CMD_DISPLAY_CONTENT:tarot_promo.mp4".

[0321] The lighting / sound controller (6330) controls the atmosphere and emotional elements of the entire store or a specific area, controls the lighting scene using a professional protocol such as DMX or Art-Net, and changes the lighting color, brightness, music genre or volume, etc. in real time according to the customer's appearance or current conversation state to maximize the level of the customer's sensory experience.

[0322] The payment and touch terminal (6340) is an interface that ultimately processes purchases and interactions, receiving payment request commands for paid options (print, SNS sharing) of the photo booth experience results, or receiving direct touch input (selection, rejection) from the customer and transmitting it to the core server. This plays an important role in the conversation and interaction state (4600) and the result guidance state (4700).

[0323] You can request payment for Photo Booth A via the example command "CMD_PAYMENT_REQUEST:booth_A".

[0324] FIG. 6 illustrates the command transmission and feedback flow of an AI-based store device control system according to one embodiment of the present invention as follows.

[0325] 1. Command generation step (S1)

[0326] The command generator (6110) of the core server (6100) receives a customer event and generates a control command suitable for the device.

[0327] 2. Command Posting Step (S2)

[0328] The generated command is published to a specific topic of the MQTT broker (6200) through the topic router (6120).

[0329] 3. Command reception and execution step (S3)

[0330] Each device (6310–6340) receives a command from the topic it subscribes to and performs the corresponding action.

[0331] The execution result (success / fail, processing time, etc.) is published to the broker as a Telemetry message.

[0332] 4. Feedback Collection Step (S4)

[0333] The MQTT broker (6200) transmits a feedback message to the core server (6140) through the Telemetry Collector (6240).

[0334] 5. Status Update Step (S5)

[0335] The core server updates the status of each device based on received feedback and sends redo or recovery commands if necessary.

[0336] FIG. 7 is a block diagram illustrating the privacy protection step and lifecycle management procedure of customer data according to one embodiment of the present invention.

[0337] This system classifies various data collected from customers, such as images, voice, and behavioral logs, according to privacy levels, and applies differentiated retention periods, processing methods, and deletion policies for each level.

[0338] The first step of FIG. 7 is the data input step, which collects original image, voice, and sensor data generated during the customer recognition process (7100). This is to collect all raw data generated during the interaction with the customer from the camera (111) and sensor located at the edge layer (100) of the system, and to use it as key input data to be used for high-precision individual customer recognition and feature extraction.

[0339] At this stage, the following data is collected that can directly identify the customer's identity or sensitive behavior.

[0340] Face / body images (D image ) : Core visual data including customer identity (face) and external features (body), such as RGB images, depth information, and body shape information

[0341] Voice signal (D voice ) : Raw voice files generated from conversations between customers and AI, where voice features can be classified as biometric information

[0342] Proximity signal (D signal ) : Sensor signal data, such as BLE and UWB, used to determine a customer's precise location or movement path

[0343] Interaction Event Log (D event ) : Logs regarding the time when a customer touches the screen or takes a specific action

[0344] Since the data still has the potential for personal identification at the time of collection, it is classified as Level C (high risk). Face / body images (D image ) and voice signal (D voice Original data such as ) is immediately classified as Level C because it has the highest potential for personal identification (highest risk level) and the greatest risk of privacy infringement upon leakage.

[0345] Data classified as Level C must be managed first and most strictly (immediately discarded or encrypted) in the subsequent anonymization / encryption processing unit.

[0346] The privacy level determination module (7200) classifies the collected data into the following three privacy levels based on the sensitivity of the collected data, the level of customer consent, and the purpose of use.

[0347] rating designation Data type example Processing rules Level A Completely anonymous Statistics, compilation Visit count, time spent Immediate de-identification processing and long-term retention possible Level B Pseudonym (Anonymous ID based) De-identification + Embedding Style vector, preference Retention for 30–90 days, allow identification of returning visitors Level C Original / High Risk Face, voice, video Recognition frame, voice data Encrypt immediately and discard within 24 hours

[0348] The anonymization and encryption module (7300) immediately applies double security processing to Level C data (original video, audio, etc.) collected in the data input stage (7100) and Level B data containing sensitive features even if they have already been de-identified. This fundamentally blocks the possibility of personal identification before the data is transmitted to / stored in a core server or data storage, and makes it impossible to understand the content even in the event of a data leak.

[0349] Depending on the level at which data can be utilized, the module can apply various techniques as follows.

[0350] For the purpose of de-identification and blocking the direct use of the original image, the original image (D image Face area masking can be applied to remove identifiability by blurring or pixelating areas where personal identification, such as faces, is possible.

[0351] For the purpose of shortening the lifecycle of original data with the highest risk of personal information leakage, an embedding vector is generated by extracting only the features of an individual from an image recognition module (110), etc., and then an original image (D) containing personal identification information is created. image Original disposal after embedding vector extraction can be applied, which immediately discards the original from the system to prevent storage itself.

[0352] For the purpose of protecting data content and making decryption impossible even in the event of a data leak, an encrypted storage method may be applied to data that needs to be stored after anonymization (e.g., Level B data) by applying a strong encryption algorithm such as AES256 or RSA.

[0353] The data storage and access control unit (7400) minimizes the risk of data misuse and leakage by insiders by separating storage locations according to data sensitivity (Level A, B, C) and applying the principle of least privilege to grant only the minimum necessary privileges to internal users, and includes a secure storage (7410), an aggregation storage (7420), an access controller (7430), and an access log recorder (7440).

[0354] The secure storage (7410) is a storage for highly sensitive data, such as Level B and Level C data, where feature vectors (Level B) mapped to an anonymous ID or encrypted original files (Level C) are stored. This storage stores data with strong encryption applied and is subject to the strictest control of the access controller (7430).

[0355] The aggregation repository (7420) is a repository that stores non-identifiable statistical data, such as Level A data, which makes personal identification impossible, and stores the results of statistically aggregating numerous customer behavior logs (average time spent, popular product rankings, etc.). Since the information stored in the aggregation repository has low sensitivity, it is highly accessible so that it can be used relatively freely for various purposes, such as AI model training or business analysis.

[0356] The access controller (7430) is a logical firewall that checks and controls the rights of users or systems accessing data, and the control method is based on RBAC (Role-Based Access Control).

[0357] For example, the Data Analyst role has only read access to Level A data, while the Security Administrator role has administrative access to Level B data. This allows unauthorized access to be blocked at the source.

[0358] The access log recorder (7440) is a monitoring device that records all data access attempts and success / failure events without omission, along with the time, subject, target, and result.

[0359] Recorded logs are used for auditing purposes, are essential for retrospectively tracking and analyzing attempts at internal misuse or signs of external intrusion, and serve as evidence to clarify liability in the event of a data breach.

[0360] The data lifecycle management module (7500) manages the data storage period and the timing of automatic deletion, ensuring that collected data is not preserved in the system indefinitely and controlling it to be automatically disposed of at a set time in accordance with legal requirements and privacy protection policies. This is a key safeguard that lowers the risk of long-term data leakage.

[0361] The preservation policies and utilization methods by level are as shown in Table 2 below.

[0362] Data level Conservation Policy Main purpose and processing Level C Automatically deleted within 24 hours or immediately discarded Data with the highest potential for personal identification, such as original video / audio, is discarded as quickly as possible immediately after recognition and feature extraction to ensure it does not remain within the system. Level B Automatically expires after 30–90 days and can be converted into anonymous training data. Data necessary for AI training and service provision, such as anonymized IDs and feature vectors. After being stored temporarily, it can be converted into Level A training data capable of long-term preservation once identification information (anonymous ID) is completely removed. Level A Long-term retention for statistical purposes (up to 3 years) These are fully aggregated and anonymized statistics, such as average customer behavior patterns. They are used on AI model training servers for long-term trend analysis and system performance optimization.

[0363] When deleting data, not only are files simply deleted, but a strict two-step process is followed to fundamentally block data recovery.

[0364] In the deletion step (7510), the encryption key is revoked and the data file is permanently deleted. The encryption key capable of decrypting the data is first revoked from the key management server, and then a Secure Deletion command is applied to the file containing the data to overwrite or destroy the file area so that the data cannot be restored even by a recovery tool.

[0365] In the verification step (7520), the actual deletion is verified through a hash check.

[0366] Technical verification is performed to confirm that the deletion command was actually executed successfully. The hash value of the file is recorded before deletion, and after deletion, the hash value of the corresponding storage area is checked again or an integrity check is performed to confirm and record that the data has been completely erased.

[0367] The data utilization and disposal log (7600) records the process of collecting, converting, storing, and deleting all data.

[0368] This serves as an electronic audit trail that records all significant events occurring throughout the entire lifecycle of customer data.

[0369] The purpose of this is to prove the legitimacy of access to and use of data, and to secure legal evidence to clearly trace the source of liability and the path of occurrence in the event of an incident (data leakage or misuse).

[0370] The log contains information such as the following Table 3.

[0371] Log entry explanation Purpose of use DataID (Anonymized ID) A unique ID that identifies which individual customer session the data is associated with in the log. Criteria for tracking by data object Creation time / Retention period The time when data was collected and the scheduled time for automatic deletion set by the data lifecycle management module Verification of compliance with preservation policies Access Users / Number of Accesses Internal users who accessed data by passing through the access controller and the number of accesses Detect unauthorized access attempts other than authorized users Deleter / Deletion time The point in time when data was permanently deleted through the deletion step and the system or administrator who performed the operation. Verify the final disposal history of the data and verify whether the deletion request has been fulfilled.

[0372] The logs are stored on a separate audit server (Audit Server, 7610) and can be used as evidence for external audits.

[0373] For the integrity and security of the log itself, the monitoring server (7610) is written to an independent server separated from the secure storage (7410) that stores the actual data. It is common for this server to store the log in an unmodifiable form.

[0374] This record serves as the most important evidence to demonstrate the system's transparency and security compliance efforts, not only for internal audits verifying compliance with the Personal Information Protection Act but also in response to inspection requests from external regulatory agencies.

[0375] Although the present invention has been described in relation to the preferred embodiments mentioned above, various modifications and variations are possible without departing from the essence and scope of the invention. Accordingly, the appended claims will include such modifications and variations that fall within the essence of the invention. Explanation of the symbols

[0376] 100 Edge Layer 110 Image Recognition Module 120 Feature Extraction Module 130 Conversational Interface Module 140 Edge Controller 200 Core Server Layer 210 Customer Identification Engine 220 Profile Management Module 230 Conversational AI Module 240 Recommendation / Control Decision Module 300 Device Control Layer 310 Photo Booth Device 320 Digital Signage 330 Lighting and Sound System 340 Payment and Interaction Terminal 400 Data Management Layer 410 Customer Profile DB 420 Product / Content DB 430 Model Training Server 440 Security Log Server 500 Network 501 Internal Network 502 Cloud Network 2100 Video Input 2110 RGB Camera 2120 Depth Sensor 2200 Object Detection Module 2210 Object Detection Network 2300 Object Tracking Module 2310 MOT Algorithm 2400 Appearance Feature Extraction and Re-identification Module 2410 Re-identification Neural Network 2500 Face Recognition Module 2510 Face Detector 2520 Face Embedding Generator 2600 Fusion Decision Module 2610 Probabilistic Fusion Model 2700 Result Transmission Unit 3100 Input Preprocessing Unit 3110 Image Reception Module 3120 Data Normalization Module 3200 Human Body Segmentation Module 3210 Segmentation Network 3300 Attribute Analysis Module 3310 Attribute Extractor 3320 Attribute Fusion Unit 3400 Logo / Character Recognition Module 3410 Logo Detector 3420 OCR Recognizer 3430 Logo Reliability Evaluator 3500 Feature Integration Module 3510 Feature Fusion Unit 3520 Normalization and Storage Module 5100 Input Data Collection Unit 5110 : Vector Embedding Generator 5120 Approx. Nearest Neighbor Finder 5130 Candidate Filter 5200 Reordering Module 5210 Feature Integrator 5220 Ranking Model 5230 : Multi-objective Optimizer 5300 Explainable Module 5310 Evidence Extractor 5320 Description Generator 5430 Interface Forwarder 5400 Result Output Section 6100 Core Server Section 6110 Command Generator 6120 Topic Router 6130 RPC Interface 6140 Feedback Listener 6200 MQTT Broker 6210 Publish Queue 6220 Topic Mapper 6230 Forwarding Engine 6240 Feedback Collector6300 Device Layer 6310 Photo Booth Device 6320 Digital Signage 6330 Lighting / Sound Controller 6340 Payment and Interaction Terminal 7100 Data Input Phase 7200 Privacy Class Determination Module 7300 Anonymization and Encryption Module 7310 Security Key Management Server 7400 Data Storage and Access Control Unit 7410 Secure Storage 7420 Aggregated Storage 7430 Access Controller 7440 Access Log Writer 7500 Data Lifecycle Management Module 7510 Deletion Phase 7520 Deletion Verification Phase 7600 Data Usage and Disposal Log 7610 Audit Server< / deviceid> < / storeid>

Claims

Claim 1 In an integrated environment control and customized interaction system through AI-based individual customer recognition within a store: an edge layer installed at the store site to receive a video stream of a customer, extract an appearance feature vector (Fvisual) and a face feature vector (Fface) from the video stream, and transmit the extracted vectors to a core server; a core server layer that receives the feature vectors from the edge layer, performs an integrated analysis of multiple customer features including the appearance feature vector and the face feature vector using a probabilistic fusion model, identifies the customer by generating an anonymous ID (anon_user_id) or mapping it to an existing anonymous ID according to the judgment result of the probabilistic fusion model, and calculates a customized service decision based on the anonymous ID; and a device control layer that controls at least one of a shooting mode, content display, and lighting settings in real time according to the decision of the core server layer.The edge layer includes a data management layer configured to store customer profile data and product content data linked to the anonymized ID, and to store activity records including histories of attempts to access personal information as logs, wherein the activity records include the anonymized ID; the edge layer includes a segmentation network that receives the video stream as input and separates human body regions into pixel units; an attribute analysis module that predicts clothing style, color palette, and whether accessories are worn using the separated human body regions as input; a logo / character recognition module that detects logos within clothing and shoe regions and classifies brand names using the separated human body regions as input; and a feature integration module that generates the appearance feature vector by combining the results of the attribute analysis module and the logo / character recognition module through a feature fusion unit; the core server layer includes a profile management module that stores and manages customer profiles linked to the anonymized ID and stores and manages customer profile data by loading historical data when a returning customer is mapped to the anonymized ID; a conversational AI module that generates customized response sentences based on the customer's utterances and behaviors based on the data of the profile management module; and a device control layer that recommends suitable services and products to the customer based on the anonymized ID and sends control commands to the device control layer. An integrated environment control and customized interaction system through AI-based individual customer recognition, characterized by including a transmitting recommendation and control decision module. Claim 2 delete Claim 3 delete Claim 4 A system according to claim 1, wherein the recommendation and control decision module calculates the control command using an AI conversational state machine that determines the service flow according to a plurality of predefined states and transition conditions based on the customer's interaction state and events. Claim 5 A system according to claim 1, wherein the device control layer comprises a photo booth device that changes a shooting mode, frame theme, and lighting settings according to the command of the recommendation / control decision module, and a digital signage that displays customer-customized content and AI recommendation results in real time. Claim 6 The system according to claim 1 further comprises an anonymization and encryption module that assigns a high-risk grade to original video and voice data collected from a customer, and performs at least one de-identification process among face region masking, original discarding after embedding vector extraction, and encrypted storage on the high-risk grade data. Claim 7 In claim 6, the system is characterized by further including a data lifecycle management module that applies a policy of automatic deletion or immediate disposal within 24 hours to the high-risk grade data, and performs encryption key disposal and hash check-based deletion verification upon deletion. Claim 8 A system according to claim 1, wherein the edge layer further receives a proximity signal through at least one of a BLE receiver, a Wi-Fi signal analyzer, or an IR sensor, and the core server layer inputs the proximity signal together with the appearance feature vector and the face feature vector into the probabilistic fusion model to perform customer identification. Claim 9 delete Claim 10 delete Claim 11 delete

Images