Unmanned retail interaction method and system combining AI voice shopping guide and cloud-based supervision

Abstract

The present invention relates to the field of unmanned retail interaction technology, and in particular to an unmanned retail interaction method and system combining AI voice shopping guides with cloud-based supervision. The present invention proposes the following scheme: through the collection of multimodal information such as voice commands, facial expressions, visual focus, and environmental data, a personalized product recommendation list is generated in combination with a multimodal fusion neural network model. User behavior feedback is transmitted to the system in real time. If the currently recommended product does not meet the user's needs, the rejected product and its similar products are removed from the product removal list, and the product recommendation list is dynamically updated. Accurate and personalized product recommendations are achieved, redundant product recommendations are avoided, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned retail interaction technology, and in particular to an unmanned retail interaction method and system combining AI voice shopping guide and cloud-based supervision. Background Art

[0002] Existing unmanned retail interactive systems typically rely on traditional recommendation algorithms, primarily generating recommendations based on historical purchase data, product categories, or static user needs. However, these methods often fail to respond to changes in user emotions and behavior in real time, resulting in delayed and inaccurate recommendations. Furthermore, traditional product recommendation systems often rely on a single data source, such as a user's purchase history or browsing history, lacking a deep understanding of the user's immediate needs. Especially in diverse retail environments, product recommendations can be redundant and repetitive, making it difficult to meet users' ever-changing, personalized needs.

[0003] For example, the Chinese patent with authorization announcement number CN111652606B discloses a smart retail terminal system, including a retail terminal, a server, an Internet of Things platform, and a user terminal; the retail terminal includes a network unit, a central control unit, a payment identification module, and a smart terminal interaction module; the server includes a data processing unit, a data storage unit, and a smart terminal shopping guide module; the Internet of Things platform includes a Lot network unit, a Lot message unit, and a Lot rule unit; the smart terminal interaction module conducts a voice conversation with the user, and after the user's voice command is issued, the voice command is converted into a text command, and the text command and user information are sent to the smart terminal shopping guide module; the smart terminal shopping guide module obtains the user's shopping information based on the user information, predicts the user's future shopping list through comparison and intelligent analysis of the shopping information, matches coupons, and recommends shopping information to the user based on the text command. This invention can enhance the stability of the unmanned vending machine, improve the interactivity of the unmanned vending machine, and reduce operating costs.

[0004] The above existing technologies all have the problems raised by this background technology: there is an urgent need for a more intelligent and dynamic recommendation mechanism that can combine multiple perception data and real-time feedback to flexibly adjust product recommendations to adapt to the changing needs of different users and situations. To solve the above problems, this application designs an unmanned retail interaction method and system that combines AI voice shopping guides with cloud duty. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology and provide an unmanned retail interaction method and system that combines AI voice shopping guides with cloud-based supervision. Through the collection of multimodal information such as voice commands, facial expressions, line of sight, and environmental data, a personalized product recommendation list is generated in combination with a multimodal fusion neural network model. User behavior feedback is transmitted to the system in real time. If the currently recommended product does not meet the user's needs, the rejected product and its similar products are removed from the product removal list, and the product recommendation list is dynamically updated. Accurate and personalized product recommendations are achieved, redundant product recommendations are avoided, and the user experience is improved.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service, the unmanned retail interaction method comprising:

[0008] Inputting the acquired user demand information into a multimodal fusion neural network model, the multimodal fusion neural network model outputting a product recommendation list, the product recommendation list including a first area, a second area, and a third area arranged in sequence, each area including a plurality of sequentially arranged product information;

[0009] Performing an iterative operation, the iterative operation including swapping the positions of the products currently displayed in the first area and the third area, determining the area where the user's favorite products are located based on the change range projected onto the product recommendation list display interface by the user's field of view, and updating the product information in the area to the current product recommendation list until the number of products in the product recommendation list falls below a set threshold;

[0010] Play the first product in the product recommendation list through the AI ​​shopping guide voice, and confirm whether the product information currently being broadcast meets the user's needs based on the user's behavioral feedback after listening to the AI ​​shopping guide voice;

[0011] If not, update the product recommendation list, and select the updated first product related information from the product recommendation list until it meets the user's needs.

[0012] The user demand information includes voice intention information, visual feedback information and environmental perception data;

[0013] The multimodal fusion neural network model includes:

[0014] A multimodal input layer, configured to extract acoustic features, facial emotion features, and an environmental context vector, wherein the multimodal input layer includes a speech branch, a visual branch, and an environmental branch;

[0015] A cross-modal alignment layer, configured to dynamically align the outputs of the multimodal input layer using a multi-head attention mechanism to generate a joint embedding representation, and to update the embedding weights of the joint embedding representation based on the modal confidence calculated by the multi-head attention mechanism;

[0016] The intention reasoning layer is used to input the joint embedding representation into a preset hierarchical reinforcement learning framework to generate a product recommendation list, wherein the hierarchical reinforcement learning framework includes an upper-layer policy network and a lower-layer policy network, wherein the upper-layer policy network is used to generate a product candidate pool based on the user's historical interaction data, product inventory and environmental factors, and the lower-layer policy network is used to screen the product candidate pool according to the user's interests, generate a product recommendation list, and divide the product recommendation list into a first area, a second area and a third area according to product categories.

[0017] The speech branch is used to extract acoustic features in the speech intention information according to a pre-trained speech encoder;

[0018] The visual branch is used to extract temporal features from the visual feedback information based on the dual-stream network, locate the user's visual focus area, analyze the visual feedback information through a lightweight convolutional network, and assign weighted values ​​to the visual focus area based on the analysis results to obtain facial emotion features;

[0019] The environment branch is used to process the environment perception data according to the spatiotemporal convolutional network to construct an environment context vector.

[0020] The determining, based on a change range of the user's field of view projected onto the product recommendation list display interface, of a region where the product of interest to the user is located, includes:

[0021] Get the user's head posture information;

[0022] Calculating the user's gaze direction based on the head posture information;

[0023] The user's gaze direction is mapped to the display interface of the product recommendation list through perspective projection transformation to generate a field of view projection area, wherein the field of view projection area is cross-calculated with the product area coordinates on the product recommendation list to determine whether the product the user is gazing at belongs to the first area or the third area;

[0024] Aggregate the user's visual field projection area within a fixed time window to determine the area where the products the user is interested in are located.

[0025] Determining the area where the product of interest to the user is located includes:

[0026] Obtain the visual field projection trajectory and the visual field projection trajectory before the product position is swapped based on the aggregation result;

[0027] If the user's field of view projection trajectory shifts in the swapped product area, and the new field of view projection area is concentrated in a specific area, an update of the product recommendation list is triggered based on the specific area;

[0028] If the user's visual field projection trajectory remains stable before and after the product positions are swapped and does not deviate from the changes in the first area and the third area, an update of the product recommendation list is triggered based on the second area.

[0029] The confirmation of whether the currently reported product-related information meets the user's needs includes:

[0030] Collect the user's movement trajectory data after listening to the AI ​​shopping guide voice, and extract the user's spatial movement characteristics and time stay characteristics;

[0031] Determine whether the user moves to the area where the recommended product is located based on the spatial movement characteristics and the time residence characteristics. If not, it indicates that the user's needs are not met;

[0032] If yes, acquiring interaction image data of the user and the product shelf through an image sensor arranged in the area where the recommended product is located, processing the interaction image data, and determining whether the user has performed an interaction behavior of picking up the recommended product;

[0033] The output is judged to determine whether it meets the user's needs.

[0034] The extracting of the user's spatial movement features and temporal stay features includes:

[0035] Extracting the user's movement path within the commodity shelf area based on the user's position coordinates at each moment in the movement trajectory data;

[0036] and calculating the instantaneous speed of the user based on the continuous positions, performing cluster analysis on the motion path and instantaneous speed, and extracting the spatial movement characteristics of the user;

[0037] According to the motion trajectory data, the user's stay time in each area is obtained through a sliding time window, and the stay density is calculated according to the number of windows. The periodic characteristics are obtained according to the user's stay behavior, and the stay pattern is calculated according to the stay density and periodic characteristics;

[0038] The time series in the user motion trajectory data is modeled according to the dwell time and dwell pattern, and the user's time dwell feature is extracted.

[0039] The processing of the interactive image data includes:

[0040] According to the interactive image data captured by the image sensor, interactive inter-frame images are acquired frame by frame, and differential denoising and background subtraction are performed on the interactive inter-frame images to acquire optimized inter-frame images;

[0041] Calculating pixel displacement vectors corresponding to user pixels in the optimized inter-frame image by an optical flow method, and determining user skeleton information based on the pixel displacement vectors;

[0042] Skeletonize the skeleton information according to morphological operations to obtain a user skeleton graph, process the user skeleton graph according to adaptive graph convolution, and calculate key points of the user's upper limb joints;

[0043] Using the key points as input parameters of a preset posture recognition network, training the input parameters through the posture recognition network, and outputting a body movement category;

[0044] It is determined whether the user performs an interactive behavior of picking up the product based on the body movement category.

[0045] The updating of the product recommendation list includes:

[0046] Identify and mark the currently advertised product as a rejected product;

[0047] Use a similarity matching algorithm to retrieve products in the product recommendation list that are similar to the rejected products in terms of attributes and categories, and obtain a similar product set;

[0048] Remove the currently reported product and similar product sets from the product recommendation list.

[0049] The unmanned retail interaction method further includes:

[0050] Generate a product rejection list based on the rejected products;

[0051] The product elimination list is used as a constraint condition to update the hierarchical reinforcement learning framework of the multimodal fusion neural network model.

[0052] An unmanned retail interactive system combining AI voice shopping guides and cloud-based on-call services, comprising a data collection module, a product recommendation module, and a recommendation feedback module;

[0053] The data acquisition module is used to collect the user's voice, visual behavior and environmental data in real time through various sensor components to generate user demand information;

[0054] The product recommendation module is used to generate a product recommendation list through a multimodal fusion neural network model based on the user demand information provided by the data acquisition module;

[0055] The recommendation feedback module is used to receive and analyze user behavior feedback information, update the product recommendation list according to the behavior feedback information, and remove rejected products and similar products.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. This invention combines AI voice shopping guides with cloud monitoring and uses multimodal data to recommend products, thereby improving the real-time and personalization of the recommendation system.

[0058] 2. Through the introduction of a product elimination list and a reinforcement learning framework, the present invention can adjust the recommendation strategy in real time, eliminate products that do not meet user needs and their similar products, and ensure that each recommendation is more accurate and in line with user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0060] Figure 1 This is a flow chart of the unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service in Example 1 of the present invention;

[0061] Figure 2 This is a structural diagram of a multimodal fusion neural network model according to Example 1 of the present invention;

[0062] Figure 3 This is a structural diagram of the multimodal input layer of Example 1 of the present invention;

[0063] Figure 4 Schematic diagram of dimensionality reduction of the product recommendation list in Example 1 of the present invention Figure 1 ;

[0064] Figure 5 Schematic diagram of dimensionality reduction of the product recommendation list in Example 1 of the present invention Figure 2 ;

[0065] Figure 6 This is a flowchart of interactive inter-frame image difference denoising processing according to embodiment 1 of the present invention;

[0066] Figure 7 This is a module diagram of the unmanned retail interactive system combining AI voice shopping guide and cloud-based supervision in Example 2 of the present invention. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0068] Example 1:

[0069] See also Figure 1 The present invention provides an embodiment: an unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service. The specific steps of the method are as follows:

[0070] S1: Input the acquired user demand information into the multimodal fusion neural network model and output a product recommendation list;

[0071] In this embodiment, the acquired speech, visual, and environmental information is used as input and passed to a multimodal fusion neural network model. The model processes different modal data through the speech branch, visual branch, and environmental branch respectively, uses the self-attention mechanism for dynamic alignment, fuses the data of different modalities, and generates a joint embedding representation of user needs. In this way, it can not only process data from different sensors, but also combine the complementarity of multimodal data to ensure that the recommendation list can fully reflect the user's interests and needs. For example, facial expressions and voice commands used can better guide the recommendation algorithm, avoid the information bias that may occur in a single modality, and improve the accuracy of recommendations.

[0072] S2: Perform dimensionality reduction on the product recommendation list;

[0073] In this embodiment, according to the three-area division of the product recommendation list, a dynamic dimensionality reduction strategy based on user gaze tracking is used to gradually screen out products that better meet user needs, thereby reducing the dimensions of recommended products and improving recommendation accuracy and user interaction efficiency.

[0074] S3: Play the information about the first product in the product recommendation list through the AI ​​shopping guide voice;

[0075] In this embodiment, the AI ​​shopping guide uses voice to broadcast relevant information about the first product in the product recommendation list, so that users can immediately obtain the recommended product details. The voice broadcast content includes key information such as the product name, price, characteristics, and location. The AI ​​shopping guide's voice content is dynamically generated according to user needs using natural language processing technology, ensuring that the broadcast content accurately fits the user's needs. The advantage of this is that users can interact with the in-store system instantly through voice, saving the time of manually searching for product information and improving the shopping experience.

[0076] S4: Obtaining behavioral feedback information from the user after listening to the AI ​​shopping guide voice;

[0077] In this embodiment, visual and motion sensors are used to monitor user behavioral feedback, including body movements (such as whether they move toward the recommended product area) and whether they remain near the recommended products. Behavioral analysis algorithms can capture user behavioral tendencies in real time. For example, if a user moves toward the recommended product area, it can be assumed that the recommended product is likely to meet their needs. If the user expresses doubt or leaves the area, it can be determined that the recommended product is not of interest.

[0078] S5: Confirm whether the product-related information currently being broadcast meets the user's needs;

[0079] S6: If not, update the product recommendation list and select the first updated product-related information from the product recommendation list until the user's needs are met;

[0080] In this embodiment, if the currently advertised product information is determined to not meet user needs, the product recommendation list is updated based on a product similarity matching algorithm. The currently recommended product and similar products are removed to avoid recommending similar products to users who are not interested. The updated recommendation list is then re-ranked based on user feedback, and the remaining products are selected as the ones most likely to meet the user's needs for new recommendations. This process dynamically adjusts the recommendation list to ensure that each recommendation is more closely aligned with user needs, avoids redundant recommendations, and provides an efficient and personalized shopping experience.

[0081] Traditional product recommendation systems primarily rely on static data models and historical purchase data to generate recommendations. However, these methods often lack real-time and dynamic feedback. When user needs change rapidly or product placement within a store shifts, traditional recommendation systems may be unable to adjust their recommendations in a timely manner, resulting in recommendations that don't align with user needs. This is especially true in the dynamic, unmanned retail environment, where traditional recommendation methods struggle to capture real-time user behavioral feedback, such as facial expressions and eye focus, which in turn affects the accuracy and personalization of recommendations.

[0082] Specifically, this embodiment proposes an unmanned retail interaction method based on the combination of AI voice shopping guides and cloud-based on-call services, which can respond to user needs in real time and achieve higher recommendation accuracy. By adopting a multimodal fusion neural network model, this application can comprehensively consider the user's voice commands, visual behavior feedback (such as facial expressions and eye focus), and environmental data (such as store temperature and lighting), and generate a personalized product recommendation list in real time based on this data. For example, when a user issues a voice command "I want to buy red sneakers," the system not only makes recommendations based on the voice content, but also considers the user's facial expressions (such as smiling or confused) and eye focus (such as whether they are staring at a pair of shoes on the shoe rack) to determine their points of interest and further optimize the recommended content. Compared with traditional methods, it can make real-time adjustments based on user feedback in a dynamic environment and avoid the lag in traditional recommendation systems.

[0083] Furthermore, this embodiment incorporates real-time user feedback. By analyzing behavioral signals such as user dwell time, eye focus, and changes in facial expression, it can determine whether the user has expressed interest in a recommended item. If the user shows no interest in a recommended item (e.g., quickly leaves the item area or displays a doubtful expression), the item is automatically removed from the recommendation list, and similar items are removed from the list, ensuring the novelty and accuracy of the recommendations. Unlike traditional systems with static recommendations, this invention dynamically updates the recommendation list, responding to user needs in real time and avoiding repeated recommendations of items that the user is not interested in, significantly improving the personalization and effectiveness of recommendations.

[0084] In this embodiment, cloud-based monitoring refers to optimizing the accuracy and real-time performance of product recommendations by collaborating local devices with cloud servers and utilizing the powerful computing and data processing capabilities of the cloud.

[0085] Specifically, the cloud server is responsible for conducting in-depth analysis of multimodal data from local devices, adjusting the product recommendation list in real time, and dynamically updating it based on user behavioral feedback (such as changes in facial expressions and dwell time). Cloud monitoring also ensures data synchronization across multiple terminal devices, ensuring that recommendations remain consistent regardless of which terminal a user interacts with in the store and can be optimized based on global information. Furthermore, the cloud continuously updates recommendation strategies and models through the collaboration of reinforcement learning and multimodal fusion neural networks, ensuring that recommendation results continue to improve as user needs evolve, thereby enhancing the personalization and accuracy of product recommendations.

[0086] See also Figure 2 The structure diagram of the multimodal fusion neural network model in an embodiment of the present invention combines voice, visual, and environmental data to provide a highly personalized and dynamically optimized product recommendation system that can meet the complex and ever-changing user needs in unmanned retail environments. The multimodal fusion neural network model includes:

[0087] A multimodal input layer, configured to extract acoustic features, facial emotion features, and an environmental context vector, wherein the multimodal input layer includes a speech branch, a visual branch, and an environmental branch;

[0088] See also Figure 3 Figure 2 shows the structure of the multimodal input layer in an embodiment of the present invention. In this embodiment, the model's input layer is multimodal, responsible for extracting and processing various user data from different sources, including voice, visual feedback, and environmental data. Each input type has a corresponding branch, including a voice branch, a visual branch, and an environmental branch, each responsible for extracting information from different modalities to ensure the system fully understands user needs.

[0089] The speech branch is used to extract acoustic features in the speech intention information according to a pre-trained speech encoder;

[0090] Specifically, because voice is a natural way for users to interact with the system and efficiently express their needs, the core task of the voice branch is to extract the acoustic features of speech from the user's voice commands. This application uses a pre-trained speech encoder and a long short-term memory (LSTM) speech model to extract acoustic features from audio data. Through these acoustic features, the system can understand the user's basic needs (for example, inquiries about product type and features).

[0091] Furthermore, before being input into the speech model, audio data undergoes preprocessing, such as denoising and normalization, to ensure input signal quality and improve the accuracy of subsequent model processing. It then passes through a speech feature extractor, typically using techniques such as Mel-Frequency Cepstral Coefficients (MFCCs) or Mel-Filter Spectrum (MFBs). These extractors can capture energy variations across different frequency ranges in audio signals and are particularly well-suited for capturing information such as the pitch and frequency of speech. By extracting these intonation features, it is possible to distinguish different aspects of speech content and, consequently, identify the user's intent.

[0092] Furthermore, based on the extracted intonation features, the system performs temporal processing using an LSTM. LSTM is particularly well-suited for processing time series data, retaining information over long time periods and handling long-term dependencies. In voice commands, user expressions are often sequential, and LSTM effectively captures these dependencies across time periods through its memory mechanism. Specifically, LSTM can process sentences such as "buy a pair of red running shoes," understanding the multi-layered structure and contextual meaning of the entire sentence. By using LSTM, the system can understand the details within long sentences while avoiding the vanishing gradient problem of traditional RNNs, thereby improving the recognition accuracy of voice commands.

[0093] The visual branch is used to extract temporal features from the visual feedback information based on the dual-stream network, locate the user's visual focus area, analyze the visual feedback information through a lightweight convolutional network, and assign weighted values ​​to the visual focus area based on the analysis results to obtain facial emotion features;

[0094] Specifically, because facial expressions and gaze focus accurately reflect users' immediate emotional reactions, the system dynamically adjusts recommendations to avoid recommending products that the user is not interested in. By accurately analyzing visual data, the system can improve the relevance of product recommendations and update recommendations in real time based on user emotional feedback, making recommendations more tailored to user needs.

[0095] In this embodiment, a dual-stream network processes each frame of a video sequence and combines it with temporal information to analyze changes in the user's facial expression and the area of ​​their gaze. This allows the system to not only determine the user's current emotions (such as preference, dissatisfaction, or confusion), but also analyze their level of interest in products. For example, if a user is smiling and gazing at a particular product, the system infers a high level of interest in that product and prioritizes its recommendation.

[0096] Specifically, the dual-stream network is a deep learning architecture that consists of two sub-networks, one of which focuses on processing the spatial features of the image, and the other focuses on processing the temporal features of the image. In this embodiment, the two sub-networks of the dual-stream network process the user's facial expressions and eye focus respectively. Specifically, spatial features are extracted for each frame in the video sequence, and then the temporal relationship between the frames is analyzed to capture the temporal changes in facial expressions and eye focus. The sub-network can be replaced with the existing network according to specific needs. In this embodiment, the sub-network includes an RNN neural network and a DCN neural network.

[0097] Furthermore, the DCN neural network extracts the user's facial expression features from each image frame. Using a trained facial expression recognition model, the system can accurately identify the user's emotional state, such as smile, frown, surprise, and confusion. Facial expressions, as direct expressions of emotion, effectively reflect the user's interests and attitudes. Based on the user's facial expressions, the system infers their emotional response to the recommended products, further helping the system determine whether the recommendations meet the user's needs.

[0098] Furthermore, the dual-stream network uses an RNN neural network to perform time-series analysis, tracking the user's gaze focus and determining the area of ​​interest within the image. The system can identify the direction and duration of a user's eye movements while viewing a product, thereby determining their level of attention to a particular product or category. If a user's gaze remains focused on a particular product for an extended period, the system infers that the product is of interest and prioritizes recommendations.

[0099] In this embodiment, facial expressions and eye focus accurately reflect the user's immediate emotional response, helping the system quickly capture the user's focus and emotional state. Changes in facial expression, such as joy or confusion, can reflect the user's interest in the recommended product; changes in eye focus can intuitively indicate whether the user has developed interest in a particular product. Therefore, through weighted assignment, the system can more accurately analyze user needs, dynamically adjust recommendations, and avoid recommending products that the user is not interested in.

[0100] Furthermore, the facial expression and gaze focus information obtained through the two-stream network analysis is further processed by a lightweight convolutional network. The lightweight convolutional network's task is to assign weighted values ​​to the user's facial emotions and gaze focus areas, dynamically adjusting the weights of each area based on its importance and user emotional feedback. For example, a user's smile and prolonged gaze on a product will give the product a higher weight, indicating that the user has shown strong interest in the product and thus prioritizes recommendations for that product.

[0101] The environment branch is used to process the environment perception data according to the spatiotemporal convolutional network to construct an environment context vector;

[0102] In this embodiment, environmental perception data includes information such as temperature, light intensity, and store customer flow. This information helps to determine the user's current shopping environment and needs. For example, if the lighting in the store is too strong or the environment is too crowded, the user may prefer to make quick decisions on products. By processing this environmental perception data, the system can optimize the recommended content, such as recommending products that facilitate quick decision-making. Through the spatiotemporal convolutional network, it is possible to efficiently analyze the impact of different environmental conditions on user needs, thereby improving the adaptability and accuracy of the recommendation list. This is because environmental factors have a significant impact on users' purchasing decisions in retail scenarios. Using environmental data to dynamically adjust product recommendations makes the recommended content more in line with actual shopping needs, increasing the efficiency of users' purchasing decisions.

[0103] A cross-modal alignment layer, configured to dynamically align the outputs of the multimodal input layer using a multi-head attention mechanism to generate a joint embedding representation, and to update the embedding weights of the joint embedding representation based on the modal confidence calculated by the multi-head attention mechanism;

[0104] Specifically, the goal of the cross-modal alignment layer is to generate a unified embedding representation by analyzing the correlation of multiple modal information. This representation integrates relevant information of speech, vision, and environment to facilitate subsequent recommendation decisions.

[0105] Specifically, the cross-modal alignment layer feeds the output of each input modality into a multi-head attention mechanism. The core idea of ​​the multi-head attention mechanism is to dynamically select the most relevant information for the current task by performing weighted calculations on multiple subspaces of the input. Therefore, during the computational process, multiple different "attention heads" are processed in parallel, each focusing on a different feature or dimension of the input information, thereby fully exploiting the complementary information of the multimodal input.

[0106] Furthermore, in cross-modal alignment, the speech, visual, and environmental input data are first mapped to the same embedding space. For example, the acoustic features of speech are mapped to a continuous vector space using a pre-trained speech encoder. Visual data such as facial expressions and gaze focus are extracted using a convolutional neural network and mapped to the same space. Environmental data is also mapped to the same embedding space after being processed by a spatiotemporal convolutional network. This ensures that the outputs of all modal data have the same dimensionality, enabling straightforward alignment calculations.

[0107] Furthermore, the multi-head attention mechanism performs a weighted summation of these embedding vectors, dynamically assigning different attention weights to each modality based on its importance in the current task. For example, when recognizing a user's voice needs, the voice modality may be given a higher weight, while when the user expresses a strong emotional reaction or expresses interest in a certain product, the visual modality may be given a relatively higher weight. In this way, each modality can be dynamically weighted according to task requirements, ensuring that the most relevant information is prioritized.

[0108] Furthermore, through calculations using a multi-head attention mechanism, the system obtains confidence scores for each modality. These confidence scores represent the relative importance of each modality in the current decision-making task. For example, if a user inquires about a product via voice, the voice modality may be given a higher weight. If the user expresses strong interest in a product (e.g., by gazing at it for a long time or expressing a favorable facial expression), the visual modality's weight increases, and vice versa. These confidence scores are calculated using a self-attention mechanism, where each modality's input dynamically adjusts attention allocation by calculating the relationship between the query, key, and value.

[0109] an intent inference layer, configured to input the joint embedding representation into a preset hierarchical reinforcement learning framework to generate a product recommendation list, wherein the hierarchical reinforcement learning framework includes an upper-layer policy network and a lower-layer policy network, wherein the upper-layer policy network is configured to generate a product candidate pool based on the user's historical interaction data, product inventory, and environmental factors, and the lower-layer policy network is configured to filter the product candidate pool based on the user's interests to generate a product recommendation list, and divide the product recommendation list into a first region, a second region, and a third region based on product category;

[0110] In this embodiment, the intent reasoning layer is used to convert the information after the fusion of multiple modalities into the final product recommendation list. The core idea is to realize personalized product recommendations through the collaboration of the upper policy network and the lower policy network based on the hierarchical reinforcement learning framework. In traditional recommendation systems, static rules or a single data source are usually relied upon to generate recommendations. However, this application can provide more accurate, dynamic and personalized recommendation effects by combining real-time feedback and global state information. The upper policy network and the lower policy network respectively undertake different tasks. The upper policy network is responsible for generating a preliminary list of candidate products from the global state, while the lower policy network refines and optimizes the candidate product list based on real-time feedback information. The cooperation between the two can make product recommendations have a global perspective and can quickly respond to users' immediate needs.

[0111] Specifically, the upper-level policy network processes input from the global state to generate a preliminary list of candidate products. This global state includes static data such as historical interaction data, product inventory information, and promotional events. This data helps the system understand long-term user preference trends and product availability. The upper-level policy network analyzes this global state to select products that are likely to align with the user's interests for the candidate list. This process aims to provide a broad pool of product candidates that encompasses potential user interest. The upper-level policy network uses a deep neural network to process global information. Combining historical data, product categories, and other global variables, it generates a diverse set of candidate products, ensuring coverage of potential needs across diverse users. Specifically, the upper-level policy network uses user behavior patterns in historical data (such as user preferences across product categories) to predict products that may be of interest to the user. The network then selects a list of products that align with the user's interests based on characteristics such as product category, price, and availability.

[0112] Furthermore, the upper-level policy network is trained by inputting the global state into a deep neural network, enabling it to learn the underlying patterns and regularities of user preferences. This deep neural network combines historical shopping trends, interactions between product categories, and current product inventory information to generate a preliminary pool of candidate products for each user. This pool contains multiple products that can cover the potential needs of different users. For example, based on past data, the system predicts which products may be attractive to a specific user and prioritizes those with advantages in inventory and promotions. The generated list of candidate products comprehensively considers product categories, sales trends, inventory status, and promotional information, ensuring a diverse selection of candidate products that meet the needs of different types of users.

[0113] In this embodiment, the system dynamically adjusts the order and partitioning of product recommendations through real-time feedback information, such as facial expressions, dwell time, eye focus, and interactive behaviors with products (such as clicking, picking up, etc.).

[0114] Specifically, the lower-level policy network analyzes user behavioral patterns during interactions to select products that best meet current needs and optimize recommendations. For example, if a user displays a prolonged gaze or a pleasant facial expression on a product in a certain category, the system prioritizes products in that category in the recommendation list and distributes them appropriately to different areas. For example, highly compatible products may be placed in the second area to increase user attention to products in that category; low-matching products may be moved to the first or third area, or removed in subsequent iterations to avoid disrupting the user.

[0115] Furthermore, the underlying policy network employs a reinforcement learning mechanism, continuously optimizing the recommendation strategy through a reward and penalty mechanism. When a user expresses positive feedback on a product (e.g., clicks, extended stays, purchases, etc.), the system assigns a higher weight to that product, ensuring that it is prioritized in future recommendations and placed in the area most likely to attract users. Conversely, if a user's feedback on a product is negative (e.g., leaving quickly, expressing confusion), the system lowers the product's recommendation weight and may relocate it to a less popular area (such as the first or third area), or remove it from subsequent recommendation list updates. This mechanism allows the order of product recommendations to dynamically adjust based on user interests, ensuring that each round of recommendations better meets the user's immediate needs and improves the accuracy and personalization of recommendations.

[0116] In this embodiment, the advantage of this framework lies in its ability to dynamically optimize recommendations, ensuring that recommendations are not solely based on historical user preferences but also instantly respond to user behavior and sentiment. By combining global information with real-time feedback, the system can quickly and accurately rank the initially screened candidate products, providing personalized and precise recommendations.

[0117] See also Figure 4 and Figure 5, a schematic diagram of dimensionality reduction of the product recommendation list in an embodiment of the present invention, first, the positions of the products in the first area and the third area are swapped, and then the area of ​​user attention is calculated through the range of change of the user's field of view projection area, and the degree of interest in the replaced product is analyzed. If the user's sight remains in the original area after the product swap, and no obvious interest is shown in the replaced product, it is determined that the user is more concerned about the second area, and then the product recommendation content in this area is adjusted first, and the uninterested products are removed from the recommendation list. If the user's field of view projection shifts, indicating that he is more interested in the products in the first area or the third area, the product content in this area will be further refined and optimized, and the products that fail to attract the user's attention will be removed. By Figure 5 It can be seen that when the first area is the user's focus area, the products in the first area are used as a new product recommendation list to replace the original product recommendation list, and the products in the first area are divided into new first, second and third areas.

[0118] Through this dimensionality reduction processing based on eye tracking, this embodiment can effectively reduce invalid information in the product recommendation list, improve the accuracy of product matching, and at the same time reduce the user's decision-making cost, making the recommendation process more intelligent and efficient. Compared with the traditional recommendation method based on fixed sorting or manual screening, this application can automatically optimize recommended products through real-time interaction to ensure that the products finally presented to the user are more in line with their interests, thereby improving the user experience and the recommendation efficiency in unmanned retail environments. The specific steps of S2 are as follows:

[0119] S2.1: Obtain the user's head posture information;

[0120] S2.2: Calculating the user's gaze direction based on the head posture information;

[0121] S2.3: Mapping the user's gaze direction to the display interface of the product recommendation list through perspective projection transformation to generate a field of view projection area, wherein the field of view projection area is cross-calculated with the coordinates of the product areas on the product recommendation list to determine whether the product being gazed by the user belongs to the first area or the third area;

[0122] S2.4: Aggregate the user's visual field projection area within a fixed time window to determine the area where the products the user is interested in are located;

[0123] S2.5: Obtain the visual field projection trajectory and the visual field projection trajectory before the product position is swapped based on the aggregation result;

[0124] S2.6: If the user's field of view projection trajectory shifts within the swapped product area, and the new field of view projection area is concentrated in a specific area, an update of the product recommendation list is triggered based on the specific area;

[0125] S2.7: If the projection trajectory of the user's visual field remains stable before and after the product positions are swapped, and does not deviate from the changes in the first area and the third area, then an update of the product recommendation list is triggered based on the second area.

[0126] The specific steps of S5 are as follows:

[0127] S5.1: Collect the user's movement trajectory data after listening to the AI ​​shopping guide voice, and extract the user's spatial movement characteristics;

[0128] In this embodiment, system-installed motion sensors (such as infrared sensors, ultrasonic sensors, or Wi-Fi-based positioning systems) track users' movements within the store in real time. These sensors can capture users' location information and movement paths with high precision. By recording the user's location coordinates at every moment, the system can generate spatial movement characteristics, such as the user's movement direction, speed, and frequently visited areas within the store. Furthermore, the system uses time-based dwell characteristics to analyze the duration and frequency of users' dwell time in specific product areas. This process monitors user dwell time in real time and can cluster dwell patterns based on time windows, accurately identifying whether users have a sustained interest in a particular product category.

[0129] Specifically, based on the position coordinates of the user at each moment in the motion trajectory data, the user's motion path in the shelf area is extracted, and the user's instantaneous speed is calculated based on the continuous position. Cluster analysis is performed on the motion path and instantaneous speed to extract the user's spatial movement characteristics.

[0130] In this embodiment, spatial movement features refer to the user's movement trajectory within the shelf area, including the user's movement path, speed, and movement pattern. The purpose of extracting these features is to provide the system with the user's spatial activity pattern, so that it can identify behaviors related to certain actions (such as selecting or browsing products) during the user's shopping process.

[0131] Specifically, the sensor obtains the user's position coordinates at every moment. Whenever the user moves in the shelf area, the system records their position coordinates and forms a time series that reflects the user's trajectory in space. Based on the user's coordinates at each time point, the user's movement path can be drawn. This path shows the user's movement trajectory in the shelf area. For example, if a user walks from one end of the shelf to the other, or stops and browses in front of a shelf, the instantaneous speed at each time point can be calculated by calculating the difference between two consecutive position coordinates. The instantaneous speed can help the system identify whether the user is browsing quickly, standing still, or moving slowly.

[0132] Furthermore, by performing cluster analysis on user movement paths and instantaneous speed data, the system can categorize user behavior into different movement patterns. For example, users may browse shelves in different ways: quickly passing by, carefully selecting items, or pausing to observe. Cluster analysis can identify and categorize these patterns, helping the system understand the user's spatial behavior characteristics. Spatial movement characteristics help analyze the user's areas of interest, leading to more accurate product recommendations. For example, if a user spends a long time in front of a certain shelf, they may have a high interest in products in that category.

[0133] S5.2: Determine whether the user has moved to the area where the recommended product is located based on the spatial movement characteristics. If not, it indicates that the user's needs are not met.

[0134] In this embodiment, this judgment method, through real-time analysis of users' spatial behavior and dwell time, avoids the shortcomings of traditional recommendation systems that rely solely on voice commands or historical data, ensuring that recommendations can adapt to dynamic user behavior and needs. In unmanned retail environments, user needs are often difficult to accurately capture through a single method. However, this application enhances the flexibility of recommendations through a dynamic feedback mechanism, avoiding the possibility of missing potential user needs.

[0135] Specifically, based on data extracted from motion sensors, a behavioral inference algorithm can determine whether the user has moved into the area where recommended items are located. If the user shows interest in the recommended item area (for example, by walking toward it or spending a significant amount of time there), the system considers the item recommendation to be in line with the user's needs. Conversely, if the user does not move toward the recommended item area, or spends very little time there, the system determines that the recommended item does not meet the user's needs.

[0136] S5.3: If yes, obtain image data of the user's interaction with the product shelf using an image sensor positioned in the area where the recommended product is located, process the image data, and determine whether the user has engaged in an interactive behavior of picking up the recommended product.

[0137] In this embodiment, the system deploys image sensors (such as high-definition cameras or depth cameras) in each product area to capture real-time image data of users interacting with the shelves. Using computer vision algorithms (such as object recognition, motion recognition, and deep learning techniques), the system can determine whether the user has attempted to pick up the product. By capturing the user's hand movements, the image sensors identify whether the user has picked up the product and, therefore, confirm whether the user has expressed interest in the recommended product.

[0138] By processing and analyzing interactive image data in real time, the system can accurately identify user interactions with products and dynamically determine whether a product meets user needs. It can identify a user's basic interest in a product (through movement trajectory and dwell time) and accurately capture whether a user interacts with a product, thereby improving the accuracy and practicality of the recommendation system. Compared to traditional data input methods that rely on clicks or touchscreens, this approach based on image recognition and motion analysis can more seamlessly interact with users' natural behaviors, reducing the complexity of user operations while improving the smoothness and experience of interaction.

[0139] The specific steps of S5.3 are as follows:

[0140] S5.3.1: Based on the interactive image data captured by the image sensor, obtain interactive inter-frame images frame by frame, and perform differential denoising and background subtraction on the interactive inter-frame images to obtain optimized inter-frame images;

[0141] See also Figure 6 Flowchart of the differential denoising process for interactive inter-frame images in an embodiment of the present invention. This process is performed using Gaussian difference filtering, a commonly used image processing technique used to extract features and reduce noise from images. The core concept of Gaussian difference filtering is to filter the image using two Gaussian filters with different standard deviations and calculate the difference between the two to highlight edges and features in the image. For each pair of adjacent frame images, each frame is first Gaussian blurred, and then the difference between the two is calculated. This process effectively reduces low-frequency components in the image while retaining higher-frequency details and edge information, thereby enhancing the regional characteristics of user actions. In an embodiment of the present invention, Gaussian difference filtering is used to process the interactive frame images between the current frame and the subsequent frame. To capture image information at different scales, four layers of Gaussian filtering at different scales are used, each with a different standard deviation. By differencing the filtering results of each layer, the edges and motion features of the image at each scale can be highlighted, thereby obtaining richer feature information.

[0142] Extract feature information from the interactive inter-frame image between the current frame and the next frame using Gaussian difference filtering to obtain a four-layer two-dimensional Gaussian difference filtering interactive inter-frame image;

[0143] The four-layer two-dimensional Gaussian difference filtering interacting inter-frame image is segmented into four directions, and the high-frequency noise is filtered out of the segmented four-layer two-dimensional Gaussian difference filtering interacting inter-frame image. The four-layer two-dimensional Gaussian difference filtering interacting inter-frame image after denoising in four directions is fused to obtain the four-layer two-dimensional Gaussian difference filtering interacting inter-frame image after high-frequency noise removal.

[0144] Specifically, the four-layer two-dimensional Gaussian difference filtering interactive inter-frame image is segmented in four directions, each direction representing a different motion trend. Image segmentation can be achieved by calculating the gradient of the image and assigning segmentation regions based on the image's directionality. For example, the image can be segmented into four directions: up, down, left, and right. The image features in each direction are analyzed and processed separately. In the segmented image, the image in each direction is subjected to noise removal using a filter. High-frequency noise often appears in the details of the image. By smoothing the image in each direction and removing the noise in the details, the main motion features of the image are retained after directional segmentation and high-frequency noise removal, while reducing the impact of background noise and enhancing the recognizability of the target object (such as the user's skeleton).

[0145] The optimized inter-frame images are fused together using the two-dimensional Gaussian difference filtering after denoising each layer. These images contain more precise motion information, particularly in the user's upper limbs, skeleton, and other joints. The clarity and accuracy of these optimized inter-frame images are significantly improved, providing high-quality input for subsequent skeleton detection and pixel displacement calculations.

[0146] S5.3.2: Calculate pixel displacement vectors corresponding to pixels of the user in the optimized inter-frame image using an optical flow method, and determine the user's skeleton information based on the pixel displacement vectors. The optical flow method can estimate the temporal change of each pixel in the image by calculating the displacement of pixels in adjacent frame images.

[0147] S5.3.3: Skeletonize the skeleton information according to morphological operations to obtain a user skeleton graph, process the user skeleton graph according to adaptive graph convolution, and calculate key points of the user's upper limb joints;

[0148] Specifically, morphological operations are commonly used for shape extraction in image processing and can enhance the structural features of an image. In this step, morphological operations such as dilation, erosion, opening, and closing are used to enhance the skeleton information and make the skeleton image clearer. First, dilation is applied to expand the joint area, making the small connected areas of the skeleton more distinct. Erosion is then used to reduce unnecessary connected areas and remove noise. Opening (erosion followed by dilation) and closing (dilation followed by erosion) are used to further optimize the skeleton structure, removing small errors or breaks and ensuring the continuity of the skeleton image.

[0149] Furthermore, adaptive graph convolutional networks can adaptively process the relationships between different nodes in graph data. This is particularly well-suited for the irregular skeleton structures found in images. Adaptive graph convolution captures the spatial relationships between joints by applying convolution operations to the image skeleton graph. In the user's skeleton graph, each node represents a joint, and edges represent the relative positions or connections between joints. In this way, adaptive graph convolution can adaptively learn the spatial characteristics of each joint and their interactions, extracting more accurate keypoint information.

[0150] S5.3.4: Using the key points as input parameters of a preset gesture recognition network, training the input parameters through the gesture recognition network, and outputting a limb movement category;

[0151] The posture recognition network includes:

[0152] The posture feature embedding layer is used to embed the key points of the upper limb joints into a high-dimensional space through a normalized Gaussian function, model the upper limb joints in time series, calculate the relative position, angle change, and displacement vector, and obtain the upper limb change characteristics;

[0153] Specifically, the posture feature embedding layer is the first processing stage of the posture recognition network. It is primarily used to embed the key points of the upper limb joints into a high-dimensional space. By modeling these key points, the changing characteristics of the user's movements are captured. The key points of the upper limb joints are first embedded using a normalized Gaussian function. The normalized Gaussian function helps convert the original joint coordinates into position coordinates in a high-dimensional space through a certain standardization, making the spatial relationship of the joints more distinct and reducing changes caused by factors such as scale and posture. The joint motion is then modeled over time using a long-short-term memory neural network. This captures the temporal variation of the upper limb joints. The relative position changes between the joints reflect the movement of the upper limbs, and the angle changes between the joints capture the posture characteristics of the limb movement. The displacement vectors of the upper limb joints in the time series help capture more detailed movement patterns, especially movements such as arm extension and flexion, converting the user's upper limb movement information into a feature vector that encompasses both spatial and temporal dimensions.

[0154] The action classification layer is used to enhance and classify the upper limb change features through the adjacency matrix and mask, aggregate the upper limb change features according to the perceptron, perform confidence judgment on the classification results through the classifier, and output the confidence value of the binary classification result, wherein the binary classification result includes picking up the product and not picking up the product. The calculation formula of the confidence value of the binary classification result is:

[0155] ,

[0156] y represents the confidence value of the binary classification result, represents the activation function, represents the classifier weight, Represents the classifier bias term, t represents the image frame index, T represents the total number of image frames, A represents the adjacency matrix, M represents the mask, Indicates the upper limb change characteristics corresponding to the t-th frame image, represents the time series corresponding to the t-th frame image, represents feature fusion, represents a multi-layer perceptron network, represents the perceptron weight, i represents the image frame index, and satisfies , Indicates the upper limb change characteristics corresponding to the i-th frame image;

[0157] Specifically, the action classification layer aims to enhance and classify the features of the user's upper limb changes, ultimately determining whether the user has "picked up" or "not picked up" the item. The adjacency matrix is ​​used to describe the connections between joints. For example, the connection between the shoulder and elbow, or the connection between the elbow and wrist. The adjacency matrix helps the model capture the relationships between joints and guides the model to focus on changes between relevant joints. The mask matrix optimizes the network's learning process by screening important joints and suppressing irrelevant ones. The mask weights the relationships between joints to increase focus on specific joints. For example, in action recognition, the shoulder, elbow, and wrist are key joints. The mask increases the weight of these joints to enhance their influence. A multi-layer perceptron aggregates the features of each joint, such as displacement and angle change, to generate global motion features. The perceptron network combines these features using a nonlinear activation function to capture global information. For example, the perceptron layer aggregates the motion features of each joint through weighted aggregation to more accurately classify the overall action, and finally outputs a confidence value of the binary classification result through the activation function, and outputs the classification result based on the size of the confidence value of the binary classification result.

[0158] S5.3.5: Determine, based on the body movement category, whether the user has engaged in an interactive behavior of picking up the product;

[0159] For example, when a user stands in front of a shelf with a recommended product and is about to pick up the product, the user's upper limb movements are monitored and analyzed in real time to determine whether the user is "picking up the product." The following is a detailed description of the entire process:

[0160] First, the user's upper limb movements are captured using an image sensor. After the captured image is processed, the key points of the user's upper limb joints (shoulders, elbows, wrists, etc.) are extracted and their positions are recorded. For example, if a user stands and raises their right hand, the positions of the right shoulder, right elbow, and right wrist are identified and the coordinates of these joints are recorded.

[0161] After key point extraction, the relative position and angle changes between joints are calculated. This helps determine whether the user's upper limb is performing a specific action (such as raising or extending the hand). The position change of each joint relative to other joints is calculated. For example, when a user raises their arm from a naturally hanging position, the relative distance between the right shoulder and right elbow changes. The relative position between the elbow and wrist also changes with the arm's movement. The angle between the right shoulder and right elbow is calculated. When the user raises the arm, the elbow's bending angle decreases, while when the user extends the arm, the angle increases. This angle change reflects the arm's extension and flexion process.

[0162] When a user's arm begins to move, it's important not only to consider the joint positions in a single frame but also to track changes in the joints over time. For example, when a user moves from a natural position to picking up an item, the changes in the time series help determine whether it's a "picking up an item" action.

[0163] For example, as a user moves from looking down at a product to reaching out to pick it up, the system records the evolution of joint position and angle changes over time. For example, it can detect that the elbow angle gradually decreases, while the relative position of the shoulder and wrist increases until the user's hand touches the product.

[0164] The user's action type can be determined based on the position and angle changes of the joints, as well as the dynamic changes in the time series.

[0165] During the user's upper limb movements, if the changes in the wrist, elbow, and shoulder are detected to be consistent with the characteristics of "picking up a product" (for example, the wrist approaches the product, and the angle changes of the elbow and shoulder are consistent with the gesture of raising the hand), it will be inferred that the user is performing the action of "picking up a product."

[0166] Specifically, at the beginning of the user's action, it may be detected that the angle of the arm between the shoulder and the elbow contracts, and then the wrist gradually approaches the product. These changes indicate that the user is performing an action from being still to touching the product.

[0167] When a user reaches out to pick up an item placed on a shelf, the system continuously tracks the dynamic movements of the right shoulder, elbow, and wrist to accurately determine the arm's trajectory and identify whether the user has touched the item. These movements are manifested not only in space (changes in joint position) but also in the timing of the movement (speed of arm movement, speed of angle change, etc.).

[0168] S5.4: Determine whether the output meets user needs;

[0169] In this embodiment, the system integrates the user's spatial movement characteristics and interactive behavior data to ultimately determine whether a product meets the user's needs. If the system determines that the user moves toward a recommended product area and demonstrates interest (e.g., by staying there for a long time or picking up the product), the product is deemed to meet the user's needs and remains in the recommended list. If the system determines that the user does not show interest or does not pick up the product, the system updates the recommended product list based on a real-time feedback mechanism, removing the currently recommended product and recalculating new recommendations.

[0170] Specifically, by integrating multiple feedback factors (such as spatial movement, dwell time, and behavioral interactions), product recommendations can be dynamically adjusted, significantly improving the accuracy of recommendations. This not only addresses the problem of traditional recommendation systems often failing to quickly adapt to changing user needs, but also effectively avoids incorrect recommendations, optimizes the user experience, and increases purchase conversion rates. Through a dynamic recommendation update mechanism, users receive product recommendations that best meet their current needs every time they interact with the system. This is all achieved through seamless system integration and real-time feedback.

[0171] The specific steps of S6 are as follows:

[0172] S6.1: Identify and mark the currently advertised product as a rejected product;

[0173] S6.2: Use a similarity matching algorithm to retrieve products in the product recommendation list that are similar to the rejected products in terms of attributes and categories, and obtain a similar product set;

[0174] In this embodiment, when a user explicitly rejects a product, not only is the product removed from the recommendation list, but similar or similar products are also prevented from being recommended to the user again. This improves the diversity and innovation of recommendations, preventing users from repeatedly receiving similar, inappropriate recommendations over multiple interactions. By precisely matching product features, we can better understand user preferences and further enhance the personalization of product recommendations.

[0175] Specifically, based on product attributes and categories, a similarity matching algorithm is used to retrieve products from the recommendation list that are similar to the rejected product. This similarity matching algorithm employs a content-based recommendation approach, first analyzing the basic attributes of a product (such as category, brand, color, price, etc.) as well as other features (such as function, size, and material). These features can be extracted from structured data in the product database. Next, based on the characteristics of the rejected product, similar products are searched and selected. The core technology of this step lies in the feature vector representation between products. By calculating the similarity between products, it is possible to accurately identify those products that are most similar to the rejected product in terms of attributes and categories.

[0176] S6.3: Remove the currently reported product and the set of similar products from the product recommendation list;

[0177] In this embodiment, the similarity set of products obtained through the similarity matching algorithm (i.e., the set of products similar to the rejected products) is removed from the recommendation list. This removal process is not simply marking the products as not recommended, but also involves updating the recommendation algorithm itself.

[0178] Specifically, the current recommendation list needs to be reordered, and a dynamic filtering mechanism is used to ensure the diversity and relevance of the remaining items in the recommendation list after removing items. To avoid gaps in the recommendation list, the vacancies left by removed items are filled based on factors such as user history, product popularity, and inventory information, ensuring that the recommended content maintains high quality after the update.

[0179] Furthermore, gaps are filled based on the product’s category distribution and thematic diversity. For example, if a pair of sneakers is removed from the list and there are too many products in the sports shoe category, other categories (such as casual shoes or running accessories) are preferred to fill the gaps rather than simply recommending more sports shoes, thus maintaining the richness of the product variety.

[0180] Furthermore, the diversity of recommended content is achieved through a regularized ranking algorithm based on product features. Specifically, by analyzing the user's interest graph, the product recommendations are diverse, avoiding recommending products of only one type or brand. For example, if a user has previously browsed for sneakers, sports equipment, and health-related products, products that match these interests will be prioritized, while maintaining a sufficient distribution of categories and brands to accommodate the user's potentially diverse needs.

[0181] Furthermore, when updating the recommendation list, in addition to traditional behavior-based ranking, collaborative filtering and content filtering methods are combined. Collaborative filtering recommends products based on the behavioral patterns of user groups, such as recommending products favored by other users who have purchased the same product. Content filtering makes recommendations based on product attributes (such as category, brand, price, size, etc.). By combining these two methods, we can more accurately generate a list of user-preferred products and ensure that the recommended list still meets the user's interests after eliminating items.

[0182] The unmanned retail interaction method further includes:

[0183] Generate a product rejection list based on the rejected products;

[0184] The product elimination list is used as a constraint condition to update the hierarchical reinforcement learning framework of the multimodal fusion neural network model.

[0185] In this embodiment, the present application also includes an update of the hierarchical reinforcement learning framework based on the multimodal fusion neural network model, which is mainly used to optimize the product recommendation list in real time and accurately adapt to user needs.

[0186] Specifically, the hierarchical reinforcement learning framework updates its recommendation strategy by continuously learning and adapting to user behavior. When the system detects that a user has rejected and removed a product, the removed product information is fed back to the model as input to correct the model weights. Through the reinforcement learning framework's learning mechanism, the system dynamically adjusts its recommendation strategy and optimizes the order in which recommended products are selected. For example, when a user rejects a category of products (such as sneakers), the system adjusts the model parameters to lower the priority of such products in future recommendations. The product removal list serves as a constraint, helping the reinforcement learning framework avoid selecting rejected or similar products in the next round of recommendations, ensuring the novelty and diversity of recommendation results, thereby better meeting user needs.

[0187] Furthermore, through the constraints of the product elimination list, the recurrence of products that users have already rejected is dynamically avoided, reducing the redundancy of product recommendations. Once the eliminated products enter the product elimination list, the next recommendation will prioritize products that highly match the user's interests, rather than relying on old, outdated recommendation data or static rule-based recommendation strategies. At the same time, through the continuous feedback mechanism of the reinforcement learning framework, the recommendation strategy can be gradually improved based on user feedback, making each recommendation more accurate and able to quickly respond to changes in user needs.

[0188] Example 2:

[0189] See also Figure 7, the present invention provides an embodiment: an unmanned retail interactive system combining AI voice shopping guide and cloud-based on-call service, the system comprising a data acquisition module, a product recommendation module, and a recommendation feedback module;

[0190] The data acquisition module is used to collect the user's voice, visual behavior and environmental data in real time through various sensor components to generate user demand information;

[0191] The product recommendation module is used to generate a personalized product recommendation list through a multimodal fusion neural network model based on the user demand information provided by the data acquisition module;

[0192] The recommendation feedback module is used to receive and analyze user behavior feedback information, update the product recommendation list according to the behavior feedback information, and remove rejected products and similar products.

[0193] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service is characterized by: The unmanned retail interaction method includes: Inputting the acquired user demand information into a multimodal fusion neural network model, the multimodal fusion neural network model outputting a product recommendation list, the product recommendation list including a first area, a second area, and a third area arranged in sequence, each area including a plurality of sequentially arranged product information; Performing an iterative operation, the iterative operation including swapping the positions of the products currently displayed in the first area and the third area, determining the area where the user's favorite products are located based on the change range projected onto the product recommendation list display interface by the user's field of view, and updating the product information in the area to the current product recommendation list until the number of products in the product recommendation list falls below a set threshold; Play the first product in the product recommendation list through the AI ​​shopping guide voice, and confirm whether the product information currently being broadcast meets the user's needs based on the user's behavioral feedback after listening to the AI ​​shopping guide voice; If not, update the product recommendation list and select the first updated product-related information from the product recommendation list until the user's needs are met; The user demand information includes voice intention information, visual feedback information and environmental perception data, and the multimodal fusion neural network model includes: A multimodal input layer, configured to extract acoustic features, facial emotion features, and an environmental context vector, wherein the multimodal input layer includes a speech branch, a visual branch, and an environmental branch; A cross-modal alignment layer, configured to dynamically align the outputs of the multimodal input layer using a multi-head attention mechanism to generate a joint embedding representation, and to update the embedding weights of the joint embedding representation based on the modal confidence calculated by the multi-head attention mechanism; The intention reasoning layer is used to input the joint embedding representation into a preset hierarchical reinforcement learning framework to generate a product recommendation list, wherein the hierarchical reinforcement learning framework includes an upper-layer policy network and a lower-layer policy network, wherein the upper-layer policy network is used to generate a product candidate pool based on the user's historical interaction data, product inventory and environmental factors, and the lower-layer policy network is used to screen the product candidate pool according to the user's interests, generate a product recommendation list, and divide the product recommendation list into a first area, a second area and a third area according to product categories.

2. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 1 is characterized in that: The speech branch is used to extract acoustic features in the speech intention information according to a pre-trained speech encoder; The visual branch is used to extract temporal features from the visual feedback information based on the dual-stream network, locate the user's visual focus area, analyze the visual feedback information through a lightweight convolutional network, and assign weighted values ​​to the visual focus area based on the analysis results to obtain facial emotion features; The environment branch is used to process the environment perception data according to the spatiotemporal convolutional network to construct an environment context vector.

3. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 1 is characterized in that: The determining, based on the changing range of the user's field of view projected onto the product recommendation list display interface, of the area where the product of interest to the user is located includes: Get the user's head posture information; Calculating the user's gaze direction based on the head posture information; The user's gaze direction is mapped to the display interface of the product recommendation list through perspective projection transformation to generate a field of view projection area, wherein the field of view projection area is cross-calculated with the product area coordinates on the product recommendation list to determine whether the product the user is looking at belongs to the first area or the third area; Aggregate the user's visual field projection area within a fixed time window to determine the area where the products the user is interested in are located.

4. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 3 is characterized in that: Determining the area where the product of interest to the user is located includes: Obtain the visual field projection trajectory and the visual field projection trajectory before the product position is swapped based on the aggregation result; If the user's field of view projection trajectory shifts in the swapped product area, and the new field of view projection area is concentrated in a specific area, an update of the product recommendation list is triggered based on the specific area; If the user's visual field projection trajectory remains stable before and after the product positions are swapped and does not deviate from the changes in the first area and the third area, an update of the product recommendation list is triggered based on the second area.

5. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 1 is characterized in that: The confirmation of whether the currently reported product-related information meets the user's needs includes: Collect the user's movement trajectory data after listening to the AI ​​shopping guide voice and extract the user's spatial movement characteristics; Determine whether the user moves to the area where the recommended product is located based on the spatial movement characteristics; if not, it indicates that the user's needs are not met; If yes, acquiring interaction image data of the user and the product shelf through an image sensor arranged in the area where the recommended product is located, processing the interaction image data, and determining whether the user has performed an interaction behavior of picking up the recommended product; The output is judged to determine whether it meets the user's needs.

6. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 5 is characterized in that: The processing of the interactive image data includes: According to the interactive image data captured by the image sensor, interactive inter-frame images are acquired frame by frame, and differential denoising and background subtraction are performed on the interactive inter-frame images to acquire optimized inter-frame images; Calculating pixel displacement vectors corresponding to user pixels in the optimized inter-frame image by an optical flow method, and determining user skeleton information based on the pixel displacement vectors; Skeletonize the skeleton information according to morphological operations to obtain a user skeleton graph, process the user skeleton graph according to adaptive graph convolution, and calculate key points of the user's upper limb joints; Using the key points as input parameters of a preset posture recognition network, training the input parameters through the posture recognition network, and outputting a body movement category; It is determined whether the user performs an interactive behavior of picking up the product based on the body movement category.

7. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 1 is characterized in that: The updating of the product recommendation list includes: Identify and mark the currently advertised product as a rejected product; Use a similarity matching algorithm to retrieve products in the product recommendation list that are similar to the rejected products in terms of attributes and categories, and obtain a similar product set; Remove the currently reported product and similar product sets from the product recommendation list.

8. The unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service according to claim 7 is characterized in that: The unmanned retail interaction method further includes: Generate a product rejection list based on the rejected products; The product elimination list is used as a constraint condition to update the hierarchical reinforcement learning framework of the multimodal fusion neural network model.

9. An unmanned retail interaction system combining AI voice shopping guide and cloud-based on-call service, for implementing the unmanned retail interaction method combining AI voice shopping guide and cloud-based on-call service as described in any one of claims 1 to 8, characterized in that: The system includes a data collection module, a product recommendation module and a recommendation feedback module; The data acquisition module is used to collect the user's voice, visual behavior and environmental data in real time through various sensor components to generate user demand information; The product recommendation module is used to generate a product recommendation list through a multimodal fusion neural network model based on the user demand information provided by the data acquisition module; The recommendation feedback module is used to receive and analyze user behavior feedback information, update the product recommendation list according to the behavior feedback information, and remove rejected products and similar products.

Images