AI virtual human shopping guide system and method based on facial expression recognition

Abstract

The invention relates to the technical field of expression recognition, and particularly discloses an AI virtual human shopping guide system and method based on facial expression recognition, and the system comprises a face data obtaining module which is used for obtaining a face image sequence in the interaction process of a shopping guide target user and an AI virtual human; the expression information recognition module is used for recognizing facial expressions in the interaction process of the shopping guide target user and the AI virtual human based on the face image sequence to obtain real-time facial expression information of the shopping guide target user, and the real-time facial expression information comprises a real-time facial expression type of the shopping guide target user and a corresponding real-time expression index; the shopping guide information generation module is used for generating latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI virtual human; the method and the device are used for improving the interactive experience of a user and an AI virtual human, enhancing the satisfaction degree and the acceptability of the user for shopping guide services, and improving the sales conversion rate and the service quality.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to an AI virtual human shopping guide system and method based on facial expression recognition. Background Art

[0002] At present, with the rapid development of artificial intelligence technology, virtual human shopping guide systems have gradually become an important application direction in the business field. In e-commerce platforms, offline stores and other scenarios, in order to provide consumers with more intimate and personalized services, AI virtual human shopping guide systems based on facial expression recognition have emerged. At present, such systems usually use cameras and other devices to collect users' facial image data, and analyze these data through certain algorithms and models, trying to understand users' emotions and needs, so as to provide users with shopping guide suggestions that are more in line with their psychological expectations.

[0003] In terms of technical implementation, existing facial expression recognition technologies are mostly based on traditional image processing and machine learning methods. These methods can recognize some basic expression categories, such as happiness, sadness, anger, etc. to a certain extent. However, the existing virtual human shopping guide system can only simply recognize the user's expression type, but cannot accurately recognize the degree of expression, and the interactive coherence analysis performance of shopping guide information is poor, resulting in low user satisfaction and acceptance of shopping guide services, and low sales conversion rate and service quality.

[0004] Therefore, the present invention proposes an AI virtual human shopping guide system and method based on facial expression recognition. Summary of the invention

[0005] The present invention provides an AI virtual human shopping guide system and method based on facial expression recognition, which can acquire facial image sequences by using a facial data acquisition module, provide a data basis for subsequent analysis of user expressions, and ensure the integrity and accuracy of information. The facial expression information recognition module accurately recognizes the user's real-time facial expression and expression index by processing the facial image sequence, which can not only identify the user's expression type, but also the degree of expression of the user's expression type, which helps to deeply understand the user's emotions and reactions. Based on the accurate recognition of the user's real-time facial expression and combined with the latest shopping guide information given by the AI ​​virtual person, the shopping guide information generation module can generate the latest shopping guide information that is more in line with the user's current emotions and needs, improve the coherent analysis performance of the shopping guide information in the interactive process, and improve the pertinence and effectiveness of the shopping guide. It is helpful to improve the interactive experience between the user and the AI ​​virtual person, and enhance the user's satisfaction and acceptance of the shopping guide service. The shopping guide strategy can be adjusted in time according to the user's real-time feedback, and the sales conversion rate and service quality can be improved.

[0006] The present invention provides an AI virtual human shopping guide system based on facial expression recognition, comprising: The facial data acquisition module is used to obtain the facial image sequence during the interaction between the shopping guide target user and the AI ​​virtual person; The facial expression information recognition module is used to recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; The shopping guide information generation module is used to generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

[0007] Preferably, the facial data acquisition module includes: The monitoring video acquisition submodule is used to obtain real-time monitoring video during the interaction between the user and the AI ​​virtual person; The facial image capture submodule is used to capture all complete facial images of the shopping guide target user in the real-time monitoring video; The image sequence generation submodule is used to generate a facial image sequence of the shopping guide target user based on all the captured complete facial images of the shopping guide target user.

[0008] Preferably, the facial image capture submodule includes: A shopping guide target user extraction unit is used to extract all head images from the real-time monitoring video, determine whether there is a video frame containing more than one head image in the real-time monitoring video, and if so, determine a complete limb image sequence of each head image in the corresponding video frame; otherwise, extract the head image contained in the latest video frame in the real-time monitoring video as the head image of the shopping guide target user; A shopping guide target user head image determination unit, used to determine the shopping guide target user head image based on a complete limb image sequence of each head image in a video frame containing more than one head image in the real-time monitoring video; A head image extraction unit, used to extract all head monitoring images of the shopping guide target user's head image from the real-time monitoring video based on the head morphological features of the shopping guide target user's head image; The facial image extraction unit is used to extract all complete facial images of the shopping guide target user from all head monitoring images of the shopping guide target user's head images.

[0009] Preferably, the shopping guide target user head image determination unit includes: Other information acquisition subunits are used to acquire touch screen interaction record sequences and online interaction record sequences in application scenarios; a transaction dominance index determination subunit, for evaluating a transaction dominance index of each head image in a video frame containing more than one head image in a real-time monitoring video based on a touch screen interaction record sequence and an online interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video; The shopping guide target user head image determination subunit is used to use the head image with the largest transaction dominance index as the shopping guide target user head image.

[0010] Preferably, the transaction leading index determination subunit includes: A touch screen behavior matching degree determination terminal is used to input a touch screen interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video into a touch screen behavior matching degree analysis model to obtain a touch screen behavior matching degree of each head image in a video frame containing more than one head image in a real-time monitoring video; An online interaction behavior matching degree determination terminal is used to input the online interaction record sequence in the application scenario and the complete limb image sequence of each head image in the video frame containing more than one head image in the real-time monitoring video into the online interaction behavior matching degree analysis model to obtain the online interaction behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video; The transaction dominance index determination terminal is used to use the sum of the touch screen behavior matching degree and the online interactive behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video as the transaction dominance index of each head image in the video frame containing more than one head image in the real-time monitoring video.

[0011] Preferably, the expression information recognition module includes: The expression type recognition submodule is used to analyze the real-time facial expression type of the shopping guide target user based on the facial image sequence; The expression index recognition submodule is used to determine the real-time expression index of the real-time facial expression type of the shopping guide target user based on the current facial image sequence and the historical facial image sequence of the shopping guide target user.

[0012] Preferably, the performance index identification submodule comprises: A correction factor determination unit, used to input a historical facial image sequence of a shopping guide target user into a facial expression macro-index relative correction factor determination model to obtain a facial expression macro-index relative correction factor of the shopping guide target user; A macro index determination unit, used to input the current facial image sequence into the expression expression macro index determination model to obtain the expression expression macro index of the shopping guide target user; The expression index determination unit is used to use the sum of the expression expression macro index and the expression expression macro index relative correction factor of the shopping guide target user as the real-time expression index of the real-time facial expression type of the shopping guide target user.

[0013] Preferably, the shopping guide information generating module includes: The feedback interpretation submodule is used to evaluate the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person based on the user's real-time facial expression information; The shopping guide information generation submodule is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the shopping guide information latest given by the AI ​​virtual person and the shopping guide information latest given by the AI ​​virtual person.

[0014] Preferably, the shopping guide information generation submodule includes: A first shopping guide information generating unit is configured to generate the latest shopping guide information based on the latest touch screen feedback information and / or voice feedback information given by the user, the feedback type and corresponding feedback performance index of the user to the latest shopping guide information given by the AI ​​virtual person, and the latest shopping guide information given by the AI ​​virtual person when the user gives the latest touch screen feedback information and / or voice feedback information; The second shopping guide information generating unit is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person and the latest shopping guide information given by the AI ​​virtual person when the user has not given the latest touch screen feedback information and / or voice feedback information.

[0015] The present invention provides an AI virtual human shopping guide method based on facial expression recognition, which is applied to any of the above AI virtual human shopping guide systems based on facial expression recognition, comprising: S1: Obtain facial image sequences during the interaction between the shopping guide target user and the AI ​​virtual person; S2: Recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; S3: Generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

[0016] The beneficial effects of the present invention compared with the prior art are as follows: the facial data acquisition module can be used to obtain facial image sequences, providing a data basis for the subsequent analysis of user expressions, ensuring the integrity and accuracy of the information. The facial expression information recognition module uses the facial image sequence to accurately identify the user's real-time facial expression and expression index, which can not only identify the user's expression type, but also the degree of expression of the user's expression type, which helps to deeply understand the user's emotions and reactions. Based on the accurate recognition of the user's real-time facial expression and combined with the latest shopping guide information given by the AI ​​virtual person, the shopping guide information generation module can generate the latest shopping guide information that is more in line with the user's current emotions and needs, improve the coherent analysis performance of the shopping guide information in the interactive process, and improve the pertinence and effectiveness of the shopping guide. It helps to improve the interactive experience between users and AI virtual people, and enhance users' satisfaction and acceptance of shopping guide services. It can adjust the shopping guide strategy in time according to the user's real-time feedback, and improve the sales conversion rate and service quality.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures specifically pointed out in this application document.

[0018] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 Schematic diagram of an AI virtual human shopping guide system based on facial expression recognition in an embodiment of the present invention; Figure 2 The figure is a flow chart of an AI virtual human shopping guide method based on facial expression recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0021] Example 1 The present invention provides an AI virtual human shopping guide system based on facial expression recognition, referring to Figure 1 ,include: The facial data acquisition module is used to obtain the facial image sequence during the interaction between the shopping guide target user and the AI ​​virtual person; The facial expression information recognition module is used to recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; The shopping guide information generation module is used to generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

[0022] In this embodiment, the shopping guide target user refers to the customer who is interacting with the AI ​​virtual person and has the intention to buy the product. He is also the main target user of this shopping guide promotion.

[0023] In this embodiment, the facial image sequence is a series of facial images of the shopping guide target user taken continuously.

[0024] In this embodiment, the real-time facial expression type of the shopping guide target user refers to the specific expression category presented by the shopping guide target user at the moment of interacting with the AI ​​virtual person, such as happiness, surprise, confusion, etc.

[0025] In this embodiment, the real-time expression index of a real-time facial expression type is used to quantify the value of the obviousness or intensity of a certain real-time facial expression type. For example, if the customer's surprised expression is very strong, the corresponding real-time expression index of the surprised expression may be higher.

[0026] In this embodiment, the latest shopping guide information provided by the AI ​​virtual person refers to the information about the introduction, recommendation, promotional activities, etc. of the product that the AI ​​virtual person has just provided to the shopping guide target user.

[0027] For example: The AI ​​virtual person said, "This drink is on sale, it's a good deal."

[0028] In this embodiment, the latest shopping guide information is regenerated based on the real-time facial expression information of the shopping guide target user and the shopping guide information previously given by the AI ​​virtual person, and the shopping guide content is more in line with the user's current status and needs.

[0029] For example: When a user shows a puzzled expression, the latest shopping guide information may be a more detailed explanation of the previously recommended product.

[0030] The beneficial effects of the above technology are as follows: the facial data acquisition module can be used to obtain facial image sequences, providing a data basis for the subsequent analysis of user expressions, ensuring the integrity and accuracy of the information. The facial expression information recognition module uses the facial image sequence to accurately identify the user's real-time facial expression and expression index, which can not only identify the user's expression type, but also the degree of expression of the user's expression type, which helps to deeply understand the user's emotions and reactions. Based on the accurate recognition of the user's real-time facial expression and combined with the latest shopping guide information given by the AI ​​virtual person, the shopping guide information generation module can generate the latest shopping guide information that is more in line with the user's current emotions and needs, improve the coherent analysis performance of the shopping guide information in the interactive process, and improve the pertinence and effectiveness of the shopping guide. It helps to improve the interactive experience between users and AI virtual people, and enhance users' satisfaction and acceptance of shopping guide services. It can adjust the shopping guide strategy in time according to the user's real-time feedback, and improve the sales conversion rate and service quality.

[0031] Example 2 Based on the first embodiment, the facial data acquisition module includes: The monitoring video acquisition submodule is used to obtain real-time monitoring video during the interaction between the user and the AI ​​virtual person; The facial image capture submodule is used to capture all complete facial images of the shopping guide target user in the real-time monitoring video; The image sequence generation submodule is used to generate a facial image sequence of the shopping guide target user based on all the captured complete facial images of the shopping guide target user.

[0032] In this embodiment, the complete facial image in the context of a vending machine refers to a facial image that can clearly and comprehensively display the entire facial features of the shopping guide target user without being blocked or missing important parts.

[0033] For example, the customer's face captured by the camera in front of the vending machine includes the eyes, nose, and mouth, and no key parts are blocked by items such as hats and masks.

[0034] The beneficial effects of the above technology are as follows: the monitoring video acquisition submodule obtains real-time monitoring video, which can fully record the interaction process and ensure the richness and integrity of the data source. The facial image capture submodule captures the complete facial image from the real-time monitoring video, which improves the accuracy and pertinence of the data and eliminates irrelevant interference. The image sequence generation submodule generates a sequence based on the captured facial image, providing orderly and coherent data support for subsequent expression recognition. It helps to improve the efficiency and quality of facial data acquisition and lay a good foundation for accurately analyzing user expressions. It can provide more reliable and valuable user information for subsequent shopping guide services, and improve the shopping guide effect and user experience.

[0035] Example 3 Based on Example 2, the facial image capture submodule includes: A shopping guide target user extraction unit is used to extract all head images from the real-time monitoring video, determine whether there is a video frame containing more than one head image in the real-time monitoring video, and if so, determine a complete limb image sequence of each head image in the corresponding video frame; otherwise, extract the head image contained in the latest video frame in the real-time monitoring video as the head image of the shopping guide target user; A shopping guide target user head image determination unit, used to determine the shopping guide target user head image based on a complete limb image sequence of each head image in a video frame containing more than one head image in the real-time monitoring video; A head image extraction unit, used to extract all head monitoring images of the shopping guide target user's head image from the real-time monitoring video based on the head morphological features of the shopping guide target user's head image; The facial image extraction unit is used to extract all complete facial images of the shopping guide target user from all head monitoring images of the shopping guide target user's head images.

[0036] In this embodiment, the head image in the vending machine scenario refers to an image that only contains the user's head portion in the real-time surveillance video.

[0037] In this embodiment, the complete limb image sequence of the head image is a series of continuous image sets including the head and other limb parts (such as shoulders, limbs, torso, etc.).

[0038] In this embodiment, the head image of the shopping guide target user is a head image of the shopping guide target user. For example, in a group of people gathered in front of a vending machine, a head image of a user who is communicating with an AI virtual person and preparing to buy.

[0039] In this embodiment, based on the head morphological features of the shopping guide target user's head image, all head monitoring images of the shopping guide target user's head image are extracted from the real-time monitoring video: based on the shape, size, contour and other features of the shopping guide target user's head that have been determined, all monitoring images containing the user's head are found from the real-time monitoring video.

[0040] For example: By identifying the unique features of the target shopping guide user's head, such as hairstyle, head size, etc., all head shots with this feature are filtered out from the video.

[0041] In this embodiment, all complete facial images of the shopping guide target user are extracted from all head monitoring images of the shopping guide target user's head. From the screened monitoring images containing the shopping guide target user's head, further images are found that can clearly show the user's entire face (including eyes, nose, mouth, etc.) and are not blocked.

[0042] For example: From a series of images containing the user's head, select those images in which the user's facial features can be fully seen and are not blocked by hands or other objects.

[0043] The beneficial effects of the above technology are as follows: the shopping guide target user extraction unit can handle complex situations involving multiple head images, accurately judge and extract possible user head images, and improve the accuracy of target user identification. The shopping guide target user head image determination unit determines the target user head image based on the complete limb image sequence, which increases the reliability and accuracy of the judgment. The head image extraction unit extracts the head monitoring image based on the head morphological features, which improves the accuracy and pertinence of data extraction. The facial image extraction unit further accurately extracts the complete facial image from the head monitoring image, providing high-quality data for subsequent expression analysis. The process of the entire facial image capture submodule helps to improve the accuracy and completeness of facial image capture, and provides strong support for accurately analyzing user expressions and generating effective shopping guide information.

[0044] Example 4 On the basis of Embodiment 3, the shopping guide target user head image determination unit includes: Other information acquisition subunits are used to acquire touch screen interaction record sequences and online interaction record sequences in application scenarios; a transaction dominance index determination subunit, for evaluating a transaction dominance index of each head image in a video frame containing more than one head image in a real-time monitoring video based on a touch screen interaction record sequence and an online interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video; The shopping guide target user head image determination subunit is used to use the head image with the largest transaction dominance index as the shopping guide target user head image.

[0045] In this embodiment, the application scenario: in the context of a vending machine, refers to the specific environment and conditions in which a user interacts with the vending machine and the AI ​​virtual person therein and generates a transaction.

[0046] In this embodiment, the touch screen interaction record sequence is a series of records of user operations on the touch screen of the vending machine.

[0047] For example: including records such as the order in which users click on products and the number of times they view product details.

[0048] In this embodiment, the online interaction record sequence refers to a series of records of the user communicating and operating with the vending machine-related system through the network.

[0049] For example, a user browses the records of vending machine products on a mobile phone app.

[0050] In this embodiment, the transaction dominance index of the head image is used to measure the numerical indicator of the dominance or participation degree of the user corresponding to each head image in the transaction process when there are multiple head images.

[0051] For example: If the user corresponding to a head image has more purchase intention operations on the touch screen and online, its transaction dominance index will be higher.

[0052] The beneficial effects of the above technology are as follows: other information acquisition subunits acquire touch screen interaction record sequences and online interaction record sequences, enriching the information sources used for judgment and improving the comprehensiveness of the analysis. The transaction dominance index determination subunit evaluates the transaction dominance index by integrating multiple records and body image sequences, making the judgment more scientific and objective. It can accurately determine the head image of the shopping guide target user with the largest transaction dominance index among multiple head images, thereby improving the accuracy of target user identification. It helps to focus on key users more effectively and provide accurate objects for subsequent expression analysis and shopping guide information generation. It improves the ability of the entire system to accurately identify target users in complex interaction scenarios, thereby improving the pertinence and effectiveness of shopping guide services.

[0053] Example 5 Based on Example 4, the transaction leading index determination subunit includes: A touch screen behavior matching degree determination terminal is used to input a touch screen interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video into a touch screen behavior matching degree analysis model to obtain a touch screen behavior matching degree of each head image in a video frame containing more than one head image in a real-time monitoring video; An online interaction behavior matching degree determination terminal is used to input the online interaction record sequence in the application scenario and the complete limb image sequence of each head image in the video frame containing more than one head image in the real-time monitoring video into the online interaction behavior matching degree analysis model to obtain the online interaction behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video; The transaction dominance index determination terminal is used to use the sum of the touch screen behavior matching degree and the online interactive behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video as the transaction dominance index of each head image in the video frame containing more than one head image in the real-time monitoring video.

[0054] In this embodiment, the touch screen behavior matching analysis model is a model used to analyze the matching degree between the body movements of a single head image in a real-time monitoring video and a touch screen interaction record sequence.

[0055] For example, determining the degree of compatibility between the hand movements of a user corresponding to a head image and operations such as clicking and sliding on a touch screen.

[0056] The training samples of the touch screen behavior matching analysis model may include: A large number of video clips of users operating on the touch screen of vending machines, including the user's head image and the corresponding touch screen operation records; These samples are annotated with different degrees of touch screen behavior matching, such as complete match, partial match, no match, etc.

[0057] For example, for a touch screen behavior matching analysis model, a training sample may be a user continuously clicking on several items on the touch screen of a vending machine, while the user's head image is clearly visible in the surveillance video and is labeled as a "highly matching" touch screen behavior.

[0058] In this embodiment, the touch screen behavior matching degree of the head image refers to a quantitative value of the degree of matching between the body movements of the user represented by the specific head image and the touch screen interaction behavior.

[0059] For example: If a user looks at the screen and quickly clicks on multiple products with his hands, the touch screen behavior match of his head image is high.

[0060] In this embodiment, the online interaction behavior matching degree analysis model is a model for evaluating the matching degree between the body movements of the head image in the real-time monitoring video and the online interaction record sequence.

[0061] Example: Analyze the correlation between the user's operation on the online platform of a vending machine and the user's hand movements.

[0062] The training samples of the online interactive behavior matching analysis model may include: Records of users’ online interactions with vending machine-related systems (e.g., browsing products and checking promotions through mobile apps), combined with images of users’ heads in real-time surveillance video; These samples are also labeled with different levels of online interaction behavior matching.

[0063] For example, for the online interactive behavior matching analysis model, the training samples may be users viewing information about a certain type of product on a mobile phone APP for a long time, and the corresponding head image in the surveillance video, which is marked as an online interactive behavior of "moderate match".

[0064] In this embodiment, the online interactive behavior matching degree of the head image represents a numerical value of the matching degree between the action of the user corresponding to the specific head image and the online interactive behavior.

[0065] For example: If a user corresponding to a head image has a lot of online searches for specific products while looking at the mobile phone screen, the matching degree of his online interactive behavior is higher.

[0066] The beneficial effects of the above technology are as follows: the touch screen behavior matching degree determination end can accurately measure the matching degree of each head image and the touch screen interaction behavior by inputting data into the touch screen behavior matching degree analysis model. The online interaction behavior matching degree determination end uses a similar method to determine the online interaction behavior matching degree, which comprehensively covers different types of interaction behaviors. The transaction dominance index determination end comprehensively considers the matching degree of touch screen and online interaction behaviors to determine the transaction dominance index, making the index more comprehensive and accurate. It helps to more accurately evaluate the dominant position of each head image in the transaction, thereby more accurately identifying the target users of shopping guides. It provides a key and reliable basis for the subsequent generation of shopping guide information that better meets the needs of target users, and improves the quality and effectiveness of shopping guide services.

[0067] Example 6 Based on Example 1, the expression information recognition module includes: The expression type recognition submodule is used to analyze the real-time facial expression type of the shopping guide target user based on the facial image sequence; The expression index recognition submodule is used to determine the real-time expression index of the real-time facial expression type of the shopping guide target user based on the current facial image sequence and the historical facial image sequence of the shopping guide target user.

[0068] In this embodiment, the real-time facial expression type of the shopping guide target user is analyzed based on the facial image sequence: this means that by analyzing and judging a series of continuously shot facial images of the shopping guide target user, the specific expression category of the user at the moment, such as happiness, sadness, surprise, etc., is determined.

[0069] For example: From continuous facial images, it is found that the user's mouth corners are raised and the eyes are narrowed, so the real-time facial expression type is judged to be happy.

[0070] In this embodiment, the historical facial image sequence of the shopping guide target user refers to a series of facial images of the shopping guide target user in the process of interacting with the vending machine and the AI ​​virtual person, which have been acquired and stored before the current interaction.

[0071] For example, a collection of facial images taken when a user shopped at this vending machine multiple times before.

[0072] The beneficial effects of the above technology are as follows: the expression type recognition submodule can accurately identify the real-time facial expression type of the shopping guide target user by analyzing the facial image sequence, providing a basis for understanding the user's emotional state. The expression index recognition submodule combines the current and historical facial image sequences to determine the real-time expression index, making the assessment of the user's expression intensity more comprehensive and accurate. It helps to more deeply understand the degree of emotional response of the user during the interaction process, thereby providing a key basis for providing personalized shopping guide services. It can capture the changing trend of user emotions more timely and provide real-time guidance for adjusting shopping guide strategies. It improves the accuracy and reliability of expression information recognition, thereby enhancing the pertinence and adaptability of shopping guide services.

[0073] Example 7 Based on Example 6, the performance index identification submodule includes: A correction factor determination unit, used to input a historical facial image sequence of a shopping guide target user into a facial expression macro-index relative correction factor determination model to obtain a facial expression macro-index relative correction factor of the shopping guide target user; A macro index determination unit, used to input the current facial image sequence into the expression expression macro index determination model to obtain the expression expression macro index of the shopping guide target user; The expression index determination unit is used to use the sum of the expression expression macro index and the expression expression macro index relative correction factor of the shopping guide target user as the real-time expression index of the real-time facial expression type of the shopping guide target user.

[0074] In this embodiment, the expression expression macro index relative correction factor determination model is an algorithm or model for calculating the relevant factor for correcting the expression expression macro index based on the historical facial image sequence of the shopping guide target user, and its training samples include: Multiple sets of historical facial image sequences of the same user at different times and scenes; Each set of historical facial image sequences needs to be labeled with a corresponding correction factor, and the labeled value may be determined based on factors such as the user's facial expression habits, general reactions in specific scenarios, etc.

[0075] For example, the training samples may be a set of continuous facial images of users in front of a vending machine when they see different product recommendations, and the corresponding macro index or correction factor of facial expression is annotated. Such a set of samples is learned and trained by the model, so that it can accurately calculate the corresponding index or correction factor based on the input facial image sequence.

[0076] For example, by analyzing the changing patterns of a user's facial expressions during their past shopping trips, a computational model for adjusting the current macro index of facial expression can be derived.

[0077] In this embodiment, the relative correction factor of the macro index of expression expression is a value calculated based on the user's historical expression data and used to correct and adjust the current macro index of expression expression. For example, if the user's expression was usually more excited when buying certain products in the past (i.e., more excited than most ordinary people), then this correction factor may reduce the expression index of the current similar expression.

[0078] In this embodiment, the expression expression macro index determination model is an algorithm or model for calculating the macro level of the user's expression expression (i.e., the level of expression compared with the expressions of most ordinary people) based on the current facial image sequence. That is, in the training structure, the model learns the different expression types and levels of expression of a large number of people in front of the vending machine to achieve a relative evaluation of the expression level of the input facial image sequence of a single user, and its training samples include: A large number of facial image sequences containing different users in different situations (e.g. different vending machine scenes, different times, etc.); These facial image sequences should be annotated with corresponding macro-indexes of facial expression, for example, by evaluation by professionals or based on certain known standards.

[0079] In this embodiment, the expression macro index is a comprehensive quantitative representation of the user's current expression, reflecting the intensity or obviousness of the expression. A larger value may indicate a stronger expression of the user, such as being very happy or very surprised; a smaller value may indicate a milder expression.

[0080] The beneficial effects of the above technology are as follows: the correction factor determination unit obtains the relative correction factor by inputting the historical facial image sequence, which can take into account the user's long-term expression habits and make the evaluation more personalized. The macro index determination unit uses the current facial image sequence to determine the expression expression macro index, focusing on the current expression state. The expression index determination unit combines the macro index and the correction factor to determine the real-time expression index, so that the evaluation result is more comprehensive, accurate and in line with the user's actual situation. It helps to more accurately quantify the intensity of the user's real-time facial expression, and provide strong support for the subsequent generation of accurate shopping guide information. It can improve the scientificity and reliability of expression index recognition, thereby optimizing the quality and effect of shopping guide services and better meeting user needs.

[0081] Example 8 Based on Example 1, the shopping guide information generation module includes: The feedback interpretation submodule is used to evaluate the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person based on the user's real-time facial expression information; The shopping guide information generation submodule is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the shopping guide information latest given by the AI ​​virtual person and the shopping guide information latest given by the AI ​​virtual person.

[0082] In this embodiment, the type of user feedback on the latest shopping guide information provided by the AI ​​virtual person refers to the type of user response to the shopping guide information provided by the AI ​​virtual person.

[0083] For example: The user's facial expression changes, language responses, operating actions, etc. reflect whether the user is interested, confused or dissatisfied with the shopping guide information.

[0084] In this embodiment, the feedback performance index is used to quantify the phenotype of the user's feedback type on the latest shopping guide information given by the AI ​​virtual person. If the user's expression is very surprised, the feedback performance index may be high; if the user just frowns slightly, the index may be low.

[0085] The beneficial effects of the above technology are: the feedback interpretation submodule can evaluate the feedback type and feedback performance index from the user's real-time facial expression information, providing a quantitative basis for accurately understanding the user's response to the shopping guide information. This quantitative evaluation helps to grasp the user's attitude and emotional tendency more accurately and improve the accuracy of interpreting user feedback. The shopping guide information generation submodule generates new shopping guide information based on accurate feedback evaluation and original shopping guide information, making the new shopping guide more targeted and adaptable. It can better meet the needs and expectations of users and improve user satisfaction and acceptance of shopping guide services. It helps to optimize the shopping guide process, improve sales efficiency and service quality, and enhance the competitiveness and practicality of AI virtual human shopping guides.

[0086] Example 9 Based on Example 8, the shopping guide information generation submodule includes: A first shopping guide information generating unit is configured to generate the latest shopping guide information based on the latest touch screen feedback information and / or voice feedback information given by the user, the feedback type and corresponding feedback performance index of the user to the latest shopping guide information given by the AI ​​virtual person, and the latest shopping guide information given by the AI ​​virtual person when the user gives the latest touch screen feedback information and / or voice feedback information; The second shopping guide information generating unit is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person and the latest shopping guide information given by the AI ​​virtual person when the user has not given the latest touch screen feedback information and / or voice feedback information.

[0087] In this embodiment, the touch screen feedback information refers to feedback data generated by the user performing operations through the touch screen, such as clicking on a product, sliding the screen, selecting an option, etc.

[0088] For example: A user clicks on the icon of a beverage on the touch screen of a vending machine to indicate interest in the beverage.

[0089] In this embodiment, the voice feedback information refers to the feedback content expressed by the user to the vending machine or the AI ​​virtual person through language.

[0090] Example: User says "I want chocolate flavor."

[0091] In this embodiment, the latest shopping guide information is generated based on the latest touch screen feedback information and / or voice feedback information given by the user, the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person, and the latest shopping guide information given by the AI ​​virtual person: this means that the latest feedback expressed by the user through the touch screen or voice, the feedback on the previous shopping guide information (such as type and performance intensity) and the shopping guide content given by the AI ​​virtual person before are comprehensively considered to regenerate new shopping guide information that is more in line with user needs and reactions.

[0092] For example: A user selects a drink through the touch screen and says "this is too expensive". The feedback performance index shows a high level of dissatisfaction, so new shopping guide information is generated, such as recommending a similar drink with a more suitable price.

[0093] In this embodiment, the latest shopping guide information is generated based on the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person, as well as the latest shopping guide information given by the AI ​​virtual person: when the user has no new touch screen or voice feedback, the new shopping guide information is regenerated based only on the feedback on multiple aspects of the previous shopping guide information (such as dimensions such as expression changes) and its intensity, as well as the previous shopping guide content.

[0094] For example: If the user does not speak or touch the screen, but his expression shows confusion and the feedback performance index is average, then new shopping guide information may be generated to further explain the characteristics of the previously recommended products.

[0095] The beneficial effects of the above technology are: it can generate the latest shopping guide information in different ways according to whether the user has the latest touch screen and voice feedback information, thereby improving the flexibility and adaptability of the generation strategy. When the user has touch screen and voice feedback, the shopping guide information is generated by integrating various feedback information, so that the shopping guide is more in line with the user's clear needs and expressions. When the user does not have such clear feedback, the shopping guide information is generated only based on the expression feedback, making full use of all available feedback channels to ensure the continuity and effectiveness of the shopping guide service. It helps to meet the needs of users more comprehensively and accurately, and improve the user experience and the success rate of shopping guides. This refined processing method can optimize resource utilization and improve the efficiency and quality of shopping guide information generation.

[0096] Embodiment 10: The present invention provides an AI virtual human shopping guide method based on facial expression recognition, which is applied to the AI ​​virtual human shopping guide system based on facial expression recognition described in any one of embodiments 1 to 9, with reference to Figure 2 ,include: S1: Obtain facial image sequences during the interaction between the shopping guide target user and the AI ​​virtual person; S2: Recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; S3: Generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

[0097] The beneficial effects of the above technology are as follows: the facial data acquisition module can be used to obtain facial image sequences, providing a data basis for the subsequent analysis of user expressions, ensuring the integrity and accuracy of the information. The facial expression information recognition module uses the facial image sequence to accurately identify the user's real-time facial expression and expression index, which can not only identify the user's expression type, but also the degree of expression of the user's expression type, which helps to deeply understand the user's emotions and reactions. Based on the accurate recognition of the user's real-time facial expression and combined with the latest shopping guide information given by the AI ​​virtual person, the shopping guide information generation module can generate the latest shopping guide information that is more in line with the user's current emotions and needs, improve the coherent analysis performance of the shopping guide information in the interactive process, and improve the pertinence and effectiveness of the shopping guide. It helps to improve the interactive experience between users and AI virtual people, and enhance users' satisfaction and acceptance of shopping guide services. It can adjust the shopping guide strategy in time according to the user's real-time feedback, and improve the sales conversion rate and service quality.

[0098] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. AI virtual human shopping guide system based on facial expression recognition, characterized by: include: The facial data acquisition module is used to obtain the facial image sequence during the interaction between the shopping guide target user and the AI ​​virtual person; The facial expression information recognition module is used to recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; The shopping guide information generation module is used to generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

2. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 1 is characterized in that: The facial data acquisition module includes: The monitoring video acquisition submodule is used to obtain real-time monitoring video during the interaction between the user and the AI ​​virtual person; The facial image capture submodule is used to capture all complete facial images of the shopping guide target user in the real-time monitoring video; The image sequence generation submodule is used to generate a facial image sequence of the shopping guide target user based on all the captured complete facial images of the shopping guide target user.

3. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 2 is characterized in that: The facial image capture submodule includes: A shopping guide target user extraction unit is used to extract all head images from the real-time monitoring video, determine whether there is a video frame containing more than one head image in the real-time monitoring video, and if so, determine a complete limb image sequence of each head image in the corresponding video frame; otherwise, extract the head image contained in the latest video frame in the real-time monitoring video as the head image of the shopping guide target user; A shopping guide target user head image determination unit, used to determine the shopping guide target user head image based on a complete limb image sequence of each head image in a video frame containing more than one head image in the real-time monitoring video; A head image extraction unit, used to extract all head monitoring images of the shopping guide target user's head image from the real-time monitoring video based on the head morphological features of the shopping guide target user's head image; The facial image extraction unit is used to extract all complete facial images of the shopping guide target user from all head monitoring images of the shopping guide target user's head images.

4. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 3 is characterized in that: The shopping guide target user head image determination unit includes: Other information acquisition subunits are used to acquire touch screen interaction record sequences and online interaction record sequences in application scenarios; a transaction dominance index determination subunit, for evaluating a transaction dominance index of each head image in a video frame containing more than one head image in a real-time monitoring video based on a touch screen interaction record sequence and an online interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video; The shopping guide target user head image determination subunit is used to use the head image with the largest transaction dominance index as the shopping guide target user head image.

5. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 4 is characterized in that: The transaction dominance index determines the sub-units, including: A touch screen behavior matching degree determination terminal is used to input a touch screen interaction record sequence in an application scenario and a complete limb image sequence of each head image in a video frame containing more than one head image in a real-time monitoring video into a touch screen behavior matching degree analysis model to obtain a touch screen behavior matching degree of each head image in a video frame containing more than one head image in a real-time monitoring video; An online interaction behavior matching degree determination terminal is used to input the online interaction record sequence in the application scenario and the complete limb image sequence of each head image in the video frame containing more than one head image in the real-time monitoring video into the online interaction behavior matching degree analysis model to obtain the online interaction behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video; The transaction dominance index determination terminal is used to use the sum of the touch screen behavior matching degree and the online interactive behavior matching degree of each head image in the video frame containing more than one head image in the real-time monitoring video as the transaction dominance index of each head image in the video frame containing more than one head image in the real-time monitoring video.

6. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 1, characterized in that: Expression information recognition module, including: The expression type recognition submodule is used to analyze the real-time facial expression type of the shopping guide target user based on the facial image sequence; The expression index recognition submodule is used to determine the real-time expression index of the real-time facial expression type of the shopping guide target user based on the current facial image sequence and the historical facial image sequence of the shopping guide target user.

7. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 6 is characterized in that: Performance index identification submodule, including: A correction factor determination unit, used to input a historical facial image sequence of a shopping guide target user into a facial expression macro-index relative correction factor determination model to obtain a facial expression macro-index relative correction factor of the shopping guide target user; A macro index determination unit, used to input the current facial image sequence into the expression expression macro index determination model to obtain the expression expression macro index of the shopping guide target user; The expression index determination unit is used to use the sum of the expression expression macro index and the expression expression macro index relative correction factor of the shopping guide target user as the real-time expression index of the real-time facial expression type of the shopping guide target user.

8. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 1, characterized in that: The shopping guide information generation module includes: The feedback interpretation submodule is used to evaluate the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person based on the user's real-time facial expression information; The shopping guide information generation submodule is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the shopping guide information latest given by the AI ​​virtual person and the shopping guide information latest given by the AI ​​virtual person.

9. The AI ​​virtual human shopping guide system based on facial expression recognition according to claim 8, characterized in that: The shopping guide information generation submodule includes: A first shopping guide information generating unit is configured to generate the latest shopping guide information based on the latest touch screen feedback information and / or voice feedback information given by the user, the feedback type and corresponding feedback performance index of the user to the latest shopping guide information given by the AI ​​virtual person, and the latest shopping guide information given by the AI ​​virtual person when the user gives the latest touch screen feedback information and / or voice feedback information; The second shopping guide information generating unit is used to generate the latest shopping guide information based on the user's feedback type and corresponding feedback performance index on the latest shopping guide information given by the AI ​​virtual person and the latest shopping guide information given by the AI ​​virtual person when the user has not given the latest touch screen feedback information and / or voice feedback information.

10. The AI ​​virtual human shopping guide method based on facial expression recognition is characterized by: The AI ​​virtual human shopping guide system based on facial expression recognition applied to any one of claims 1 to 9 comprises: S1: Obtain facial image sequences during the interaction between the shopping guide target user and the AI ​​virtual person; S2: Recognize the facial expressions of the shopping guide target user in the interaction process with the AI ​​virtual person based on the facial image sequence, and obtain the real-time facial expression information of the shopping guide target user, wherein the real-time facial expression information includes the real-time facial expression type of the shopping guide target user and the corresponding real-time expression index; S3: Generate the latest shopping guide information based on the real-time facial expression information of the shopping guide target user and the latest shopping guide information given by the AI ​​virtual person.

Images