Intelligent commodity vending machine system with machine intelligence

Abstract

The invention discloses an intelligent commodity vending machine system with machine intelligence, which analyzes and captures physiological state time sequence joint features and human face state features of a user by using an artificial intelligence technology and a deep learning technology so as to judge the physical state of the user. And whether the beverage is fatigue, thirst, hunger sensation and the like can be judged, so that the adaptive type of beverage can be intelligently recommended. Therefore, the user can further select the commodity according to the recommendation, and can also give feedback through voice or a touch screen, so that the system can further optimize the recommendation result. The dynamic interaction mode not only improves the user experience, but also provides a basis for self-learning and improvement of the commodity intelligent vending machine system. In this way, the intelligent degree of the vending machine can be improved, so that more personalized and accurate vending service can be provided for consumers.

Description

Technical Field

[0001] The present application relates to the field of machine intelligence, and more specifically, to an intelligent commodity vending machine system with machine intelligence. Background Art

[0002] A vending machine is a device that provides goods to consumers in a self-service manner, usually with functions such as currency reception, goods storage and distribution. Modern vending machines not only provide traditional beverages and snacks, but also expand to fresh food, medicine, daily necessities and other fields. The advantage of vending machines lies in their convenience and efficiency. Consumers can quickly obtain the required goods at any time and any place without queuing or interacting with store clerks. In addition, the operating cost of vending machines is relatively low, and they can complete sales without manpower.

[0003] With the development of Internet of Things technology and the growing demand for convenient life, vending machines, as a form of traditional retail, are gradually transforming towards intelligence and personalization. However, traditional vending machines usually sell based on a fixed list of goods, that is, traditional vending machines can only realize the preset product display and sales functions, lack the ability to understand and respond to consumers' personalized needs in a timely manner, and reduce the user's consumption experience. In addition, this type of vending machine requires consumers to select and operate goods, and the level of intelligence is low, which may cause users to often face confusion when choosing goods and cannot quickly find the most suitable goods for themselves, so it cannot meet the actual needs of modern consumers.

[0004] Therefore, a commodity intelligent automatic vending machine system is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a commodity intelligent vending machine system with machine intelligence, which uses artificial intelligence technology and deep learning technology to analyze and capture the user's physiological state time series joint features and facial state features, so as to judge the user's physical state, such as whether he is tired, thirsty, hungry, etc., so as to intelligently recommend suitable types of drinks. In this way, the user can further select commodities according to the recommendation, and can also give feedback through voice or touch screen so that the system can further optimize the recommendation results. This dynamic interaction method not only improves the user experience, but also provides a basis for the self-learning and improvement of the commodity intelligent vending machine system. In this way, the intelligence level of the vending machine can be improved so as to provide consumers with more personalized and accurate vending services.

[0006] According to one aspect of the present application, there is provided a commodity intelligent automatic vending machine system with machine intelligence, which includes:

[0007] a communication module, configured to receive, via a Bluetooth communication module, a data set of physiological state data collected by a wearable device within a predetermined time period before a current time point, wherein the physiological state data includes a heart rate value, a blood oxygen level value, and a skin temperature;

[0008] A facial state acquisition module, used for receiving a facial state image of a human face acquired by a camera;

[0009] A body state analysis module, used to extract body state features from the data set of physiological state data and the facial state image of the human face;

[0010] The product recommendation module is used to recommend suitable types of drinks based on the physical condition characteristics.

[0011] Compared with the prior art, the present application provides a commodity intelligent vending machine system with machine intelligence, which uses artificial intelligence technology and deep learning technology to analyze and capture the user's physiological state time series joint features and facial state features, so as to judge the user's physical state, such as whether he is tired, thirsty, hungry, etc., so as to intelligently recommend suitable types of drinks. In this way, users can further select commodities based on recommendations, and can also give feedback through voice or touch screen so that the system can further optimize the recommendation results. This dynamic interaction method not only improves the user experience, but also provides a basis for self-learning and improvement of the commodity intelligent vending machine system. In this way, the intelligence level of the vending machine can be improved to provide consumers with more personalized and accurate vending services. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 is a block diagram of a commodity intelligent vending machine system with machine intelligence according to an embodiment of the present application;

[0014] Figure 2 A data flow diagram of a commodity intelligent vending machine system with machine intelligence according to an embodiment of the present application;

[0015] Figure 3 is a block diagram of a body state analysis module in a commodity intelligent vending machine system with machine intelligence according to an embodiment of the present application;

[0016] Figure 4 It is a block diagram of a feature interactive screening unit in a commodity intelligent vending machine system with machine intelligence according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0018] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0019] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0020] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0021] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0022] Traditional vending machines usually sell based on a fixed list of goods, that is, they can only realize the preset product display and sales functions, lack the ability to understand and respond to consumers' personalized needs in a timely manner, and reduce the user's consumption experience. In addition, this type of vending machine requires consumers to select and operate goods, and the level of intelligence is low, which may cause users to often face confusion when choosing goods and cannot quickly find the most suitable goods for themselves, so it cannot meet the actual needs of modern consumers. Therefore, a commodity intelligent vending machine system is desired.

[0023] In the technical solution of the present application, a commodity intelligent vending machine system with machine intelligence is proposed. Figure 1 The block diagram is a commodity intelligent vending machine system with machine intelligence according to an embodiment of the present application. Figure 2 FIG. 1 is a data flow diagram of a commodity intelligent automatic vending machine system with machine intelligence according to an embodiment of the present application. Figure 1 and Figure 2 As shown, the intelligent commodity vending machine system with machine intelligence according to the embodiment of the present application includes: a communication module 310, which is used to receive a data set of physiological state data within a predetermined time period before the current time point collected by a wearable device through a Bluetooth communication module, and the physiological state data includes a heart rate value, a blood oxygen level value and a skin temperature; a facial state acquisition module 320, which is used to receive a facial state image of a human face collected by a camera; a body state analysis module 330, which is used to extract body state features from the data set of the physiological state data and the facial state image of the human face; a commodity recommendation module 340, which is used to recommend a suitable type of beverage based on the body state features.

[0024] In particular, the communication module 310 and the facial state acquisition module 320 are used to receive a data set of physiological state data collected by the wearable device within a predetermined time period before the current time point through the Bluetooth communication module, and the physiological state data includes a heart rate value, a blood oxygen level value and a skin temperature; and receive a facial state image of a human face collected by a camera. It should be understood that by analyzing the comprehensive features of the data set of the physiological state data and the facial state image of the human face, the current physical state, such as fatigue, thirst, hunger, etc., can be evaluated, so as to intelligently recommend an adapted type of beverage. In one example, if the user has a high heart rate and a high stress level, a soothing beverage such as herbal tea or juice is recommended; if the user has a low blood oxygen level, a beverage with a high oxygen content such as a sports drink or oxygenated water is recommended; if the user has a high skin temperature, a refreshing beverage such as ice water or iced beverage is recommended. In this way, the intelligence level of the vending machine can be improved to provide consumers with more personalized and accurate vending services.

[0025] In particular, the body state analysis module 330 is used to extract body state features from the data set of the physiological state data and the facial state image of the human face. Figure 3As shown, the body state analysis module 330 includes: a physiological state parameter time series joint construction unit 331, which is used to arrange the data set of the physiological state data into a physiological state parameter time series joint matrix according to the parameter sample dimension and the time dimension; a physiological state parameter time series association pattern capture unit 332, which is used to input the physiological state parameter time series joint matrix into a physiological state parameter time series association pattern feature extractor based on a converter structure to obtain a physiological state parameter time series association feature vector; a human face state capture unit 333, which is used to input the human face state image into a human face extractor based on a pyramid network to obtain a human face state multi-scale fusion feature map; a feature interaction screening unit 334, which is used to perform feature screening processing based on a gradient amplitude mask on the physiological state parameter time series association feature vector and the human face state multi-scale fusion feature map to obtain a body state multi-modal joint enhanced representation feature map as the body state feature.

[0026] Specifically, the physiological state parameter time series joint construction unit 331 is used to arrange the data set of the physiological state data into a physiological state parameter time series joint matrix according to the parameter sample dimension and the time dimension. It is worth mentioning that the physiological state parameter time series joint matrix can help observe and extract different physiological state parameters, such as the changing trends of heart rate value, blood oxygen level value and skin temperature over time and their relationships with each other.

[0027] Specifically, the physiological state parameter temporal association pattern capture unit 332 is used to input the physiological state parameter temporal joint matrix into the physiological state parameter temporal association pattern feature extractor based on the converter structure to obtain the physiological state parameter temporal association feature vector. In order to capture the long-distance dependency temporal association characteristics between the physiological state parameters, in the technical solution of the present application, the physiological state parameter temporal joint matrix is ​​further input into the physiological state parameter temporal association pattern feature extractor based on the converter structure to obtain the physiological state parameter temporal association feature vector. The physiological state parameter temporal association pattern feature extractor can utilize the self-attention advantage of the converter structure to capture the temporal long-distance dependency association pattern characteristics between each physiological state parameter, and focus on the complex temporal dependency relationship of physiological parameters related to the user's physical state detection and expression tasks, which is crucial for analyzing the user's physical condition.

[0028] Specifically, the face state capturing unit 333 is used to input the face state image into a face extractor based on a pyramid network to obtain a multi-scale fusion feature map of the face state. That is, the face state image is input into a face extractor based on a pyramid network for feature mining to extract multi-scale and multi-level fusion feature information about the face state of the user, thereby obtaining a multi-scale fusion feature map of the face state. It is worth mentioning that the pyramid network can capture features of different scales by using multiple pyramid paths, and objects can appear in different areas and scales of the image; in addition, the intermediate paths in the pyramid network allow information to be exchanged between feature maps of different scales. This helps the model capture global and local contextual information in the image.

[0029] Specifically, the feature interaction screening unit 334 is used to perform feature screening processing based on the gradient amplitude mask on the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map to obtain a body state multi-modal joint enhanced representation feature map as the body state feature. It should be understood that the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map respectively contain the temporal correlation representation information of multiple physiological state parameters of the user in the time dimension and the user's face facial state multi-scale fusion representation information, which usually contain a large amount of redundant information. This information is not helpful for the final body state judgment and may even introduce noise interference. Therefore, in order to remove this unnecessary information and retain the key parts and features that are most relevant to the user's body state detection task, in the technical solution of the present application, the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map are further subjected to feature screening processing based on the gradient amplitude mask to obtain a body state multi-modal joint enhanced representation feature map. In a specific example of the present application, Figure 4 As shown, the feature interaction screening unit 334 includes: a gradient amplitude mask subunit 3341, which is used to perform mask processing on the physiological state parameter time-series association feature vector based on gradient amplitude calculation to obtain a masked first physiological state parameter time-series association auxiliary feature mapping pattern representation vector and a masked second physiological state parameter time-series association auxiliary feature mapping pattern representation vector; a feature screening subunit 3342, which is used to perform feature screening on the multi-scale fusion feature map of the facial state of the human face based on the masked first physiological state parameter time-series association auxiliary feature mapping pattern representation vector and the masked second physiological state parameter time-series association auxiliary feature mapping pattern representation vector to obtain the multi-modal joint enhanced representation feature map of the body state.

[0030] More specifically, the gradient amplitude mask subunit 3341 is used to perform mask processing based on gradient amplitude calculation on the physiological state parameter time series associated feature vector to obtain a masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector and a masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector. First, a linear transformation based on a first weight modifier matrix and a second weight modifier matrix is ​​applied to the physiological state parameter time series associated feature vector to generate two auxiliary feature mapping mode representation vectors. The linear transformation step expands the expression range of the physiological state parameter time series associated feature, so that the model can capture the relationship between the user's physiological state time series semantics and the facial state of the face from different angles. Next, considering that in the feature map, the area with a large gradient amplitude usually indicates that the information in the area is richer or has a larger change, these areas are likely to contain important physiological state information. Therefore, through gradient amplitude mask processing, those features with larger changes and more likely to carry important information can be highlighted. The gradient amplitude reflects the degree of change of each element in the auxiliary feature mapping pattern representation vector of the physiological state parameter temporal association. A large gradient amplitude means that the feature at that position has a significant change in space, and such a feature is likely to carry key information about the user's physical state.

[0031] In an embodiment of the present application, the physiological state parameter time series association feature vector is subjected to mask processing based on gradient amplitude calculation to obtain a masked first physiological state parameter time series association auxiliary feature mapping pattern representation vector and a masked second physiological state parameter time series association auxiliary feature mapping pattern representation vector, including: the physiological state parameter time series association feature vector is subjected to linear transformation based on a first weight modifier matrix and a second weight modifier matrix respectively to obtain a first physiological state parameter time series association auxiliary feature mapping pattern representation vector and a second physiological state parameter time series association auxiliary feature mapping pattern representation vector; based on the gradient amplitude of each position in the first physiological state parameter time series association auxiliary feature mapping pattern representation vector, the first physiological state parameter time series association auxiliary feature mapping pattern representation vector is subjected to mask processing based on gradient amplitude to obtain the masked first physiological state parameter time series association auxiliary feature mapping pattern representation vector; based on the gradient amplitude of each position in the second physiological state parameter time series association auxiliary feature mapping pattern representation vector, the second physiological state parameter time series association auxiliary feature mapping pattern representation vector is subjected to mask processing based on gradient amplitude to obtain the masked second physiological state parameter time series association auxiliary feature mapping pattern representation vector.

[0032] Among them, the process of performing linear transformation based on the first weight modifier matrix and the second weight modifier matrix on the physiological state parameter time series association feature vector to obtain the first physiological state parameter time series association auxiliary feature mapping mode representation vector and the second physiological state parameter time series association auxiliary feature mapping mode representation vector includes: calculating the physiological state parameter time series association feature vector and the transpose of the first weight modifier matrix, multiplying it, and then adding it to the first bias modifier vector by position to obtain the first physiological state parameter time series association auxiliary feature mapping mode representation vector; calculating the physiological state parameter time series association feature vector and the transpose of the second weight modifier matrix, multiplying it, and then adding it to the second bias modifier vector by position to obtain the second physiological state parameter time series association auxiliary feature mapping mode representation vector; and, based on the gradient amplitude of each position in the first physiological state parameter time series association auxiliary feature mapping mode representation vector, performing gradient-based The process of masking the amplitude to obtain the masked first physiological state parameter time-series associated auxiliary feature mapping pattern representation vector includes: calculating the square of the difference between the previous position feature value and the next position feature value of each position feature value in the first physiological state parameter time-series associated auxiliary feature mapping pattern representation vector to obtain the first physiological state parameter time-series gradient amplitude representation vector; calculating the absolute value of the square root of each first physiological state parameter time-series gradient amplitude in the first physiological state parameter time-series gradient amplitude representation vector, and inputting the obtained feature vector into the masking function for processing to obtain the masked first physiological state parameter time-series associated auxiliary feature mapping pattern representation vector; wherein, in the masking function, in response to the first physiological state parameter time-series gradient amplitude in the first physiological state parameter time-series gradient amplitude representation vector being greater than a predetermined threshold, the first physiological state parameter time-series gradient amplitude greater than the predetermined threshold is set to 1, otherwise it is set to 0 to obtain the masked first physiological state parameter time-series associated auxiliary feature mapping pattern representation vector.

[0033] More specifically, the feature screening subunit 3342 is used to perform feature screening on the multi-scale fusion feature map of the face state based on the masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector and the masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector to obtain the multi-modal joint enhanced representation feature map of the body state. Specifically, first, the multi-scale fusion feature map of the face state is preliminarily screened by using the masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector combined with the hyperbolic tangent function. The hyperbolic tangent function is a commonly used activation function with an output range between -1 and +1, which can effectively distinguish different areas in the multi-scale fusion feature map of the face state, thereby highlighting those features that are highly related to the user's face state features in the preliminary screening stage to strengthen the performance of these features in the final feature map. Furthermore, based on the preliminary screening, the feature map after the preliminary screening is further screened using the masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector combined with the Sigmoid function. The Sigmoid function is also an activation function, but its output range is between 0 and 1, which is suitable for probability estimation and binary decision-making. In this step, the Sigmoid function can help determine which features are truly important and highly relevant to subsequent tasks, and generate the final multimodal joint enhanced representation feature map of the body state. This feature map not only contains the information that best represents the user's current physical state, but also has been effectively integrated with multimodal data, enhancing the feature map's ability to represent the user's physical state. Ultimately, this feature map will serve as the user's physical state feature to help the system determine the user's physical state for subsequent product recommendations.

[0034] In an embodiment of the present application, feature screening is performed on the multi-scale fusion feature map of the face state based on the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector and the masked second physiological state parameter temporal association auxiliary feature mapping pattern representation vector to obtain the multi-modal joint enhanced representation feature map of the body state, including: performing preliminary feature screening on the multi-scale fusion feature map of the face state based on the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector and the hyperbolic tangent function to obtain the multi-scale fusion feature map of the face state after preliminary screening; and performing feature screening again on the multi-scale fusion feature map of the face state after preliminary screening based on the masked second physiological state parameter temporal association auxiliary feature mapping pattern representation vector and the Sigmoid function to obtain the multi-modal joint enhanced representation feature map of the body state.

[0035] Among them, the process of performing preliminary feature screening on the multi-scale fusion feature map of the face state based on the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector and the hyperbolic tangent function to obtain the multi-scale fusion feature map of the face state after preliminary screening includes: using the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement on the multi-scale fusion feature map of the face state along the channel dimension, and then inputting the weighted enhanced feature map into the hyperbolic tangent function for activation to obtain the multi-scale fusion feature map of the face state after preliminary screening. And, based on the masked second physiological state parameter temporal association auxiliary feature mapping pattern representation vector and the Sigmoid function, the process of performing feature screening again on the multi-scale fusion feature map of the facial state after the preliminary screening to obtain the multi-modal joint enhanced representation feature map of the body state includes: using the masked second physiological state parameter temporal association auxiliary feature mapping pattern representation vector as the channel weight vector, performing weighted enhancement along the channel dimension on the multi-scale fusion feature map of the facial state after the preliminary screening, and then inputting the weighted enhanced feature map into the Sigmoid function for activation to obtain the multi-modal joint enhanced representation feature map of the body state.

[0036] In summary, in the above embodiment, the physiological state parameter time series associated feature vector is subjected to mask processing based on gradient amplitude calculation to obtain a masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector and a masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector, including:

[0037] The physiological state parameter time series correlation feature vector and the face state multi-scale fusion feature map are subjected to feature screening processing based on gradient amplitude mask using the following feature interaction screening formula to obtain a body state multi-modal joint enhanced representation feature map as the body state feature; wherein the feature interaction screening formula includes:

[0038]

[0039]

[0040] Among them, V meta is the temporal correlation feature vector of the physiological state parameters, and w 0f are the first weight modifier matrix and the first bias modifier vector, respectively, and w 0g are the second weight modifier matrix and the second bias modifier vector, respectively, f b (·) and g b(·) are the first linear transformation and the second linear transformation respectively, V1 and V2 are the first physiological state parameter time series associated auxiliary feature mapping mode representation vector and the second physiological state parameter time series associated auxiliary feature mapping mode representation vector respectively, V1(i+1) and V1(i-1) are the feature values ​​of the (i+1)th position and the (i-1)th position in the first physiological state parameter time series associated auxiliary feature mapping mode representation vector respectively, V2(i+1) and V2(i-1) are the feature values ​​of the (i+1)th position and the (i-1)th position in the second physiological state parameter time series associated auxiliary feature mapping mode representation vector respectively, mask(·) is a mask operation, and are the eigenvalues ​​of the ith position in the masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector and the masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector, θ is the preset threshold, F main is the multi-scale fusion feature map of the face state, V 1s is the masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector, V 2s is the masked second physiological state parameter time series associated auxiliary feature mapping mode representation vector, is the position-wise weighted multiplication along the channel dimension, tanh(·) is the hyperbolic tangent function, T gate is the multi-scale fusion feature map of the face state after preliminary screening, sigmoid(·) is the sigmoid function, S gate A multi-modal joint enhanced representation feature map is provided for the body state.

[0041] It is worth mentioning that in other specific examples of the present application, the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map can also be subjected to feature screening processing based on the gradient amplitude mask in other ways to obtain a multi-modal joint enhanced representation feature map of the body state as the body state feature, for example: inputting the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map; calculating the gradient amplitude of the physiological state parameter time-series correlation feature vector and the face facial state multi-scale fusion feature map respectively; wherein the gradient amplitude measures the intensity of the change of the pixel value in the feature map; adding the gradient amplitude of the physiological state parameter time-series correlation feature vector to the gradient amplitude of the face facial state multi-scale fusion feature map; performing a thresholding operation on the added gradient amplitude to generate a gradient amplitude mask; applying the gradient amplitude mask to the face facial state multi-scale fusion feature map; areas with higher gradient amplitudes (i.e., areas with larger changes) will be retained, while areas with lower gradient amplitudes (i.e., areas with smaller changes) will be suppressed; to obtain the multi-modal joint enhanced representation feature map of the body state.

[0042] It is worth mentioning that in other specific examples of the present application, body state features can also be extracted from the dataset of the physiological state data and the facial state image in other ways, for example: inputting the dataset of the physiological state data and the facial state image; preprocessing the physiological state data, including missing value processing, data standardization, etc.; extracting relevant features from the physiological state data; using feature selection technology to select features most relevant to the body state; using a face detection algorithm to detect the face area; using a deep learning model (such as VGGNet, ResNet, etc.) to extract features from the facial image; and fusing the features extracted from the physiological state data and the facial state image to form body state features to obtain the body state features.

[0043] In particular, the commodity recommendation module 340 is used to recommend an adapted type of beverage based on the physical state feature. In a specific example of the present application, the multimodal joint enhanced representation feature map of the physical state is passed through a classifier-based beverage intelligent recommender to obtain a recommendation result. In this way, the user's physical state can be comprehensively judged based on the user's physiological state and facial state, such as whether he is tired, thirsty, hungry, etc., so as to intelligently recommend an adapted type of beverage. For example, if the user looks very tired, the system may recommend energy drinks or coffee; if the user appears thirsty, water or juice may be recommended. In this way, the user can further select the product based on the recommendation, and can also give feedback through voice or touch screen so that the system can further optimize the recommendation results. This dynamic interaction method not only improves the user experience, but also provides a basis for self-learning and improvement of the commodity intelligent vending machine system. In this way, the intelligence level of the vending machine can be improved so as to provide consumers with more personalized and accurate vending services.

[0044] In a preferred example, the multimodal joint enhanced representation feature graph of the body state is passed through a classifier-based beverage intelligent recommender to obtain a recommendation result, including:

[0045] Calculating the sum of the absolute values ​​of each eigenvalue of the multimodal joint enhanced representation feature map of the body state to obtain a first multimodal joint enhanced representation of the body state and a modulation value, and calculating the square root of the sum of the squares of each eigenvalue of the multimodal joint enhanced representation feature map of the body state to obtain a second multimodal joint enhanced representation of the body state and a modulation value;

[0046] After performing dot-wise subtraction on the body state multimodal joint enhanced representation feature map and the second body state multimodal joint enhanced representation and modulation value, perform dot-wise multiplication on the number of eigenvalues ​​of the body state multimodal joint enhanced representation feature map and the reciprocal of the first body state multimodal joint enhanced representation and modulation value, and take the reciprocal of each eigenvalue to obtain the first body state multimodal joint enhanced representation phase transition feature map;

[0047] After performing dot-wise subtraction on the body state multimodal joint enhanced representation feature map and the first body state multimodal joint enhanced representation and modulation value, perform dot-wise multiplication on the square root of the number of eigenvalues ​​of the body state multimodal joint enhanced representation feature map and the reciprocal of the second body state multimodal joint enhanced representation and modulation value, and take the reciprocal of each eigenvalue to obtain a second body state multimodal joint enhanced representation phase transition feature map;

[0048] Subtract the dot product feature map of the second body state multimodal joint enhanced representation phase transition feature map and the weighted hyperparameter from the first body state multimodal joint enhanced representation phase transition feature map to obtain an optimized body state multimodal joint enhanced representation feature map;

[0049] The optimized multimodal joint enhanced representation feature map of the body state is passed through a classifier-based beverage intelligent recommender to obtain a recommendation result.

[0050] Here, the multi-modal joint enhanced representation feature map of the body state, denoted as F, is optimized as follows:

[0051]

[0052] f i ∈F∈R W×H×C

[0053] n=W×H×C

[0054] Where F is the multi-modal joint enhanced representation feature map of the body state, f i are the eigenvalues ​​of the multimodal joint enhanced representation feature map of the body state, α is the multimodal joint enhanced representation and modulation value of the first body state, and β is the multimodal joint enhanced representation and modulation value of the second body state, n is the number of eigenvalues ​​of the multimodal joint enhanced representation feature map of the body state, F1 is the phase transition feature map of the first multimodal joint enhanced representation; F2 is the phase transition feature map of the second multimodal joint enhanced representation of the body state, ω is the weighted hyperparameter, is point subtraction, ⊙ is point multiplication, and F' is the optimized multi-modal joint enhanced representation feature map of the body state.

[0055] In particular, in the technical solution of the present application, considering that the physiological state parameter temporal correlation feature vector and the human face facial state multi-scale fusion feature map respectively represent the physiological state parameter temporal correlation features and the facial state image multi-scale fusion features of the human face facial state image, when they are passed through the feature interaction screening encoder based on the metadata block, the feature interaction screening of different dimensions based on the cross-modal feature distribution will also have the cross-modal context feature interaction encoding difference, so it is expected to improve the detail semantic aggregation expression effect of the multimodal semantic interaction representation difference of the body state multimodal joint enhanced representation feature map.

[0056] Therefore, in the preferred example, the difference of the feature value of the multimodal joint enhanced representation feature map relative to the overall feature set of the multimodal joint enhanced representation feature map of the body state is used as the semantic change intensity information, and the phase-like conversion corresponding to the position-based intensity modulation is performed through the different and modulated representation forms, so that the aggregation enhancement of the semantic change phase perception can improve the axial aggregation receptive field along the feature aggregation direction by performing a spatial translation operation based on alternating stacking under the set scale balance of the multimodal joint enhanced representation feature map of the body state, thereby improving the perception effect of the aggregate semantics of the multimodal joint enhanced representation feature map of the body state for the change of detailed semantics, so as to improve the expression effect of the multimodal joint enhanced representation feature map of the body state, and improve the accuracy of the recommendation results obtained by the beverage intelligent recommender based on the classifier. In this way, it is possible to more accurately recommend and select commodities for the user based on the user's physical state. This dynamic interaction method not only improves the user experience, but also provides a basis for the self-learning and improvement of the commodity intelligent vending machine system. In this way, the intelligence of the vending machine can be improved, so as to provide consumers with more personalized and accurate vending services.

[0057] As described above, the intelligent commodity vending machine system 300 with machine intelligence according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an intelligent commodity vending algorithm with machine intelligence. In one possible implementation, the intelligent commodity vending machine system 300 with machine intelligence according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the intelligent commodity vending machine system 300 with machine intelligence can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the intelligent commodity vending machine system 300 with machine intelligence can also be one of the many hardware modules of the wireless terminal.

[0058] Alternatively, in another example, the intelligent automatic vending machine system 300 for goods with machine intelligence and the wireless terminal may also be separate devices, and the intelligent automatic vending machine system 300 for goods with machine intelligence may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0059] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A commodity intelligent vending machine system with machine intelligence, characterized in that: include: a communication module, configured to receive, via a Bluetooth communication module, a data set of physiological state data collected by a wearable device within a predetermined time period before a current time point, wherein the physiological state data includes a heart rate value, a blood oxygen level value, and a skin temperature; A facial state acquisition module, used for receiving a facial state image of a human face acquired by a camera; A body state analysis module, used to extract body state features from the data set of physiological state data and the facial state image of the human face; The product recommendation module is used to recommend suitable types of drinks based on the physical condition characteristics.

2. The intelligent commodity vending machine system with machine intelligence according to claim 1, characterized in that: The body state analysis module comprises: A physiological state parameter time series joint construction unit, used for arranging the data set of the physiological state data into a physiological state parameter time series joint matrix according to the parameter sample dimension and the time dimension; A physiological state parameter time series association pattern capture unit, used for inputting the physiological state parameter time series joint matrix into a physiological state parameter time series association pattern feature extractor based on a converter structure to obtain a physiological state parameter time series association feature vector; A face state capturing unit, used for inputting the face state image into a face extractor based on a pyramid network to obtain a multi-scale fusion feature map of the face state; A feature interaction screening unit, used for performing feature screening processing based on a gradient amplitude mask on the physiological state parameter time series correlation feature vector and the multi-scale fusion feature map of the facial state to obtain a body state multi-modal joint enhanced representation feature map as the body state feature; Among them, the feature interaction screening unit includes: a gradient amplitude mask subunit, which is used to perform mask processing on the physiological state parameter time-related feature vector based on gradient amplitude calculation to obtain a masked first physiological state parameter time-related auxiliary feature mapping pattern representation vector and a masked second physiological state parameter time-related auxiliary feature mapping pattern representation vector; a feature screening subunit, which is used to perform feature screening on the multi-scale fusion feature map of the facial state of the human face based on the masked first physiological state parameter time-related auxiliary feature mapping pattern representation vector and the masked second physiological state parameter time-related auxiliary feature mapping pattern representation vector to obtain the multi-modal joint enhanced representation feature map of the body state.

3. The intelligent commodity vending machine system with machine intelligence according to claim 2, characterized in that: The gradient magnitude mask subunit comprises: A linear transformation secondary subunit, used for performing linear transformations on the physiological state parameter time series associated feature vectors based on the first weight modifier matrix and the second weight modifier matrix to obtain the first physiological state parameter time series associated auxiliary feature mapping mode representation vector and the second physiological state parameter time series associated auxiliary feature mapping mode representation vector; a first mask secondary subunit, configured to perform a masking process based on the gradient amplitude of each position in the first physiological state parameter time series association auxiliary feature mapping pattern representation vector on the first physiological state parameter time series association auxiliary feature mapping pattern representation vector to obtain the masked first physiological state parameter time series association auxiliary feature mapping pattern representation vector; The second mask secondary subunit is used to perform gradient amplitude-based mask processing on the second physiological state parameter timing-associated auxiliary feature mapping pattern representation vector based on the gradient amplitude of each position in the second physiological state parameter timing-associated auxiliary feature mapping pattern representation vector to obtain the masked second physiological state parameter timing-associated auxiliary feature mapping pattern representation vector.

4. The intelligent commodity vending machine system with machine intelligence according to claim 3 is characterized in that: The linear transformation secondary subunit is used for: Calculate the physiological state parameter time series associated feature vector and the transpose of the first weight modifier matrix, multiply the result, and then add the result to the first bias modifier vector by position to obtain the first physiological state parameter time series associated auxiliary feature mapping mode representation vector; The second physiological state parameter time series associated auxiliary feature mapping mode representation vector is obtained by calculating the physiological state parameter time series associated feature vector and the transpose of the second weight modifier matrix, multiplying the resultant result and then adding the resultant result and the second bias modifier vector in position.

5. The intelligent commodity vending machine system with machine intelligence according to claim 4 is characterized in that: The first mask secondary subunit is used for: Calculating the square of the difference between the previous position feature value and the next position feature value of each position feature value in the first physiological state parameter time series associated auxiliary feature mapping mode representation vector to obtain the first physiological state parameter time series gradient amplitude representation vector; Calculating the absolute value of the square root of each first physiological state parameter temporal gradient amplitude in the first physiological state parameter temporal gradient amplitude representation vector, and inputting the obtained feature vector into the mask function for processing to obtain the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector; Among them, in the mask function, in response to the first physiological state parameter temporal gradient amplitude in the first physiological state parameter temporal gradient amplitude representation vector being greater than a predetermined threshold, the first physiological state parameter temporal gradient amplitude greater than the predetermined threshold is set to 1, otherwise it is set to 0 to obtain the masked first physiological state parameter temporal association auxiliary feature mapping pattern representation vector.

6. The intelligent commodity vending machine system with machine intelligence according to claim 5, characterized in that: The feature screening subunit comprises: A preliminary feature screening secondary subunit is used to perform preliminary feature screening on the multi-scale fusion feature map of the face state based on the masked first physiological state parameter time series associated auxiliary feature mapping mode representation vector and the hyperbolic tangent function to obtain a multi-scale fusion feature map of the face state after preliminary screening; The feature screening and enhancement secondary subunit is used to perform feature screening again on the multi-scale fusion feature map of the face state after the preliminary screening based on the masked second physiological state parameter time-series associated auxiliary feature mapping pattern representation vector and the Sigmoid function to obtain the multi-modal joint enhanced representation feature map of the body state.

7. The intelligent commodity vending machine system with machine intelligence according to claim 6, characterized in that: The preliminary feature screening secondary subunit is used to: use the masked first physiological state parameter time-series associated auxiliary feature mapping pattern representation vector as the channel weight vector, perform weighted enhancement on the multi-scale fusion feature map of the facial state of the human face along the channel dimension, and then input the weighted enhanced feature map into the hyperbolic tangent function for activation to obtain the multi-scale fusion feature map of the facial state of the human face after the preliminary screening.

8. The intelligent commodity vending machine system with machine intelligence according to claim 7, characterized in that: The feature screening and enhancement secondary subunit is used to: use the masked second physiological state parameter time-series associated auxiliary feature mapping pattern representation vector as the channel weight vector, perform weighted enhancement along the channel dimension on the multi-scale fusion feature map of the facial state after the preliminary screening, and input the weighted enhanced feature map into the Sigmoid function for activation to obtain the multi-modal joint enhanced representation feature map of the body state.

Images