Intelligent cabinet user perception method and system based on multi-modal perception

Abstract

This invention relates to the field of data processing technology, and more particularly to a user perception method and system for smart lockers based on multimodal perception. The method includes: acquiring vibration and gravity data of the smart locker during the current interaction time period, and acquiring background vibration and gravity data of the smart locker; determining the preliminary interaction complexity of the current interaction time period; determining the environmental noise suppression factor of the current interaction time period; determining the actual interaction complexity of the current interaction time period based on the preliminary interaction complexity and the environmental noise suppression factor; and determining the type of user interaction behavior based on the magnitude of the actual interaction complexity, thereby achieving user perception of the smart locker. This invention constructs a noise suppression model and combines it with multi-scale entropy analysis to more accurately distinguish between interaction and noise, improve perception accuracy, reduce false alarms and manual costs, and provide reliable user perception support for smart lockers.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a user perception method and system for smart cabinets based on multimodal perception. Background Technology

[0002] With the application of IoT technology in the new retail field, smart lockers, as core terminal devices, are distributed in shopping malls, communities, and transportation hubs. While the unattended nature of smart lockers improves operational efficiency, it also exposes them to the risk of improper user interaction or even malicious damage. Actual operational data shows that if user haste in closing the door, violent shaking caused by goods getting stuck, and malicious slapping and kicking behaviors cannot be accurately identified and stopped, it will significantly increase the equipment damage rate and the cost of damaged goods.

[0003] When conducting user perception of smart cabinets, the multivariate multiscale entropy (MMSE) algorithm based on dynamic analysis can usually be used to monitor the system status. The MMSE algorithm can theoretically capture the abnormal dynamic behavior of smart cabinets by evaluating the complexity and nonlinear coupling characteristics of multidimensional time series.

[0004] However, the disorder of the data in the traditional MMSE algorithm is entirely due to changes in the internal state of the system. But in the actual operating environment of smart lockers, there is complex environmental background noise, such as the resonance of the compressor inside the locker and the micro-vibration of the floor caused by vehicles passing by outside. Although the environmental background noise signal has low energy, it has a high degree of randomness and disorder. When there is only slight touch or no operation, the environmental background noise will cause the entropy benchmark calculated by the traditional MMSE algorithm to drift significantly. At this time, once the user's normal shopping action is superimposed with an artificially high benchmark entropy value, it is very easy to be misjudged as a highly complex violent destructive behavior, resulting in a large number of false alarms, affecting the accuracy of the user's perception of the smart locker, and increasing the cost of manual verification. Summary of the Invention

[0005] To address the problem that traditional MMSE algorithms suffer from entropy value drift due to environmental background noise, leading to false alarms, affecting the accuracy of smart cabinet user perception, and increasing manual verification costs, this invention provides a smart cabinet user perception method and system based on multimodal perception.

[0006] In a first aspect, the present invention provides a user perception method for smart cabinets based on multimodal perception, employing the following technical solution:

[0007] A user perception method for smart lockers based on multimodal perception includes: acquiring vibration and gravity data of the smart locker during the current interaction time period, and acquiring background vibration and gravity data of the smart locker; performing multi-scale coarse-grained processing on the vibration and gravity data to obtain multivariate sample entropy; determining the preliminary interaction complexity of the current interaction time period based on the multivariate sample entropy; determining the environmental noise suppression factor of the current interaction time period based on the difference between the vibration data and the background vibration data, and the difference in dispersion of the gravity data and the background gravity data; determining the actual interaction complexity of the current interaction time period based on the preliminary interaction complexity and the environmental noise suppression factor; and determining the type of interaction behavior of the current user based on the magnitude of the actual interaction complexity, thereby realizing user perception of the smart locker.

[0008] The beneficial effects are as follows: By acquiring background benchmarks of vibration and gravity data, an evaluation model for environmental noise suppression factors was constructed, effectively distinguishing between environmental background noise and real user interactions, overcoming the limitation of traditional MMSE algorithms in handling environmental noise interference; through multi-scale coarse-grained processing and multivariate sample entropy calculation, accurate assessment of the complexity of multi-dimensional time series was achieved, and the introduction of environmental noise suppression factors effectively eliminated the drift effect of background noise on the entropy benchmark; based on the determination of interaction behavior types according to the complexity of real interactions, the accuracy and reliability of user perception were improved, effectively reducing false alarms caused by environmental noise, reducing the cost of manual verification, and providing reliable technical support for accurate user perception of smart cabinets.

[0009] Furthermore, the method for constructing the current interaction time period is as follows: in response to detecting a sudden change in the amplitude of the vibration signal or receiving a door opening command, the current moment and 1.5 seconds before and after it are extracted to form the current interaction time period.

[0010] Furthermore, acquiring the vibration and gravity data of the smart cabinet during the current interaction time period includes: using a six-axis IMU sensor integrated on the core control board to collect the vibration data; and using a weighing sensor located at the bottom of the smart cabinet to collect the gravity data.

[0011] Furthermore, the acquisition of background vibration data and background gravity data of the smart cabinet includes: recording the vibration data and gravity data of the smart cabinet during the non-operational sleep period as the background vibration data and background gravity data of the smart cabinet; the background vibration data and background gravity data adopt a circular storage mechanism to ensure that the stored data is the latest data during the non-operational sleep period.

[0012] Furthermore, the complexity of the initial interaction satisfies:

[0013] In the formula, For the first The initial interaction complexity within a given interaction time period. The total number of the maximum set time scales. For the first The interaction time period is in the first Multivariate sample entropy at various time scales It is a natural exponential function.

[0014] The beneficial effects are as follows: by constructing a weighted sum that includes multi-scale, multi-variable sample entropy and exponential time weights, a comprehensive assessment of the initial interaction complexity is achieved. The exponential weights ensure that information at longer time scales receives more attention, reflecting the cumulative effect of complexity at different time scales, thus accurately reflecting the overall complexity characteristics of the interaction period.

[0015] Furthermore, the environmental noise suppression factor satisfies:

[0016] In the formula, For the first Environmental noise suppression factor for each interaction time period For the first The average value of vibration data within a single interaction time period This represents the average value of the background vibration data. For the first The variance of gravity data within each interaction time period The variance of the background gravity data. To prevent hyperparameters with a denominator of 0, It is the maximum-minimum normalization function.

[0017] The beneficial effects are as follows: by constructing a composite function that includes the difference in mean of vibration data and the difference in variance of gravity data, the degree of difference between the current interaction time period and the background environment in two dimensions of vibration and gravity is comprehensively evaluated. The normalization process ensures the reasonable range of factor values ​​and provides a reliable basis for noise suppression for calculating the complexity of real interaction.

[0018] Furthermore, the complexity of the actual interaction satisfies:

[0019] In the formula, For the first The actual complexity of interactions within a given time period For the first The initial interaction complexity within a given interaction time period. For the first Environmental noise suppression factor for each interaction time period This is the numerical amplification factor used to adjust the range of values ​​for the result.

[0020] The beneficial effects are as follows: by constructing a product term that includes the initial complexity, environmental noise suppression factor and amplification factor, a comprehensive evaluation of the complexity of real interaction is achieved. The environmental noise suppression factor effectively eliminates the influence of background noise on the complexity evaluation. When the environmental noise is large, the suppression factor decreases, reducing the complexity evaluation. The numerical amplification factor ensures a reasonable range of values ​​for the results and improves the accuracy of user interaction behavior recognition.

[0021] Furthermore, the determination of the current user's interaction behavior type based on the complexity of the actual interaction includes: in response to the complexity of the actual interaction being greater than a set threshold, determining that the current user has engaged in violent interaction behavior and executing an alarm operation; in response to the complexity of the actual interaction not being greater than the set threshold and detecting a change in gravity data, determining it as normal shopping behavior; in response to the complexity of the actual interaction not being greater than the set threshold and no change in gravity data, determining it as invalid vibration and ignoring it.

[0022] The beneficial effects are as follows: by constructing a hierarchical judgment logic based on the complexity of real interactions and changes in gravity data, it achieves accurate identification of different types of interaction behaviors, effectively distinguishes between violent interactions, normal shopping and environmental noise, and immediately triggers an alarm when the complexity exceeds the threshold to ensure timely prevention of destructive behavior. The dual judgment based on changes in gravity data avoids false alarms and improves the accuracy of user perception and the pertinence of system response.

[0023] Furthermore, the alarm operation includes: triggering the local voice module to play a preset warning voice and pushing abnormal alarm information to the cloud management platform.

[0024] Secondly, the present invention provides a smart cabinet user sensing system based on multimodal perception, which adopts the following technical solution:

[0025] A smart cabinet user sensing system based on multimodal perception includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the aforementioned smart cabinet user sensing method based on multimodal perception.

[0026] By adopting the above technical solution, a computer program is generated from the above-mentioned intelligent cabinet user perception method based on multimodal perception, and stored in the memory so that it can be loaded and executed by the processor. In this way, a terminal device can be made based on the memory and the processor for convenient use.

[0027] The present invention has the following technical effects:

[0028] (1) In view of the problem that the traditional MMSE algorithm is caused by environmental background noise, such as the resonance of the compressor in the cabinet and the micro-vibration of the floor, which causes the entropy value reference to drift and normal interaction actions to be misjudged as violent behavior, this invention calculates the environmental noise suppression factor more accurately by comparing the difference and dispersion of the current vibration and gravity data with the background data, and makes targeted corrections to the initial interaction complexity. The noise suppression factor will weaken the disorder interference caused by environmental noise, avoid the initial complexity caused by noise being artificially high, and enable the real interaction complexity to accurately remove the influence of noise, effectively solve the problem of a large number of false alarms caused by reference drift, and improve the accuracy of user perception.

[0029] (2) Breaking through the limitations of single data perception, it integrates multimodal information of vibration data and gravity data, and obtains multivariate sample entropy by combining multi-scale coarse-grained processing. It can not only fully capture the dynamic characteristics of smart cabinet user interaction, such as the vibration intensity when closing the door forcefully and the change of gravity distribution when shaking, but also adapt to the time scale differences of different interactive behaviors through multi-scale analysis, such as the characteristic difference between instantaneous patting and continuous shaking. Compared with traditional single-modal or single-scale methods, it can more accurately depict the essential characteristics of interactive behavior and provide a more comprehensive and reliable basis for judging user behavior type.

[0030] (3) By assessing the complexity of real interactions, the boundaries between normal shopping actions, such as closing the door smoothly and picking up and putting away goods normally, and abnormal destructive behaviors, such as violent shaking, slapping and kicking, can be clearly defined. The real complexity of normal interactions is relatively low, while the real complexity of abnormal behaviors is significantly higher. This avoids the ineffective manual review caused by false alarms in traditional methods and reduces operating costs. At the same time, after accurately identifying abnormal behaviors, early warning or intervention measures can be triggered in a timely manner to reduce the risk of equipment damage and goods jamming and damage, and ensure the stable operation of smart cabinets. Attached Figure Description

[0031] Figure 1 This is a flowchart of a user perception method for a smart cabinet based on multimodal perception, according to an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram comparing the complexity of the MMSE algorithm before and after optimization in a user perception method for a smart cabinet based on multimodal perception according to an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This invention discloses a user perception method for smart cabinets based on multimodal perception, referring to... Figure 1 This includes steps S001-S005:

[0035] S001: Obtain the vibration and gravity data of the smart cabinet during the current interaction time period, and obtain the background vibration and background gravity data of the smart cabinet.

[0036] Specifically, at the hardware level of the smart cabinet, a six-axis IMU sensor integrated on the main control board, including a three-axis accelerometer and a three-axis gyroscope, collects vibration data of the cabinet. For example, the sampling frequency is set to 100Hz. At the same time, a bottom weighing sensor collects gravity data of the cabinet. The system maintains a background noise buffer pool in memory. When the system is in a sleep state without user operation, it cyclically stores the vibration and gravity data of the most recent 10 seconds as background vibration and background gravity data. This ensures that the algorithm always uses the current environmental state as the comparison benchmark, rather than a fixed factory setting, thereby adapting to the environmental differences of different installation locations. When a sudden change in vibration data or a door opening command is detected, the system extracts data for 1.5 seconds before and after the trigger time as the data for the current interaction period.

[0037] S002: Determine the initial interaction complexity for the current interaction time period.

[0038] It should be noted that this step assesses the complexity of the current interaction from a dynamic perspective. When the smart cabinet is subjected to violent interaction, such as violent shaking or continuous slapping, its physical system enters a nonlinear chaotic state, and the resulting vibration signals exhibit high irregularity and unpredictability across multiple time scales. Therefore, by calculating the sum of the multi-scale entropy values ​​of the multivariate data, the larger the sum, the greater the degree of instability in the initial system, indicating that the current interaction is more violent and abnormal.

[0039] Multi-scale coarse-grained processing is performed on vibration and gravity data to obtain multivariate sample entropy. Based on the multivariate sample entropy, the preliminary interaction complexity of the current interaction time period is determined.

[0040] Specifically, the complexity of the initial interaction satisfies:

[0041] ;

[0042] In the formula, For the first The initial interaction complexity within a given interaction time period. The total number of the maximum set time scales. For the first The interaction time period is in the first Multivariate sample entropy at various time scales It is a natural exponential function.

[0043] in, The larger the value, the higher the value. At any given time scale, the more complex the dynamic state of the smart cabinet system, the more severe the current disturbance. It is a scale-dependent The increased weighting coefficients are due to the fact that violent acts typically induce low-frequency resonances in the cabinet structure, causing it to maintain a high entropy value over a longer time scale. In contrast, ordinary high-frequency noise exhibits faster entropy decay after coarsening. Therefore, the weighting coefficients are increased. The feature contribution used to amplify violent behavior can effectively suppress high-frequency noise.

[0044] S003: Determine the environmental noise suppression factor for the current interaction time period.

[0045] It should be noted that when the smart locker is in an environment with high ambient noise, the inherent randomness of the background noise can lead to an artificially inflated complexity of the initial interactions within each interaction period. To address this technical issue, this step requires analyzing the fundamental physical differences between this special case and regular interactive behavior. Regular violent behavior is inevitably accompanied by a significant energy surge, and regular shopping behavior is inevitably accompanied by a significant change in gravity. However, the special case caused by ambient noise manifests as follows: while the energy, i.e., vibration data, fluctuates, the increase in energy relative to the background vibration data is not significant, and there is no logical support from a change in gravity. Therefore, if the energy increase of the current signal relative to the background noise is more significant, and the increase in gravity change is also more significant, then the entropy value of the current signal is more reliable and should not be suppressed; conversely, it should be considered noise and suppressed.

[0046] Based on the difference between vibration data and background vibration data, and the difference in dispersion between gravity data and background gravity data, the environmental noise suppression factor for the current interaction period is determined.

[0047] Specifically, the environmental noise suppression factor satisfies:

[0048] ;

[0049] In the formula, For the first Environmental noise suppression factor for each interaction time period For the first The average value of vibration data within a single interaction time period This represents the average value of the background vibration data. For the first The variance of gravity data within each interaction time period The variance of the background gravity data. For example, to prevent hyperparameters with a denominator of 0, , It is the maximum-minimum normalization function.

[0050] in, This reflects the increase factor of vibration energy relative to background noise. The larger the value, the greater the likelihood that the current vibration was caused by forceful user operation. The larger it is. This reflects the growth factor of gravity changes relative to background fluctuations. A larger value indicates a higher probability of a substantial action involving the handling of goods. The smaller the energy increment and gravity increment, the greater the influence of environmental noise over an interaction period. The smaller the value, the better, thus suppressing the artificially high baseline entropy.

[0051] S004: Determine the actual interaction complexity of the current interaction time period based on the initial interaction complexity and the environmental noise suppression factor.

[0052] It should be noted that this step uses an environmental noise suppression factor to correct the initial interaction complexity in order to restore the true complexity of user behavior. The smaller the environmental noise suppression factor for each interaction time period, the more environmental noise components are contained in the initial interaction complexity of each interaction time period, and it needs to be significantly attenuated. The larger the environmental noise suppression factor for each interaction time period, the greater the possibility that the initial interaction complexity of each interaction time period truly reflects the disorder of interaction behavior, and it should be retained.

[0053] Specifically, the complexity of the actual interaction satisfies the following:

[0054] ;

[0055] In the formula, For the first The actual complexity of interactions within a given time period For the first The initial interaction complexity within a given interaction time period. For the first Environmental noise suppression factor for each interaction time period This is a numerical amplification factor used to adjust the range of result values, for example, .

[0056] S005: Determine the type of current user's interaction behavior based on the complexity of the actual interaction to achieve user perception of the smart cabinet.

[0057] Specifically, determining the type of the current user's interaction behavior based on the complexity of the actual interaction includes:

[0058] If the complexity of the actual interaction exceeds a set threshold, it will determine that the current user is engaging in violent interaction behavior and trigger an alarm.

[0059] If the complexity of the actual interaction does not exceed a set threshold and a change in gravity data is detected, it is determined to be a normal shopping behavior;

[0060] If the complexity of the actual interaction does not exceed a set threshold and there is no change in gravity data, it is judged as an invalid vibration and ignored.

[0061] Implementers can set a threshold value based on the specific implementation situation, for example, 0.85.

[0062] Specifically, the alarm operation includes:

[0063] The system triggers the local voice module to play preset warning messages, such as "Please shop responsibly" and "Do not shake the cabinet," and pushes abnormal alarm information to the cloud management platform.

[0064] like Figure 2 As shown, the left column represents the interaction complexity obtained by the traditional MMSE algorithm, which only measures the signal disorder; the right column represents the actual interaction complexity obtained by the MMSE algorithm optimized by this invention, which has a strong ability to resist environmental interference, i.e., prevents false alarms. The traditional MMSE algorithm cannot distinguish between normal shopping with environmental noise and mixed noise and malicious damage. This causes normal shopping with environmental noise and mixed noise to exceed the set threshold in terms of interaction complexity due to high disorder, thus triggering false alarms. The optimized MMSE algorithm suppresses normal shopping with environmental noise and mixed noise, thereby avoiding false alarms, while the actual interaction complexity corresponding to violent damage remains high to ensure accurate alarm for real violent behavior.

[0065] This invention also discloses a smart cabinet user sensing system based on multimodal perception, including a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a smart cabinet user sensing method based on multimodal perception according to the present invention.

[0066] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0067] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A user perception method for smart cabinets based on multimodal perception, characterized in that, include: Obtain vibration and gravity data of the smart cabinet during the current interaction time period, and obtain background vibration and background gravity data of the smart cabinet; Multi-scale coarse-grained processing is performed on vibration and gravity data to obtain multivariate sample entropy. Based on the multivariate sample entropy, the preliminary interaction complexity of the current interaction time period is determined; the preliminary interaction complexity satisfies: ； For the first The initial interaction complexity within a given interaction time period. The total number of the maximum set time scales. For the first The interaction time period is in the first Multivariate sample entropy at various time scales It is a natural exponential function; Based on the difference between vibration data and background vibration data, and the difference in dispersion between gravity data and background gravity data, the environmental noise suppression factor for the current interaction period is determined. The environmental noise suppression factor satisfies: ； For the first Environmental noise suppression factor for each interaction time period For the first The average value of vibration data within a single interaction time period This represents the average value of the background vibration data. For the first The variance of gravity data within each interaction time period The variance of the background gravity data. To prevent hyperparameters with a denominator of 0, It is the maximum-minimum normalization function; Based on the initial interaction complexity and the environmental noise suppression factor, the actual interaction complexity for the current interaction time period is determined; the actual interaction complexity satisfies: ； For the first The actual complexity of interactions within a given time period This is a numerical amplification factor used to adjust the range of values ​​for the result; The type of user interaction behavior is determined based on the complexity of the actual interaction, enabling the smart cabinet to perceive the user's behavior.

2. The user perception method for a smart cabinet based on multimodal perception according to claim 1, characterized in that, The method for constructing the current interaction time period is as follows: In response to the detection of a sudden change in the amplitude of the vibration signal or the receipt of a door opening command, the current moment and 1.5 seconds before and after it are captured to form the current interaction time period.

3. The user perception method for a smart cabinet based on multimodal perception according to claim 1, characterized in that, The acquisition of vibration and gravity data of the smart cabinet during the current interaction time period includes: The vibration data is acquired using a six-axis IMU sensor integrated on the core control board; The gravity data is collected using a weighing sensor located at the bottom of the smart cabinet.

4. The user perception method for a smart cabinet based on multimodal perception according to claim 1, characterized in that, The acquisition of background vibration data and background gravity data of the smart cabinet includes: The vibration and gravity data collected during the non-operational sleep period of the smart cabinet are recorded as the background vibration data and background gravity data of the smart cabinet. Background vibration data and background gravity data are stored in a circular manner to ensure that the stored data is the latest data from the inactive sleep period.

5. The user perception method for a smart cabinet based on multimodal perception according to claim 1, characterized in that, The method of determining the type of current user interaction behavior based on the complexity of the actual interaction includes: If the complexity of the actual interaction exceeds a set threshold, it will determine that the current user is engaging in violent interaction behavior and trigger an alarm. If the complexity of the actual interaction does not exceed a set threshold and a change in gravity data is detected, it is determined to be a normal shopping behavior; If the complexity of the actual interaction does not exceed a set threshold and there is no change in gravity data, it is judged as an invalid vibration and ignored.

6. The user perception method for a smart cabinet based on multimodal perception according to claim 5, characterized in that, The alarm execution operation includes: The system triggers the local voice module to play a preset warning message and pushes anomaly alarm information to the cloud management platform.

7. A smart cabinet user sensing system based on multimodal perception, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement a user perception method for a smart cabinet based on multimodal perception according to any one of claims 1-6.

Images