Intelligent shopping cart autonomous navigation system and implementation method thereof

Abstract

The invention discloses an intelligent shopping cart autonomous navigation system and an implementation method thereof.The intelligent shopping cart autonomous navigation system comprises a sensor assembly, the sensor assembly is composed of a data acquisition module, a camera module, a laser radar module, an ultrasonic module, an infrared module and an acquisition output module, and the data acquisition module is in control connection with the camera module; the camera module is in control connection with an acquisition output module; according to the invention, the sensor assembly is used, and a plurality of sensors are adopted, so that a complex dynamic environment can be accurately sensed, and untimely obstacle avoidance or unreasonable path planning is not easily caused; the user identification component is used to actively adapt to user requirements, and personalized services are provided; by using the path planning component, the path planning algorithm is high in efficiency in a dynamic and changeable shopping environment, and can be adjusted in real time, so that the shopping efficiency is guaranteed; by using the man-machine interaction component which comprises a plurality of interaction modes, the emotion and intention of the user can be understood so as to provide humanized service.

Description

Technical Field

[0001] The present invention relates to the technical field of smart shopping carts, and in particular to a smart shopping cart autonomous navigation system and an implementation method thereof. Background Art

[0002] With the digital transformation of the retail industry, the research and development of smart shopping equipment has become a hot topic; traditional shopping carts lack intelligent functions and cannot meet the needs of modern consumers for efficient, convenient and safe shopping; some existing smart shopping carts have limitations in navigation, obstacle avoidance and human-computer interaction. First, they usually only use a single or a few sensors, which cannot accurately perceive complex dynamic environments, easily leading to untimely obstacle avoidance or unreasonable path planning; second, existing technologies rarely consider the prediction of users' shopping behavior, and cannot actively adapt to user needs and provide personalized services; third, traditional path planning algorithms are inefficient in dynamic and changeable shopping environments and cannot be adjusted in real time, affecting shopping efficiency; fourth, the interaction method is single, lacks understanding of user emotions and intentions, and it is difficult to provide humanized services; therefore, it is necessary to design an intelligent shopping cart autonomous navigation system and its implementation method. Summary of the invention

[0003] The purpose of the present invention is to provide an intelligent shopping cart autonomous navigation system and an implementation method thereof to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an intelligent shopping cart autonomous navigation system, including a sensor component, wherein the sensor component is composed of a data acquisition module, a camera module, a laser radar module, an ultrasonic module, an infrared module and an acquisition output module, the data acquisition module is controlled to be connected to the camera module, the camera module is controlled to be connected to the acquisition output module, the acquisition output module is controlled to be connected to the laser radar module, the laser radar module is controlled to be connected to the data acquisition module, the data acquisition module is controlled to be connected to the ultrasonic module, the ultrasonic module is controlled to be connected to the acquisition output module, the acquisition output module is controlled to be connected to the infrared module, the infrared module is controlled to be connected to the data acquisition module, the acquisition output module is controlled to be connected to the data processing module, and the data processing module is controlled to be connected to the prediction receiving module in the behavior prediction component.

[0005] As a further technical solution of the present invention, the behavior prediction component is composed of a prediction receiving module, a prediction processing module and a prediction output module. The prediction receiving module is controlled to be connected with the prediction processing module, the prediction processing module is controlled to be connected with the prediction output module, the prediction output module is controlled to be connected with the path receiving module in the path planning component, and the path receiving module is controlled to be connected with the data processing module.

[0006] As a further technical solution of the present invention, the path planning component is composed of a path receiving module, a path calculation module and a path output module. The path receiving module is controlled to be connected to the path calculation module, the path calculation module is controlled to be connected to the path output module, the path output module is controlled to be connected to the drive control module, the drive control module is controlled to be connected to the autonomous following module, the autonomous following module is controlled to be connected to the prediction output module, the autonomous following module is controlled to be connected to the user guidance module, the user guidance module is controlled to be connected to the interactive output module in the human-computer interaction component, the human-computer interaction component is composed of a touch screen module, a microphone module, a speaker module, an interactive processing module, a speech recognition module, a gesture recognition module, an emotion recognition module and an interactive output module, the touch screen module is controlled to be connected to the interactive processing module, the interactive processing module is controlled to be connected to the microphone module, the interactive processing module is controlled to be connected to the speaker module, the interactive output module is controlled to be connected to the interactive processing module, the interactive processing module is controlled to be connected to the speech recognition module, the interactive processing module is controlled to be connected to the gesture recognition module, the interactive processing module is controlled to be connected to the emotion recognition module, and the interactive processing module is controlled to be connected to the recognition output module in the user recognition component.

[0007] As a further technical solution of the present invention, the user identification component is composed of a facial recognition module, an identity comparison module, a history analysis module, a personality recommendation module and an identification output module. The facial recognition module controls the connection with the identity comparison module, the identity comparison module controls the connection with the history analysis module, the history analysis module controls the connection with the personality recommendation module, and the personality recommendation module controls the connection with the identification output module.

[0008] The method for realizing autonomous navigation of a smart shopping cart includes the following steps: step 1, multimodal environment perception and data fusion; step 2, user behavior prediction; step 3, path planning and drive control; step 4, autonomous following; step 5, human-computer interaction and user identification;

[0009] In the above step 1, multiple sensors are used to collect environmental data and perform data fusion;

[0010] In the above step 2, the user's next behavior is predicted;

[0011] In the above step 3, a route is planned to the destination and the shopping cart is driven and controlled;

[0012] In the above step 4, the shopping cart is controlled to follow the user;

[0013] In the above step 5, the user interacts with the shopping cart and the user is identified.

[0014] As a further technical solution of the present invention, in the step 1, the start and stop of data acquisition are controlled by the data acquisition module, the RGB image is acquired by the camera module with a resolution of 1920×1080 and a frame rate of 30 frames / second for acquiring visual information, the 360-degree point cloud data is acquired by the laser radar module with a scanning frequency of 10Hz for acquiring three-dimensional spatial information of the environment, and the close-range obstacles are detected by the ultrasonic module and the infrared module with a sampling frequency of 50Hz for supplementing the blind spots of vision and laser radar; then all the acquired data are transmitted to the data processing module through the acquisition output module, in which the data of different sensors are synchronized by using the timestamp, and the data of each sensor are converted to a unified coordinate system by using the external parameter calibration; the feature points {f i}, with scale invariance and rotation invariance; extract local plane features and curvature information from point cloud data to form a feature set {p j}; According to the confidence w of the sensor data i , weight the data of each sensor, and the weight calculation formula is:

[0015]

[0016] in, is the variance of the i-th sensor data;

[0017] The extended Kalman filter (EKF) is used for data fusion. The state vector x contains the position and velocity information, and the observation vector z contains the measurement values ​​of each sensor. The prediction steps are:

[0018] x kk-1 =F k-1 x k-1k-1 +B k-1 u k-1

[0019]

[0020] The update steps are:

[0021]

[0022] x k|k =x k|k-1 +K k (z k -H k x k|k-1 )

[0023] P k|k =(IK k H k ) k|k-1

[0024] Among them, F k is the state transfer matrix, H k is the observation matrix, Q k is the process noise covariance matrix, R k is the observation noise covariance matrix;

[0025] The fused data is used to construct an environmental map, which is represented by a raster map or a topological map. The environmental model is used for subsequent path planning and obstacle avoidance.

[0026] As a further technical solution of the present invention, in step 2, the data processed by the data processing module is received by the prediction receiving module and transmitted to the prediction processing module. In the prediction processing module, the network structure of the model adopts a double-layer LSTM network, and the input sequence is the user's historical behavior data {x1, x2, ..., x t}, each x i Including features such as position, speed, and residence time, adding an attention mechanism to the output layer of LSTM to calculate the weight α of each time step i , to capture key behavioral characteristics:

[0027] e i =v T tanh(W h h i +W s s)

[0028]

[0029] Among them, h i is the hidden state of the i-th time step, s is the context vector, v, W h , W s is a learnable parameter;

[0030] Using the cross entropy loss function L:

[0031]

[0032] Among them, y k is the true label, The probability predicted by the model;

[0033] During the training process of the model, optimization and regularization are performed. The Adam optimizer is used, the learning rate η = 0.001, the L2 regularization term is added, the coefficient is λ = 0.0001, the number of iterations is 50 rounds, and the batch size is 64. When making predictions, the x value of the user behavior at the current moment is collected. t , normalize the data, subtract the mean and divide by the standard deviation, input the preprocessed data into the model, and calculate the output That is, the probability distribution of the user's next behavior, and the prediction results are output through the prediction output module.

[0034] As a further technical solution of the present invention, in step 3, the data information output by the prediction output module and the data processing module is received by the path receiving module and transmitted to the path calculation module. In the path calculation module, the real-time environment map provided by the data processing module is used, including the obstacle position {O i} and the motion state of dynamic objects to model the environment; the improved dynamic A-Star algorithm is used, and the heuristic function h(n) considers the predicted position of dynamic obstacles to perform initial path planning; the state space S contains the position (x, y), speed v, direction θ, surrounding environment information, etc. of the shopping cart, and the action space A contains the actions that the shopping cart can perform, such as acceleration, deceleration, left turn, right turn, stop, etc., and the reward function R(s,a) is used:

[0035] R(s,a)=w1.r goal +w2.r collision +w3.r efficiency

[0036] Among them, r goal : Reward for reaching the goal (+100 points), r collision : Collision penalty (-100 points), r efficiency : Path efficiency reward, the shorter the path, the higher the reward, w1, w2, w3 are weight parameters;

[0037] The policy network uses a deep Q network (DQN), the network structure is a three-layer fully connected network, the number of hidden layer units is 128 and 64 respectively, the activation function is ReLU, and a 20,000-capacity experience replay pool is used for random sampling small batch training. The discount factor γ is 0.95, and the exploration strategy is ε-greedy. ε decreases linearly from 1 to 0.1. The results are output to the drive control module through the path output module. A motion model is set in the drive control module, using differential drive. The motion model is:

[0038]

[0039] Where v is the linear velocity and w is the angular velocity;

[0040] Design the fuzzy controller, input variables: distance error e d =d desired -d actual , angle error e θ =θ desired -θ actual ; Output variables: linear velocity adjustment Δv, angular velocity adjustment Δw; Fuzzy rules: If e dBig and e θ If e is large, Δv is small and Δw is large; if e is large, Δv is small and Δw d small and e θ If Δv is small, Δw is small; map the input variables to the fuzzy set, calculate the applicability of each rule, synthesize the output of each rule, and use the center of gravity method to calculate the precise control output; the control process is to obtain the current position and posture of the shopping cart in real time, calculate the error, input the fuzzy controller, obtain Δv and Δw, and adjust the motor speed and direction.

[0041] As a further technical solution of the present invention, in the step 4, a YOLOv5s target detection model is deployed in the autonomous following module, and 5000 pictures of users labeled with different postures and environments are used as training data. The training parameters are set as follows: learning rate η=0.001, batch size is 16, and the number of iterations is 100 rounds; in the target tracking process, the DeepSORT algorithm is adopted, and ResNet50 is used to extract the user's appearance feature vector f, and the Kalman filter is used to predict the next position of the target; the safety distance in the following strategy is kept at about 1 meter, and the threshold d is set. safe = 1 meter, according to the target position (x target ,y target ) and the shopping cart position (x cart ,y cart ) calculates the error and adjusts the speed and direction.

[0042] As a further technical solution of the present invention, in the step 5, a Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module, and the training data of the model includes Mandarin and English speech data sets with a total length of 1000 hours. In the recognition process, the speech signal is first processed, the Mel frequency cepstral coefficient (MFCC) is extracted, and the BeamSearch algorithm is used to decode the output text; BERT is used for semantic understanding in the interaction processing module, and the classification model is used to map the input text to a predefined intent set {I1, I2, ..., I n}, extract key parameters such as product name and quantity during the slot filling process; extract voice features such as tone, energy, and rhythm through the emotion recognition module, use the emotion classification model to judge emotions, and use the CNN model for classification in combination with the captured facial expressions; set up functional interfaces such as product search, recommendation, and shopping list in the touch screen module, and use them in conjunction with the voice recognition module and gesture recognition module to support voice, touch, and gesture operations; deploy the ArcFace model in the facial recognition module, use the improved Softmax loss function to enhance the classification boundary, and train the CNC with 1 million face images covering different ages and genders; use MTCNN to locate the face position in the identity comparison module, input the face image into the ArcFace model, obtain the feature vector f, calculate the cosine similarity with the feature vector stored in the database, and use the threshold to determine whether it is a registered user:

[0043]

[0044] Use the historical analysis module and the personalized recommendation module for personalized recommendations. The algorithm adopts a hybrid recommendation system that combines collaborative filtering and content recommendation. User-based collaborative filtering is used to calculate the similarity between users and recommend products that similar users like. Similarity calculation:

[0045]

[0046] Among them, r u,i is the rating of product i by user u, is the average rating of user u;

[0047] Matching is performed based on product attributes and user preferences, and sensitive data is encrypted and stored using AES, with explicit consent from the user obtained through user authorization before use.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows: the intelligent shopping cart autonomous navigation system and its implementation method control the start and stop of data acquisition through the data acquisition module, collect RGB images through the camera module, with a resolution of 1920×1080 and a frame rate of 30 frames per second, for obtaining visual information, collect 360-degree point cloud data through the laser radar module, with a scanning frequency of 10Hz, for obtaining three-dimensional spatial information of the environment, detect close-range obstacles through the ultrasonic module and the infrared module, with a sampling frequency of 50Hz, for supplementing the blind spots of vision and laser radar; multiple sensors are used to accurately perceive complex dynamic environments, and it is not easy to cause untimely obstacle avoidance or unreasonable path planning; MTCNN is used to locate the face position in the identity comparison module, and the face image is used to identify the face position. The ArcFace model is input to obtain the feature vector f, and the cosine similarity with the feature vector stored in the database is calculated. The threshold determines whether it is a registered user, and the history analysis module and the personalized recommendation module are used for personalized recommendation. The algorithm adopts a hybrid recommendation system, combining collaborative filtering and content recommendation, in which user-based collaborative filtering is used to calculate the similarity between users, recommend products that similar users like, match based on the attributes of the products and the preferences of the users, actively adapt to user needs, and provide personalized services; the data information output by the prediction output module and the data processing module is received by the path receiving module, and transmitted to the path calculation module. In the path calculation module, the real-time environment map provided by the data processing module is used, including the obstacle position {O i} and the motion state of dynamic objects to model the environment; the improved dynamic A-Star algorithm is used, and the heuristic function h(n) considers the predicted position of dynamic obstacles to perform initial path planning; the state space S contains the position (x, y), speed v, direction θ, surrounding environment information, etc. of the shopping cart, and the action space A contains the actions that the shopping cart can perform, such as acceleration, deceleration, left turn, right turn, stop, etc., and the reward function R(s, a) is used. The policy network uses a deep Q network (DQN), and the network structure is a three-layer fully connected network. The number of hidden layer units is 128 and 64 respectively, and the activation function is ReLU. A small batch training with random sampling is used using an experience replay pool with a capacity of 20,000. The discount factor γ = 0.95, and the exploration strategy is: ε-gree dy, ε decreases linearly from 1 to 0.1, and the result is output to the drive control module through the path output module. The path planning algorithm is highly efficient in a dynamic and changeable shopping environment and can be adjusted in real time to ensure shopping efficiency. A Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module. The training data of the model includes Mandarin and English speech data sets with a total duration of 1000 hours. In the recognition process, the speech signal is first processed to extract the Mel-frequency cepstral coefficient (MFCC), and the BeamSearch algorithm is used to decode the output text. BERT is used for semantic understanding in the interaction processing module, and a classification model is used to map the input text to a predefined intent set {I1,I2,...,I n}, extract key parameters such as product name and quantity during the slot filling process; extract voice characteristics such as tone, energy, and rhythm through the emotion recognition module, use the emotion classification model to judge emotions, and use the captured facial expressions to classify them using the CNN model; the touch screen module is equipped with functional interfaces such as product search, recommendation, and shopping list, which are used in conjunction with the voice recognition module and gesture recognition module to support voice, touch, and gesture operations, including a variety of interaction methods, which can understand the user's emotions and intentions to provide humanized services. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall architecture diagram of the present invention;

[0050] Figure 2 A flow chart of the sensor assembly in the present invention;

[0051] Figure 3 is a flow chart of the behavior prediction component in the present invention;

[0052] Figure 4 A flow chart of the path planning component in the present invention;

[0053] Figure 5 It is a flow chart of the human-computer interaction component in the present invention;

[0054] Figure 6 A flowchart of a user identification component in the present invention;

[0055] Figure 7 is a flow chart of the method of the present invention;

[0056] In the figure: 1. Sensor component; 2. Data processing module; 3. Behavior prediction component; 4. Path planning component; 5. Drive control module; 6. Autonomous following module; 7. User guidance module; 8. Human-computer interaction component; 9. User identification component; 11. Data acquisition module; 12. Camera module; 13. LiDAR module; 14. Ultrasonic module; 15. Infrared module; 16. Acquisition output module; 31. Prediction receiving module; 32. Prediction processing module; 33. Prediction output module; 41. Path receiving module; 42. Path calculation module; 43. Path output module; 81. Touch screen module; 82. Microphone module; 83. Speaker module; 84. Interaction processing module; 85. Speech recognition module; 86. Gesture recognition module; 87. Emotion recognition module; 88. Interaction output module; 91. Facial recognition module; 92. Identity comparison module; 93. History analysis module; 94. Personality recommendation module; 95. Recognition output module. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Please see attached Figure 1 -Attached Figure 6An embodiment of the present invention provides an intelligent shopping cart autonomous navigation system, including a sensor component 1, the sensor component 1 is composed of a data acquisition module 11, a camera module 12, a laser radar module 13, an ultrasonic module 14, an infrared module 15 and a collection output module 16, the data acquisition module 11 is controlled to be connected to the camera module 12, the camera module 12 is controlled to be connected to the collection output module 16, the collection output module 16 is controlled to be connected to the laser radar module 13, the laser radar module 13 is controlled to be connected to the data acquisition module 11, the data acquisition module 11 is controlled to be connected to the ultrasonic module 14, the ultrasonic module 14 is controlled to be connected to the collection output module 16, the collection output module 16 is controlled to be connected to the infrared module 15, and the infrared module The block 15 is controlled to be connected to the data acquisition module 11, the acquisition output module 16 is controlled to be connected to the data processing module 2, the data processing module 2 is controlled to be connected to the prediction receiving module 31 in the behavior prediction component 3, the behavior prediction component 3 is composed of the prediction receiving module 31, the prediction processing module 32 and the prediction output module 33, the prediction receiving module 31 is controlled to be connected to the prediction processing module 32, the prediction processing module 32 is controlled to be connected to the prediction output module 33, the prediction output module 33 is controlled to be connected to the path receiving module 41 in the path planning component 4, the path receiving module 41 is controlled to be connected to the data processing module 2, the path planning component 4 is composed of the path receiving module 41, the path calculation module 42 and the path output module 43, the path receiving module 4 1 is controlled and connected to a path calculation module 42, the path calculation module 42 is controlled and connected to a path output module 43, the path output module 43 is controlled and connected to a driving control module 5, the driving control module 5 is controlled and connected to an autonomous following module 6, the autonomous following module 6 is controlled and connected to the prediction output module 33, the autonomous following module 6 is controlled and connected to a user guidance module 7, the user guidance module 7 is controlled and connected to an interactive output module 88 in a human-computer interaction component 8, the human-computer interaction component 8 is composed of a touch screen module 81, a microphone module 82, a speaker module 83, an interactive processing module 84, a speech recognition module 85, a gesture recognition module 86, an emotion recognition module 87 and an interactive output module 88, the touch screen module 81 is controlled and connected to an interactive processing module Block 84, the interaction processing module 84 is controlled to be connected to the microphone module 82, the interaction processing module 84 is controlled to be connected to the speaker module 83, the interaction output module 88 is controlled to be connected to the interaction processing module 84, the interaction processing module 84 is controlled to be connected to the speech recognition module 85, the interaction processing module 84 is controlled to be connected to the gesture recognition module 86, the interaction processing module 84 is controlled to be connected to the emotion recognition module 87, the interaction processing module 84 is controlled to be connected to the recognition output module 95 in the user recognition component 9, the user recognition component 9 is composed of a facial recognition module 91, an identity comparison module 92, a history analysis module 93, a personality recommendation module 94 and a recognition output module 95, the facial recognition module 91 is controlled to be connected to the identity comparison module 92,The identity comparison module 92 is connected to the history analysis module 93, the history analysis module 93 is connected to the personality recommendation module 94, and the personality recommendation module 94 is connected to the recognition output module 95.

[0059] Please see attached Figure 7 , an embodiment provided by the present invention: a method for realizing autonomous navigation of an intelligent shopping cart, comprising the following steps: step 1, multimodal environment perception and data fusion; step 2, user behavior prediction; step 3, path planning and drive control; step 4, autonomous following; step 5, human-computer interaction and user identification;

[0060] In the above step 1, multiple sensors are used to collect environmental data and perform data fusion. The start and stop of data collection are controlled by the data acquisition module 11. The camera module 12 collects RGB images with a resolution of 1920×1080 and a frame rate of 30 frames per second to obtain visual information. The laser radar module 13 collects 360-degree point cloud data with a scanning frequency of 10 Hz to obtain three-dimensional spatial information of the environment. The ultrasonic module 14 and the infrared module 15 detect close obstacles with a sampling frequency of 50 Hz to supplement the blind spots of vision and laser radar. Subsequently, all the collected data are transmitted to the data processing module 2 through the acquisition output module 16. In the data processing module 2, the data of different sensors are synchronized by using timestamps, and the data of each sensor are converted to a unified coordinate system by using external parameter calibration. The SIFT algorithm is used to extract feature points {f i}, with scale invariance and rotation invariance; extract local plane features and curvature information from point cloud data to form a feature set {p j}; According to the confidence w of the sensor data i , weight the data of each sensor, and the weight calculation formula is:

[0061]

[0062] in, is the variance of the i-th sensor data;

[0063] The extended Kalman filter (EKF) is used for data fusion. The state vector x contains the position and velocity information, and the observation vector z contains the measurement values ​​of each sensor. The prediction steps are:

[0064] x kk-1 =F k-1 x k-1k-1 +B k-1 u k-1

[0065]

[0066] The update steps are:

[0067]

[0068] x kk =x kk-1 +K k (z k -H k x kk-1 )

[0069] P kk =(IK k H k ) kk-1

[0070] Among them, F k is the state transfer matrix, H k is the observation matrix, Q k is the process noise covariance matrix, R k is the observation noise covariance matrix;

[0071] The fused data is used to construct an environmental map, which is represented by a raster map or a topological map. The environmental model is used for subsequent path planning and obstacle avoidance.

[0072] In the above step 2, the user's next behavior is predicted, and the data processed by the data processing module 2 is received by the prediction receiving module 31 and transmitted to the prediction processing module 32. In the prediction processing module 32, the network structure of the model adopts a two-layer LSTM network, and the input sequence is the user's historical behavior data {x1, x2, ..., x t}, each x i Including features such as position, speed, and residence time, adding an attention mechanism to the output layer of LSTM to calculate the weight α of each time step i , to capture key behavioral characteristics:

[0073] e i =v T tanh(W h h i +W s s)

[0074]

[0075] Among them, h i is the hidden state of the i-th time step, s is the context vector, v, W h , W s is a learnable parameter;

[0076] Using the cross entropy loss function L:

[0077]

[0078] Among them, y k is the true label, The probability predicted by the model;

[0079] During the training process of the model, optimization and regularization are performed. The Adam optimizer is used, the learning rate η = 0.001, the L2 regularization term is added, the coefficient is λ = 0.0001, the number of iterations is 50 rounds, and the batch size is 64. When making predictions, the x value of the user behavior at the current moment is collected. t , normalize the data, subtract the mean and divide by the standard deviation, input the preprocessed data into the model, and calculate the output That is, the probability distribution of the user's next behavior, and the prediction result is output through the prediction output module 33;

[0080] In the above step 3, the path to the destination is planned, the shopping cart is driven and controlled, the data information output by the prediction output module 33 and the data processing module 2 is received by the path receiving module 41, and transmitted to the path calculation module 42. In the path calculation module 42, the real-time environment map provided by the data processing module 2, including the obstacle position {O i} and the motion state of dynamic objects to model the environment; the improved dynamic A-Star algorithm is used, and the heuristic function h(n) considers the predicted position of dynamic obstacles to perform initial path planning; the state space S contains the position (x, y), speed v, direction θ, surrounding environment information, etc. of the shopping cart, and the action space A contains the actions that the shopping cart can perform, such as acceleration, deceleration, left turn, right turn, stop, etc., and the reward function R(s,a) is used:

[0081] R(s,a)=w1.r goal +w2.r collision +w3.r efficiency

[0082] Among them, r goal : Reward for reaching the goal (+100 points), r collision : Collision penalty (-100 points), r efficiency : Path efficiency reward, the shorter the path, the higher the reward, w1, w2, w3 are weight parameters;

[0083] The policy network uses a deep Q network (DQN), the network structure is a three-layer fully connected network, the number of hidden layer units is 128 and 64 respectively, the activation function is ReLU, and a random sampling small batch training is performed using an experience replay pool with a capacity of 20,000. The discount factor γ is 0.95, and the exploration strategy is ε-greedy, where ε decreases linearly from 1 to 0.1. The result is output to the drive control module 5 through the path output module 43. A motion model is set in the drive control module 5, and differential drive is adopted. The motion model is:

[0084]

[0085] Where v is the linear velocity and w is the angular velocity;

[0086] Design the fuzzy controller, input variables: distance error e d =d desired -d actual , angle error e θ =θ desired -θ actual ; Output variables: linear velocity adjustment Δv, angular velocity adjustment Δw; Fuzzy rules: If e d Big and e θ If e is large, Δv is small and Δw is large; if e is large, Δv is small and Δw is large. d small and e θ If the error is small, Δv is large and Δw is small. The input variables are mapped to fuzzy sets, the applicability of each rule is calculated, the output of each rule is synthesized, and the center of gravity method is used to calculate the precise control output. The control process is to obtain the current position and posture of the shopping cart in real time, calculate the error, input the fuzzy controller, obtain Δv and Δw, and adjust the motor speed and direction.

[0087] In the above step 4, the shopping cart is controlled to follow the user. A YOLOv5s target detection model is deployed in the autonomous following module 6. 5,000 pictures of users with different postures and environments are used as training data. The training parameters are set as follows: learning rate η = 0.001, batch size 16, and number of iterations 100 rounds. In the target tracking process, the DeepSORT algorithm is used, ResNet50 is used to extract the user's appearance feature vector f, and the Kalman filter is used to predict the next position of the target. The safety distance in the following strategy is kept at about 1 meter, and the threshold d is set. safe = 1 meter, according to the target position (x target ,y target ) and the shopping cart position (x cart ,y cart ) Calculate the error and adjust the speed and direction;

[0088] In the above step 5, the user interacts with the shopping cart and the user is identified. A Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module 85. The training data of the model includes Mandarin and English speech data sets with a total length of 1000 hours. In the recognition process, the speech signal is first processed to extract the Mel-frequency cepstral coefficient (MFCC), and the BeamSearch algorithm is used to decode the output text; BERT is used for semantic understanding in the interaction processing module 84, and a classification model is used to map the input text to a predefined intent set {I1, I2, ..., I n}, extract key parameters such as product name and quantity during the slot filling process; extract voice features such as tone, energy, and rhythm through the emotion recognition module 87, use the emotion classification model to judge emotions, and use the CNN model for classification in combination with the captured facial expressions; the touch screen module 81 is provided with functional interfaces such as product search, recommendation, and shopping list, which are used in conjunction with the voice recognition module 85 and the gesture recognition module 86 to support voice, touch, and gesture operations; the ArcFace model is deployed in the facial recognition module 91, and the improved Softmax loss function is used to enhance the classification boundary. The training numerical control uses 1 million face pictures covering different ages and genders; the MTCNN is used to locate the face position in the identity comparison module 92, and the face image is input into the ArcFace model to obtain the feature vector f, and the cosine similarity with the feature vector stored in the database is calculated, and the threshold is used to determine whether it is a registered user:

[0089]

[0090] The history analysis module 93 and the personalized recommendation module 94 are used to perform personalized recommendations. The algorithm adopts a hybrid recommendation system, combining collaborative filtering and content recommendation. The user-based collaborative filtering is used to calculate the similarity between users and recommend products that similar users like. The similarity calculation is:

[0091]

[0092] Among them, r u,i is the rating of product i by user u, is the average rating of user u;

[0093] Matching is performed based on product attributes and user preferences, and sensitive data is encrypted and stored using AES, with explicit consent from the user obtained through user authorization before use.

[0094] Working principle: When in use, the start and stop of data acquisition is controlled by the data acquisition module 11, and RGB images are collected by the camera module 12 with a resolution of 1920×1080 and a frame rate of 30 frames / second to obtain visual information. 360-degree point cloud data is collected by the lidar module 13 with a scanning frequency of 10Hz to obtain three-dimensional spatial information of the environment. Close obstacles are detected by the ultrasonic module 14 and the infrared module 15 with a sampling frequency of 50Hz to supplement the blind spots of vision and lidar. Multiple sensors are used to accurately perceive complex dynamic environments, which is not prone to untimely obstacle avoidance or unreasonable path planning. MTCNN is used to locate the face position in the identity comparison module 92, and the face image is input into the ArcFace model to obtain The feature vector f is calculated, and the cosine similarity with the feature vector stored in the database is calculated. The threshold determines whether it is a registered user. The history analysis module 93 and the personalized recommendation module 94 are used for personalized recommendation. The algorithm adopts a hybrid recommendation system, combining collaborative filtering and content recommendation, in which user-based collaborative filtering is used to calculate the similarity between users, recommend products that similar users like, match based on the attributes of the products and the preferences of the users, actively adapt to user needs, and provide personalized services; the data information output by the prediction output module 33 and the data processing module 2 is received by the path receiving module 41, and transmitted to the path calculation module 42. In the path calculation module 42, the real-time environment map provided by the data processing module 2 is used, including the obstacle position {O i} and the motion state of dynamic objects to model the environment; the improved dynamic A-Star algorithm is used, and the heuristic function h(n) considers the predicted position of dynamic obstacles to perform initial path planning; the state space S contains the position (x, y) of the shopping cart, the speed v, the direction θ, the surrounding environment information, etc., and the action space A contains the actions that the shopping cart can perform, such as acceleration, deceleration, left turn, right turn, stop, etc., and the reward function R(s, a) is used. The policy network uses a deep Q network (DQN), the network structure is a three-layer fully connected network, the number of hidden layer units is 128 and 64 respectively, the activation function is ReLU, and a capacity of 20000 experience replay pool is used for random sampling small batch training, the discount factor γ = 0.95, and the exploration strategy is ε-greedy, ε decreases linearly from 1 to 0.1, and the result is output to the driving control module 5 through the path output module 43. The path planning algorithm is highly efficient in a dynamic and changeable shopping environment and can be adjusted in real time to ensure shopping efficiency. A Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module 85. The training data of the model includes Mandarin and English speech data sets with a total duration of 1000 hours. In the recognition process, the speech signal is first processed to extract the Mel frequency cepstral coefficient (MFCC), and the BeamSearch algorithm is used to decode the output text. BERT is used for semantic understanding in the interaction processing module 84, and a classification model is used to map the input text to a predefined intent set {I1, I2, ..., I n}, extract key parameters such as product name and quantity during the slot filling process; extract voice characteristics such as tone, energy, rhythm, etc. through the emotion recognition module 87, use the emotion classification model to judge emotions, and use the captured facial expressions to classify them using the CNN model; the touch screen module 81 is provided with functional interfaces such as product search, recommendation, shopping list, etc., which are used in conjunction with the voice recognition module 85 and the gesture recognition module 86, support voice, touch and gesture operations, and include multiple interaction methods. It can understand the user's emotions and intentions to provide humanized services.

[0095] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An intelligent shopping cart autonomous navigation system, comprising a sensor assembly (1), characterized in that: The sensor component (1) is composed of a data acquisition module (11), a camera module (12), a laser radar module (13), an ultrasonic module (14), an infrared module (15) and an acquisition output module (16); the data acquisition module (11) is controllably connected to the camera module (12); the camera module (12) is controllably connected to the acquisition output module (16); the acquisition output module (16) is controllably connected to the laser radar module (13); the laser radar module (13) is controllably connected to the data acquisition module (11); the data acquisition module (11) is controllably connected to the ultrasonic module (14); the ultrasonic module (14) is controllably connected to the acquisition output module (16); the acquisition output module (16) is controllably connected to the infrared module (15); the infrared module (15) is controllably connected to the data acquisition module (11); the acquisition output module (16) is controllably connected to the data processing module (2); the data processing module (2) is controllably connected to the prediction receiving module (31) in the behavior prediction component (3).

2. The intelligent shopping cart autonomous navigation system according to claim 1, characterized in that: The behavior prediction component (3) is composed of a prediction receiving module (31), a prediction processing module (32) and a prediction output module (33); the prediction receiving module (31) is controlled to be connected to the prediction processing module (32); the prediction processing module (32) is controlled to be connected to the prediction output module (33); the prediction output module (33) is controlled to be connected to the path receiving module (41) in the path planning component (4); and the path receiving module (41) is controlled to be connected to the data processing module (2).

3. The intelligent shopping cart autonomous navigation system according to claim 1, characterized in that: The path planning component (4) is composed of a path receiving module (41), a path calculation module (42) and a path output module (43); the path receiving module (41) is controlled to be connected to the path calculation module (42); the path calculation module (42) is controlled to be connected to the path output module (43); the path output module (43) is controlled to be connected to the drive control module (5); the drive control module (5) is controlled to be connected to the autonomous following module (6); the autonomous following module (6) is controlled to be connected to the prediction output module (33); the autonomous following module (6) is controlled to be connected to the user guidance module (7); the user guidance module (7) is controlled to be connected to the interactive output module (88) in the human-machine interaction component (8); the human-machine interaction component (8) is composed of a touch screen module (81), a microphone module (82), a speaker module (83), an interface module (84), and a control module (85). The interactive processing module (84), the speech recognition module (85), the gesture recognition module (86), the emotion recognition module (87) and the interactive output module (88) are composed of an interactive processing module (84), a speech recognition module (85), a gesture recognition module (86), a emotion recognition module (87) and an interactive output module (88); the touch screen module (81) is controlled to be connected to the interactive processing module (84), the interactive processing module (84) is controlled to be connected to the microphone module (82), the interactive processing module (84) is controlled to be connected to the speaker module (83), the interactive output module (88) is controlled to be connected to the interactive processing module (84), the interactive processing module (84) is controlled to be connected to the speech recognition module (85), the interactive processing module (84) is controlled to be connected to the gesture recognition module (86), the interactive processing module (84) is controlled to be connected to the emotion recognition module (87), and the interactive processing module (84) is controlled to be connected to the recognition output module (95) in the user recognition component (9).

4. The intelligent shopping cart autonomous navigation system according to claim 3, characterized in that: The user identification component (9) is composed of a facial recognition module (91), an identity comparison module (92), a history analysis module (93), a personality recommendation module (94) and an identification output module (95); the facial recognition module (91) is controlled to be connected to the identity comparison module (92); the identity comparison module (92) is controlled to be connected to the history analysis module (93); the history analysis module (93) is controlled to be connected to the personality recommendation module (94); and the personality recommendation module (94) is controlled to be connected to the identification output module (95).

5. A method for implementing autonomous navigation of a smart shopping cart includes the following steps: Step 1: multimodal environment perception and data fusion; Step 2: user behavior prediction; Step 3: path planning and drive control; Step 4: autonomous following; Step 5: human-computer interaction and user identification; It is characterized by: In the above step 1, multiple sensors are used to collect environmental data and perform data fusion; In the above step 2, the user's next behavior is predicted; In the above step 3, a route is planned to the destination and the shopping cart is driven and controlled; In the above step 4, the shopping cart is controlled to follow the user; In the above step 5, the user interacts with the shopping cart and the user is identified.

6. The method for realizing autonomous navigation of a smart shopping cart according to claim 5, characterized in that: In the step 1, the start and stop of data acquisition is controlled by the data acquisition module (11), RGB images are acquired by the camera module (12) with a resolution of 1920×1080 and a frame rate of 30 frames per second for acquiring visual information, 360-degree point cloud data are acquired by the laser radar module (13) with a scanning frequency of 10 Hz for acquiring three-dimensional spatial information of the environment, and close-range obstacles are detected by the ultrasonic module (14) and the infrared module (15) with a sampling frequency of 50 Hz for supplementing the blind spots of vision and laser radar; then, all the acquired data are transmitted to the data processing module (2) by the acquisition output module (16), and in the data processing module (2), the data of different sensors are synchronized by using the timestamp, and the data of each sensor is converted into a unified coordinate system by using the external parameter calibration; the feature points {f i }, with scale invariance and rotation invariance; extract local plane features and curvature information from point cloud data to form a feature set {p j }; According to the confidence w of the sensor data i , weight the data of each sensor, and the weight calculation formula is: in, is the variance of the i-th sensor data; The extended Kalman filter (EKF) is used for data fusion. The state vector x contains the position and velocity information, and the observation vector z contains the measurement values ​​of each sensor. The prediction steps are: x kk-1 =F k-1 x k-1k-1 +B k-1 u k-1 The update steps are: x kk =x kk-1 +K k (z k -H k x kk-1 ) P kk =(I-K k H k )P kk-1 Among them, F k is the state transfer matrix, H k is the observation matrix, Q k is the process noise covariance matrix, R k is the observation noise covariance matrix; The fused data is used to construct an environmental map, which is represented by a raster map or a topological map. The environmental model is used for subsequent path planning and obstacle avoidance.

7. The method for realizing autonomous navigation of a smart shopping cart according to claim 5, characterized in that: In the step 2, the data processed by the data processing module (2) is received by the prediction receiving module (31) and transmitted to the prediction processing module (32). In the prediction processing module (32), the network structure of the model adopts a double-layer LSTM network, and the input sequence is the user's historical behavior data {x1, x2, ..., x t }, each x i It includes features such as position, speed, and residence time. An attention mechanism is added to the output layer of LSTM to calculate the weight α of each time step. i , to capture key behavioral characteristics: e i =v T tanh(W h h i +W s S) Among them, h i is the hidden state of the i-th time step, s is the context vector, v, W h , W s is a learnable parameter; Using the cross entropy loss function L: Among them, y k is the true label, The probability predicted by the model; During the training process of the model, optimization and regularization are performed. The Adam optimizer is used, the learning rate η = 0.001, the L2 regularization term is added, the coefficient is λ = 0.0001, the number of iterations is 50 rounds, and the batch size is 64. When making predictions, the x value of the user behavior at the current moment is collected. t , normalize the data, subtract the mean and divide by the standard deviation, input the preprocessed data into the model, and calculate the output That is, the probability distribution of the user's next behavior, and the prediction result is output through the prediction output module (33).

8. The method for realizing autonomous navigation of a smart shopping cart according to claim 5, characterized in that: In the step 3, the data information output by the prediction output module (33) and the data processing module (2) is received by the path receiving module (41), and transmitted to the path calculation module (42). In the path calculation module (42), the real-time environment map provided by the data processing module (2), including the obstacle position {O i } and the motion state of dynamic objects to model the environment; the improved dynamic A-Star algorithm is used, and the heuristic function h(n) considers the predicted position of dynamic obstacles to perform initial path planning; the state space S contains the position (x, y), speed v, direction θ, surrounding environment information, etc. of the shopping cart, and the action space A contains the actions that the shopping cart can perform, such as acceleration, deceleration, left turn, right turn, stop, etc., and the reward function R(s,a) is used: R(s,a)=w1.r goal +w2.r collision +w3.r efficiency Among them, r goal : Reward for reaching the goal (+100 points), r collision : Collision penalty (-100 points), r efficiency : Path efficiency reward, the shorter the path, the higher the reward, w1, w2, w3 are weight parameters; The policy network adopts the deep Q network (DQN), the network structure is a three-layer fully connected network, the number of hidden layer units is 128 and 64 respectively, the activation function is ReLU, and a random sampling small batch training is performed using an experience replay pool with a capacity of 20,000. The discount factor γ is 0.95, and the exploration strategy is ε-greedy, where ε decreases linearly from 1 to 0.

1. The result is output to the drive control module (5) through the path output module (43). A motion model is set in the drive control module (5), and differential drive is adopted. The motion model is: Where v is the linear velocity and w is the angular velocity; Design the fuzzy controller, input variables: distance error e d =d desired -d actual , angle error e θ =θ desired -θ actual ; Output variables: linear velocity adjustment Δv, angular velocity adjustment Δw; Fuzzy rules: If e d Big and e θ If e is large, Δv is small and Δw is large; if e is large, Δv is small and Δw is large. d small and e θ If Δv is small, Δw is small; map the input variables to the fuzzy set, calculate the applicability of each rule, synthesize the output of each rule, and use the center of gravity method to calculate the precise control output; the control process is to obtain the current position and posture of the shopping cart in real time, calculate the error, input the fuzzy controller, obtain Δv and Δw, and adjust the motor speed and direction.

9. The method for realizing autonomous navigation of a smart shopping cart according to claim 5, characterized in that: In the step 4, a YOLOv5s target detection model is deployed in the autonomous following module (6), and 5000 pictures of users labeled with different postures and environments are used as training data. The training parameters are set as follows: learning rate η=0.001, batch size is 16, and the number of iterations is 100 rounds; during the target tracking process, the DeepSORT algorithm is adopted, ResNet50 is used to extract the user's appearance feature vector f, and the Kalman filter is used to predict the next position of the target; the safety distance in the following strategy is kept at about 1 meter, and the threshold d is set safe = 1 meter, according to the target position (x target ,y target ) and the shopping cart position (x cart ,y cart ) calculates the error and adjusts the speed and direction.

10. The method for realizing autonomous navigation of a smart shopping cart according to claim 5, characterized in that: In the step 5, a Transformer-based automatic speech recognition (ASR) model is deployed in the speech recognition module (85), and the training data of the model includes Mandarin and English speech data sets with a total length of 1000 hours. In the recognition process, the speech signal is first processed, the Mel-frequency cepstral coefficients (MFCC) are extracted, and the BeamSearch algorithm is used to decode the output text; in the interaction processing module (84), BERT is used for semantic understanding, and a classification model is used to map the input text to a predefined intent set {I1, I2, ..., I n }, extract key parameters such as product name and quantity during the slot filling process; The tone, energy, rhythm and other characteristics of the voice are extracted through the emotion recognition module (87), and the emotion is judged using the emotion classification model. The captured facial expressions are combined and classified using the CNN model; the touch screen module (81) is provided with functional interfaces such as product search, recommendation, shopping list, etc., which are used in conjunction with the voice recognition module (85) and the gesture recognition module (86) to support voice, touch and gesture operations; the face recognition module (91) is provided with an ArcFace model, and an improved Softmax loss function is used to enhance the classification boundary. The training numerical control uses 1 million face pictures covering different ages and genders; the identity comparison module (92) uses MTCNN to locate the face position, and the face image is input into the ArcFace model to obtain the feature vector f, and the cosine similarity with the feature vector stored in the database is calculated, and the threshold is used to determine whether it is a registered user: The historical analysis module (93) and the personalized recommendation module (94) are used for personalized recommendation. The algorithm adopts a hybrid recommendation system, combining collaborative filtering and content recommendation. The user-based collaborative filtering is used to calculate the similarity between users and recommend products that similar users like. The similarity calculation is: Among them, r u,i is the rating of product i by user u, is the average rating of user u; Matching is performed based on product attributes and user preferences, and sensitive data is encrypted and stored using AES, with explicit consent from the user obtained through user authorization before use.

Images