Cloud duty convenience store intelligent central control system based on multi-modal deep learning

Abstract

The invention discloses a cloud duty convenience store intelligent central control system based on multi-modal deep learning, and the system comprises a central control host module, a data collection module, a voice interaction module, an AI interaction module, a behavior recognition module, and a manual customer service intervention module. The central control host module is in data connection with the access control module, the data acquisition module, the cashier management module, the voice interaction module, the AI interaction module, the behavior recognition module and the manual customer service intervention module; the behavior recognition module adopts multi-modal data fusion and an improved deep learning algorithm to realize high-precision recognition of abnormal behaviors in a complex environment, and a multi-head self-attention mechanism is introduced to effectively reduce false alarms; the AI interaction module adopts voice recognition based on deep learning, natural language processing and voice synthesis, smooth voice interaction support is provided, and diversified requirements of customers are met; the millisecond-level system response is realized by adopting the edge computing architecture, and the customer experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a cloud-attended convenience store intelligent central control system based on multimodal deep learning. Background Art

[0002] Unmanned convenience stores are a new type of retail model that uses advanced technology to achieve self-service shopping and operation management without traditional store staff on duty. Unmanned convenience stores usually use video surveillance modules for real-time monitoring, and use advanced behavior recognition modules and algorithms to analyze customer behavior to ensure the safety of the store and normal shopping order. The access control module controls the entry and exit of personnel, and customers can enter the store by scanning codes, face recognition, etc. The central control system of existing unmanned convenience stores has technical bottlenecks in behavior recognition accuracy, interaction methods, and system response speed. Specifically, the following are the following: First, the accuracy of behavior recognition is low, and it cannot effectively respond to abnormal behavior detection in complex environments, such as lighting changes, occlusion, etc., resulting in a high false alarm rate, affecting the normal operation of the store; second, the interaction method is relatively single and cannot meet the diverse needs of customers; third, the system response speed is slow, affecting customer experience. Summary of the invention

[0003] The purpose of the present invention is to provide a cloud-attended convenience store intelligent central control system based on multimodal deep learning to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned objectives, the present invention provides the following technical solutions: a cloud-attended convenience store intelligent central control system based on multimodal deep learning, comprising a central control host module, an access control module, a data acquisition module, a cashier management module, a voice interaction module, an AI interaction module, a behavior recognition module and a manual customer service intervention module, wherein the central control host module respectively establishes data connections with the access control module, the data acquisition module, the cashier management module, the voice interaction module, the AI ​​interaction module, the behavior recognition module and the manual customer service intervention module, the voice interaction module establishes a data connection with the AI ​​interaction module, and the data acquisition module establishes a data connection with the behavior recognition module.

[0005] Preferably, the access control module is data-connected to an electronic door lock and an identity recognition device, the identity recognition device includes a face recognition camera and an RFID card reader, and the data acquisition module includes a video acquisition camera, a depth sensor and an accelerometer.

[0006] Preferably, the cashier management module includes a self-service cashier device, a barcode scanner and a weight sensor.

[0007] Preferably, the voice interaction module includes a microphone array and a speaker.

[0008] Preferably, the AI ​​interaction module includes a speech recognition unit, a natural language understanding unit and a speech synthesis unit.

[0009] Preferably, the speech recognition unit adopts end-to-end speech recognition based on Transformer, and the formula is as follows:

[0010]

[0011] Among them, X is the speech feature sequence, and Y is the recognized text sequence;

[0012] The natural language understanding unit uses the pre-trained BERT model for intent recognition and slot filling. The formula is as follows:

[0013] Intent, Slots = BERT(Y)

[0014] The speech synthesis unit uses a neural network based on Tacotron2 to generate a speech spectrogram, and then generates a speech waveform through WaveGlow. The formula is as follows:

[0015] Mel-Spectrogram=Tacotron2(Text)

[0016] Audio=WaveGlow(Mel-Spectrogram).

[0017] Preferably, the speech recognition unit includes a feature extraction subunit and a Transformer model, the natural language understanding unit includes an intent recognition subunit and a slot filling subunit, and the speech synthesis unit includes a Tacotron2 model and a WaveGlow model.

[0018] Preferably, the feature extraction subunit is used to extract Mel frequency cepstral coefficients MFCC, and the formula is as follows:

[0019] MFCC = DCT(log(Mel(STFT(x))))

[0020] The Transformer model includes an encoder and a decoder, and the formula is as follows:

[0021] Encoder output = Encoder(MFCC)

[0022] Decoder output = Decoder (encoder output, history output)

[0023] The intent recognition subunit uses BERT for text classification. The formula is as follows:

[0024] Intent=Softmax(W intent BERT(Text)+bintent )

[0025] The slot filling subunit is used to identify the key information of the text. The formula is as follows:

[0026] Slots = CRF(BERT(Text))

[0027] The Tacotron2 model is used to convert text into speech spectrograms. The formula is as follows:

[0028] Mel-Spectrogram=DecoderRNN(EncoderRNN(Text))

[0029] The WaveGlow model is used to convert the spectrogram into audio. The formula is as follows:

[0030] Audio=WaveGlow(Mel-Spectrogram).

[0031] Preferably, the behavior recognition module includes a data preprocessing unit, a spatiotemporal feature extraction unit, a graph convolutional network model, a multi-head self-attention mechanism unit and a feature fusion and classification unit.

[0032] Preferably, the data preprocessing unit is used to collect visual, depth and sensor data and perform synchronous processing, and the formula is as follows:

[0033] X={x 视觉 ,x 深度 ,x 传感器}

[0034] The spatiotemporal feature extraction unit introduces residual connection and batch normalization on the 3D-CNN model. The formula is as follows:

[0035] F 时空 =BatchNorm(ReLU(Conv3D(X)+X))

[0036] The graph convolutional network model constructs a spatiotemporal graph based on human key points and uses a graph convolutional network to extract topological structure features. The spatiotemporal graph is constructed by using nodes to represent human body relationships and edges to represent bone connections and time series relationships. The convolution calculation formula is as follows:

[0037]

[0038] Among them, A k is the adjacency matrix, Λ k is the degree matrix, W k is the weight matrix, σ is the activation function;

[0039] The formula of the multi-head self-attention mechanism unit is as follows:

[0040]

[0041] Among them, Q, K, V are query, key, and value matrices, d k is the dimension of the key;

[0042] The formulas for feature fusion and classification units are as follows:

[0043] F 融合 =Concat(F 时空 ,F GCN ,F 注意力 )

[0044] y=Softmax(WF 融合 +b)

[0045] Among them, W and b are weight and bias parameters.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows: the behavior recognition module of the present invention adopts multimodal data fusion and improved deep learning algorithm to achieve high-precision recognition of abnormal behaviors in complex environments, and the introduction of multi-head self-attention mechanism effectively reduces false alarms; the AI ​​interaction module adopts deep learning-based speech recognition, natural language processing and speech synthesis, providing smooth voice interaction support to meet the diverse needs of customers; by adopting edge computing architecture, millisecond-level system response is achieved, which improves customer experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall structural diagram of the present invention;

[0048] Figure 2 This is a structural block diagram of the AI ​​interaction module of the present invention;

[0049] Figure 3 It is a structural block diagram of the behavior recognition module of the present invention;

[0050] Figure 4 It is a flow chart of the behavior recognition algorithm of the present invention;

[0051] Figure 5 This is a flow chart of the voice interaction algorithm of the present invention.

[0052] In the figure: 1. Central control host module; 2. Access control module; 21. Electronic door lock; 22. Identity recognition device; 221. Face recognition camera; 222. RFID card reader; 3. Data acquisition module; 31. Video acquisition camera; 32. Depth sensor; 33. Accelerometer; 4. Cashier management module; 41. Self-service cashier device; 42. Barcode scanner; 43. Weight sensor; 5. Voice interaction module; 51. Microphone array; 52. Speaker; 6. AI interaction module; 61. Voice recognition unit; 611. Feature extraction subunit; 612, Transformer model; 62, natural language understanding unit; 621, intent recognition subunit; 622, slot filling subunit; 63, speech synthesis unit; 631, Tacotron2 model; 632, WaveGlow model; 7, behavior recognition module; 71, data preprocessing unit; 72, spatiotemporal feature extraction unit; 73, graph convolutional network model; 74, multi-head self-attention mechanism unit; 75, feature fusion and classification unit; 8, manual customer service intervention module. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Please see attached Figure 1 -Attached Figure 5, an embodiment provided by the present invention: a cloud-attended convenience store intelligent central control system based on multimodal deep learning, comprising a central control host module 1, an access control module 2, a data acquisition module 3, a cashier management module 4, a voice interaction module 5, an AI interaction module 6, a behavior recognition module 7 and an artificial customer service intervention module 8, the central control host module 1 respectively establishes data connections with the access control module 2, the data acquisition module 3, the cashier management module 4, the voice interaction module 5, the AI ​​interaction module 6, the behavior recognition module 7 and the artificial customer service intervention module 8, the voice interaction module 5 establishes data connections with the AI ​​interaction module 6, the data acquisition module 3 establishes data connections with the behavior recognition module 7, the central control host module 1 is responsible for the coordination and data processing of various modules of the system, the access control module 2 realizes customer identity authentication and access control, the data acquisition module 3 is used to provide multi-source data, the cashier management module 4 is used to process commodity scanning and payment, the voice interaction module 5 is used to perform voice interaction with customers, the AI ​​interaction module 6 is used for voice recognition, natural language processing and speech synthesis, the behavior recognition module 7 is used to identify abnormal behavior of customers, and the artificial customer service intervention module 8 is used to detect abnormal behavior. When the customer needs help, the remote customer service staff will be automatically notified to intervene; the access control module 2 is connected to the electronic door lock 21 and the identity recognition device 22, the identity recognition device 22 includes a face recognition camera 221 and an RFID card reader 222, the data acquisition module 3 includes a video acquisition camera 31, a depth sensor 32 and an accelerometer 33, the electronic door lock 21 is used to control the opening and closing of the store entrance, the identity recognition device 22 is used to verify the customer's identity, the video acquisition camera 31 is used to provide high-quality video data, the depth sensor 32 is used to provide depth data, and the accelerometer 33 is used to provide acceleration data; the cashier management module 4 includes a self-service cashier device 41, a barcode scanner 42 and a weight sensor 43, which are used for commodity scanning, payment processing and preventing missed scanning; the voice interaction module 5 includes a microphone array 51 and a speaker 52, which are used to interact with customers by voice and provide services; the AI ​​interaction module 6 includes a voice recognition unit 61, a natural language understanding unit 62 and a voice synthesis unit 63, which are used to realize natural language interaction; the voice recognition unit 61 adopts end-to-end voice recognition based on Transformer, and the formula is as follows:

[0055]

[0056] Among them, X is the speech feature sequence, and Y is the recognized text sequence;

[0057] The natural language understanding unit 62 uses the pre-trained BERT model for intent recognition and slot filling. The formula is as follows:

[0058] Intent, Slots = BERT(Y)

[0059] The speech synthesis unit 63 generates a speech spectrogram using a neural network based on Tacotron2, and generates a speech waveform through WaveGlow. The formula is as follows:

[0060] Mel-Spectrogram=Tacotron2(Text)

[0061] Audio=WaveGlow(Mel-Spectrogram)

[0062] The speech recognition unit 61 includes a feature extraction subunit 611 and a Transformer model 612. The natural language understanding unit 62 includes an intent recognition subunit 621 and a slot filling subunit 622. The speech synthesis unit 63 includes a Tacotron2 model 631 and a WaveGlow model 632. The feature extraction subunit 611 is used to extract the Mel frequency cepstral coefficient MFCC, and the formula is as follows:

[0063] MFCC = DCT(log(Mel(STFT(x))))

[0064] The Transformer model 612 includes an encoder and a decoder, and the formula is as follows:

[0065] Encoder output = Encoder(MFCC)

[0066] Decoder output = Decoder (encoder output, history output)

[0067] The intent recognition subunit 621 uses BERT for text classification, and the formula is as follows:

[0068] Intent=Softmax(W intent BERT(Text)+b intent )

[0069] The slot filling subunit 622 is used to identify key information of the text, and the formula is as follows:

[0070] Slots = CRF(BERT(Text))

[0071] Tacotron2 model 631 is used to convert text into speech spectrogram. The formula is as follows:

[0072] Mel-Spectrogram=DecoderRNN(EncoderRNN(Text))

[0073] WaveGlow model 632 is used to convert the spectrogram into audio. The formula is as follows:

[0074] Audio=WaveGlow(Mel-Spectrogram)

[0075] The behavior recognition module 7 includes a data preprocessing unit 71, a spatiotemporal feature extraction unit 72, a graph convolutional network model 73, a multi-head self-attention mechanism unit 74 and a feature fusion and classification unit 75; the data preprocessing unit 71 is used to collect visual, depth and sensor data and perform synchronous processing, and the formula is as follows:

[0076] X={x 视觉 ,x 深度 ,x 传感器}

[0077] The spatiotemporal feature extraction unit 72 introduces residual connection and batch normalization on the 3D-CNN model, and the formula is as follows:

[0078] F 时空 =BatchNorm(ReLU(Conv3D(X)+X))

[0079] The graph convolutional network model 73 constructs a spatiotemporal graph based on human key points and uses a graph convolutional network to extract topological structure features. The spatiotemporal graph is constructed by using nodes to represent human body relationships and edges to represent bone connections and time series relationships. The convolution calculation formula is as follows:

[0080]

[0081] Among them, A k is the adjacency matrix, Λ k is the degree matrix, W k is the weight matrix, σ is the activation function;

[0082] The formula of the multi-head self-attention mechanism unit 74 is as follows:

[0083]

[0084] Among them, Q, K, V are query, key, and value matrices, d k is the dimension of the key;

[0085] The formula of feature fusion and classification unit 75 is as follows:

[0086] F 融合 =Concat(F 时空 ,F GCN ,F 注意力 )

[0087] y=Softmax(WF 融合 +b)

[0088] Among them, W and b are weight and bias parameters.

[0089] Working principle: When using the present invention, the central control host module 1 is responsible for the coordination and data processing of various modules of the system, the access control module 2 is used to realize customer identity authentication and access control, the data acquisition module 3 is used to provide high-quality video data, the cashier management module 4 is used to process commodity scanning and payment, the voice interaction module 5 is used to interact with customers by voice, the AI ​​interaction module 6 is used to perform voice recognition, natural language processing and voice synthesis, the behavior recognition module 7 is used to identify the abnormal behavior of customers, and the manual customer service intervention module 8 is used to automatically notify the remote customer service personnel to intervene when abnormal behavior is detected or the customer needs help. Specifically: the customer passes through the human in the identity recognition device 22 The face recognition camera 221 or the RFID card reader 222 performs identity recognition. After the recognition is completed, the access control module 2 controls the electronic door lock 21 to unlock, so that the store door is opened. After the customer enters the store, the store door is closed. The video acquisition cameras 31 distributed at various locations in the store collect video data. At the same time, the depth sensor 32 collects depth data, the accelerometer 33 collects acceleration data, the weight sensor 43 collects weight data, and the microphone array 51 collects voice data. The video, depth, acceleration and weight data are transmitted to the behavior recognition module 7. The data preprocessing unit 71 is used to synchronize and normalize the data. The spatiotemporal feature extraction unit 72 extracts spatiotemporal features and outputs them to The graph convolution network model 73 constructs a spatiotemporal graph based on the key points of the human body, extracts topological structure features, and uses the multi-head self-attention mechanism unit 74 to apply the multi-head self-attention mechanism to the extracted features to strengthen the key features. The feature fusion and classification unit 75 fuses the spatiotemporal features, topological structure features and attention features, uses the fully connected layer and the Softmax function to classify the behavior, outputs the behavior category, and determines whether the behavior is abnormal, and feeds back the result to the central control host module 1. If it is abnormal, the manual customer service intervention module 8 is triggered for manual intervention; the voice data collected by the microphone array 51 is transmitted to the AI ​​interaction module 6, and after noise reduction processing, the voice The feature extraction subunit 611 of the recognition unit 61 extracts speech features such as MFCC, inputs the features into the Transformer model 612, outputs the text sequence to the natural language understanding unit 62, the intent recognition subunit 621 uses BERT to classify the text, uses the slot filling subunit 622 to identify the key information of the text, extracts the intent and slot information, generates the response content according to the customer's intent and the system status, and sends it to the speech synthesis unit 63, the Tacotron2 model 631 converts the response text into a Mel-spectrogram, and the WaveGlow model 632 converts the spectrogram into an audio waveform and sends it to the speaker 52 for playback;After the customer has selected the goods, he can settle the goods on the self-service checkout device 41 and scan the code with the barcode scanner 42. When the customer places the goods in the checkout area, the weight sensor 43 can detect the weight of the goods in real time. If the weight sensor 43 detects that the total weight of the goods after scanning is inconsistent with the weight detected by the sensor, it will prompt the customer that there may be a case of missing goods. Among them, the weight sensor 43 is set on the shopping cart and the self-service checkout device 41. After test evaluation, the behavior recognition accuracy of the behavior recognition module 7 reached 99.5%, the speech recognition word error rate of the AI ​​interaction module 6 was reduced to 2.5%, and the intention recognition accuracy reached 99.8%. When the full score is 5 points, the speech synthesis naturalness score reached an average of 4.9 points. ;

[0090] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cloud-attended convenience store intelligent central control system based on multimodal deep learning, comprising a central control host module (1), an access control module (2), a data acquisition module (3), a cashier management module (4), a voice interaction module (5), an AI interaction module (6), a behavior recognition module (7) and a manual customer service intervention module (8), characterized in that: The central control host module (1) respectively establishes data connections with the access control module (2), the data acquisition module (3), the cashier management module (4), the voice interaction module (5), the AI ​​interaction module (6), the behavior recognition module (7) and the manual customer service intervention module (8); the voice interaction module (5) establishes a data connection with the AI ​​interaction module (6); and the data acquisition module (3) establishes a data connection with the behavior recognition module (7).

2. According to claim 1, a cloud-attended convenience store intelligent central control system based on multimodal deep learning is characterized in that: The access control module (2) is data-connected to an electronic door lock (21) and an identity recognition device (22); the identity recognition device (22) comprises a face recognition camera (221) and an RFID card reader (222); and the data acquisition module (3) comprises a video acquisition camera (31), a depth sensor (32) and an accelerometer (33).

3. According to claim 1, a cloud-attended convenience store intelligent central control system based on multimodal deep learning is characterized in that: The cashier management module (4) comprises a self-service cashier device (41), a barcode scanner (42) and a weight sensor (43).

4. According to claim 1, a cloud-attended convenience store intelligent central control system based on multimodal deep learning is characterized in that: The voice interaction module (5) comprises a microphone array (51) and a loudspeaker (52).

5. According to claim 1, a cloud-attended convenience store intelligent central control system based on multimodal deep learning is characterized in that: The AI ​​interaction module (6) comprises a speech recognition unit (61), a natural language understanding unit (62) and a speech synthesis unit (63).

6. According to claim 5, a cloud-attended convenience store intelligent central control system based on multimodal deep learning is characterized in that: The speech recognition unit (61) adopts end-to-end speech recognition based on Transformer, and the formula is as follows: Among them, X is the speech feature sequence, and Y is the recognized text sequence; The natural language understanding unit (62) uses the pre-trained BERT model for intent recognition and slot filling. The formula is as follows: Intent, Slots = BERT(Y) The speech synthesis unit (63) generates a speech spectrogram using a neural network based on Tacotron2, and generates a speech waveform through WaveGlow. The formula is as follows: Mel-Spectrogram=Tacotron2(Text) Audio=WaveGlow(Mel-Spectrogram).

7. The cloud-attended convenience store intelligent central control system based on multimodal deep learning according to claim 6 is characterized by: The speech recognition unit (61) includes a feature extraction subunit (611) and a Transformer model (612), the natural language understanding unit (62) includes an intent recognition subunit (621) and a slot filling subunit (622), and the speech synthesis unit (63) includes a Tacotron2 model (631) and a WaveGlow model (632).

8. The cloud-attended convenience store intelligent central control system based on multimodal deep learning according to claim 7 is characterized by: The feature extraction subunit (611) is used to extract Mel frequency cepstral coefficients MFCC, and the formula is as follows: MFCC = DCT(log(Mel(STFT(x)))) The Transformer model (612) includes an encoder and a decoder, and the formula is as follows: Encoder output = Encoder(MFCC) Decoder output = Decoder (encoder output, history output) The intent recognition subunit (621) uses BERT for text classification, and the formula is as follows: Intent=Softmax(W intent ·BERT(Text)+b intent ) The slot filling subunit (622) is used to identify key information of the text, and the formula is as follows: Slots = CRF(BERT(Text)) The Tacotron2 model (631) is used to convert text into speech spectrograms. The formula is as follows: Mel-Spectrogram=DecoderRNN(EncoderRNN(Text)) The WaveGlow model (632) is used to convert the spectrogram into audio. The formula is as follows: Audio=WaveGlow(Mel-Spectrogram).

9. The cloud-attended convenience store intelligent central control system based on multimodal deep learning according to claim 1 is characterized by: The behavior recognition module (7) includes a data preprocessing unit (71), a spatiotemporal feature extraction unit (72), a graph convolutional network model (73), a multi-head self-attention mechanism unit (74) and a feature fusion and classification unit (75).

10. The cloud-attended convenience store intelligent central control system based on multimodal deep learning according to claim 9, characterized in that: The data preprocessing unit (71) is used to collect visual, depth and sensor data and perform synchronous processing. The formula is as follows: X={x 视觉 ,x 深度 ,x 传感器 } The spatiotemporal feature extraction unit (72) introduces residual connection and batch normalization on the 3D-CNN model, and the formula is as follows: F 时空 =BatchNorm(ReLU(Conv3D(X)+X)) The graph convolutional network model (73) constructs a spatiotemporal graph based on human key points and uses a graph convolutional network to extract topological structure features. The spatiotemporal graph is constructed by using nodes to represent human body relationships and edges to represent bone connections and time series relationships. The convolution calculation formula is as follows: Among them, A k is the adjacency matrix, Λ k is the degree matrix, W k is the weight matrix, σ is the activation function; The formula of the multi-head self-attention mechanism unit (74) is as follows: Among them, Q, K, V are query, key, and value matrices, d k is the dimension of the key; The formula for feature fusion and classification unit (75) is as follows: F 融合 =Concat(F 时空 ,F GCN ,F 注意力 ) y=Softmax(WF 融合 +b) Among them, W and b are weight and bias parameters.

Images