Multi-modal fusion intelligent unmanned retail system based on computer vision convolutional neural network

Abstract

The invention discloses a multi-mode fusion intelligent unmanned retail system based on a computer vision convolutional neural network. The system comprises a data acquisition module, a shopping process management module, an intelligent service module, a commodity quality monitoring module, a cloud platform, an electronic price tag module and a data security and privacy protection module. According to the invention, a multi-modal data fusion technology is adopted, and the analysis results based on multiple data sources are fused, so that the commodities are accurately identified and tracked; a personalized commodity recommendation service is provided for the customer by constructing a customer portrait; an AI virtual digital human guiding function and an auxiliary shopping function are integrated through the intelligent service module, a shopping route is formulated for customers, the customers are quickly guided to required commodities, and the shopping experience of the customers is improved; through comprehensive analysis of shopping feedback data, commodity search data, sales data and inventory data, the inventory demand can be accurately predicted; by monitoring the storage time and the storage environment of the commodities, the quality of the stored commodities is ensured.

Description

Technical Field

[0001] The present invention relates to the field of unmanned retail technology, and specifically to a multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network. Background Art

[0002] With the development of artificial intelligence and Internet of Things technology, unmanned retail has gradually become an important direction for the future retail industry. However, the existing unmanned retail system still has the following defects in actual use: First, it adopts a single product identification method, such as electronic tags or image recognition. Due to the single data source, the system's comprehensive perception ability is poor and it cannot accurately identify and track products; second, the existing unmanned retail system lacks in-depth analysis and understanding of customer behavior and cannot accurately recommend products to customers; third, it cannot quickly guide customers to the required products, causing customers to spend a lot of time looking for products, affecting their shopping experience; fourth, the ability to predict inventory is weak, and it is easy to be out of stock or backlog; fifth, there is a lack of effective monitoring of the quality and shelf life of inventory products. Summary of the invention

[0003] The purpose of the present invention is to provide a multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network, comprising a data acquisition module, a shopping process management module, an intelligent service module, a commodity quality monitoring module, a cloud platform, an electronic price tag module and a data security and privacy protection module, wherein the data acquisition module respectively establishes data connections with the shopping process management module, the intelligent service module and the commodity quality monitoring module, the cloud platform respectively establishes data connections with the shopping process management module, the intelligent service module, the commodity quality monitoring module, the electronic price tag module and the data security and privacy protection module, and the shopping process management module establishes data connection with the intelligent service module.

[0005] Preferably, the data acquisition module includes a visual sensor, a weight sensor, an acoustic wave sensor, a temperature and humidity sensor, and a sound sensor. The visual sensor includes a high-definition camera and a 3D camera. The high-definition camera and the 3D camera are used to obtain image data, the weight sensor is used to obtain weight data, the acoustic wave sensor obtains position data, the temperature and humidity sensor is used to obtain ambient temperature and humidity data, and the sound sensor is used to obtain sound data.

[0006] Preferably, the shopping process management module includes a customer identification submodule, a bill creation submodule, a shopping behavior judgment submodule and a bill update submodule. The shopping behavior judgment submodule includes a commodity taking behavior judgment unit, a commodity returning behavior judgment unit and an abnormal behavior judgment unit. The customer identification submodule is used for face recognition, the bill creation submodule is used to generate an initial bill, the shopping behavior judgment submodule is used to analyze shopping behavior, and the bill update submodule is used to update initial bill data.

[0007] Preferably, the intelligent service module includes an AI virtual digital human sub-module, an assisted shopping sub-module, a shopping feedback sub-module and a self-service checkout sub-module. The AI ​​virtual digital human sub-module includes a visual unit and a speech unit. The visual unit includes an appearance model and a model animation. The speech unit includes speech synthesis and natural language processing. The AI ​​virtual digital human sub-module is used for interaction. The assisted shopping sub-module is used for assisted shopping. The shopping feedback sub-module is used for feedback on shopping experience. The self-service checkout sub-module is used for settling bills. The visual unit is used to visualize the digital human. The appearance model is a digital human model. The model animation is a digital human interactive animation. Natural language processing is used to parse natural language. Speech synthesis is used to convert text into speech.

[0008] Preferably, the auxiliary shopping submodule includes a product location guidance unit, a product recommendation unit, a product search unit and a product display unit. The product location guidance unit includes a shopping route generation function, a lighting guidance function and an electronic map guidance function. The product location guidance unit is used to guide the shopping route, the product recommendation unit is used to recommend products, the product search unit is used to provide product search services, the product display unit is used to display basic product information, the shopping route generation function is used to plan the shopping route, the lighting guidance function is used to provide route guidance services, and the electronic map guidance function is used to provide route guidance services.

[0009] Preferably, the shopping feedback submodule includes a product quality feedback unit and a desired product entry unit, the self-service settlement submodule includes a code scanning payment unit, a biometric payment unit and a contactless payment unit, the product quality feedback unit is used to enter the product quality evaluation, and the desired product entry unit is used to enter the name of the product expected to be purchased.

[0010] Preferably, the commodity quality monitoring module includes a commodity expiration warning submodule and a storage environment abnormality alarm submodule, the commodity expiration warning submodule is used to issue commodity expiration reminders, and the storage environment abnormality alarm submodule is used to issue storage environment abnormality reminders.

[0011] Preferably, the cloud platform includes a price tag management module and a data analysis module, the data analysis module includes a shopping feedback data analysis submodule, a product search data analysis submodule, a personalized recommendation submodule and an inventory management submodule, the price tag management module is used to remotely manage electronic price tags, the data analysis module is used to analyze data using an AI algorithm, the shopping feedback data analysis submodule is used to count shopping feedback data within a set time period, the product search data analysis submodule is used to count product search data within a set time period, the personalized recommendation submodule is used to generate personalized product recommendation plans, and the inventory management submodule is used for inventory management plans.

[0012] Preferably, the personalized recommendation submodule includes a bill data analysis unit, a customer behavior analysis unit and a recommendation scheme generation unit. The customer behavior analysis unit includes a browsing behavior analysis function, a dwelling behavior analysis function and a picking behavior analysis function. The inventory management submodule includes a sales data analysis unit, an inventory data analysis unit and a decision support unit. The bill data analysis unit is used to statistically analyze the customer's bill data. The customer behavior analysis unit is used to analyze the browsing, dwelling and picking time in shopping behavior. The recommendation scheme generation unit is used to generate a commodity recommendation scheme based on the comprehensive analysis results. The sales data analysis unit is used to analyze commodity sales data. The inventory data analysis unit is used to analyze commodity inventory data. The decision support unit is used to generate an inventory management scheme based on the comprehensive analysis results.

[0013] Preferably, the electronic price tag module includes a wireless communication submodule, a display submodule, a data processing submodule and a power submodule, and the data processing submodule establishes data connections with the wireless communication submodule, the display submodule and the power submodule respectively. The wireless communication submodule is used to realize wireless communication, the display submodule is used to display data, the data processing submodule is the control center, and the power submodule is used to supply power.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention adopts multimodal data fusion technology to accurately identify and track commodities by fusing analysis results based on multiple data sources; provides personalized commodity recommendation services to customers by constructing customer portraits; integrates AI virtual digital human guidance function and assisted shopping function through the intelligent service module to formulate shopping routes for customers, and quickly guide customers to the required commodities, thereby improving customers' shopping experience; can accurately predict inventory demand by comprehensively analyzing shopping feedback data, commodity search data, sales data and inventory data; and ensures the quality of inventory commodities by monitoring the storage time and storage environment of commodities. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall structural diagram of the present invention;

[0016] Figure 2It is a structural block diagram of the data acquisition module of the present invention;

[0017] Figure 3 This is a structural block diagram of the shopping process management module of the present invention;

[0018] Figure 4 It is a structural block diagram of the intelligent service module of the present invention;

[0019] Figure 5 This is a structural block diagram of the AI ​​virtual digital human submodule of the present invention;

[0020] Figure 6 This is a structural block diagram of the auxiliary shopping submodule of the present invention;

[0021] Figure 7 This is a structural block diagram of the shopping feedback submodule of the present invention;

[0022] Figure 8 This is a structural block diagram of the self-service settlement submodule of the present invention;

[0023] Fig. 9 It is a structural block diagram of the commodity quality monitoring module of the present invention;

[0024] Fig.10 This is a structural block diagram of the cloud platform of the present invention;

[0025] Fig.11 It is a structural block diagram of the customer behavior analysis unit of the present invention;

[0026] Fig.12 This is a structural block diagram of the electronic price tag module of the present invention.

[0027] In the figure: 1. Data acquisition module; 11. Visual sensor; 111. High-definition camera; 112. 3D camera; 12. Weight sensor; 13. Acoustic sensor; 14. Temperature and humidity sensor; 15. Sound sensor; 2. Shopping process management module; 21. Customer identification submodule; 22. Bill creation submodule; 23. Shopping behavior judgment submodule; 231. Product taking behavior judgment unit; 232. Product putting back behavior judgment unit; 233. Abnormal behavior judgment unit; 24. Bill update submodule; 3. Intelligent service module; 3 1. AI virtual digital human submodule; 311. Visual unit; 3111. Appearance model; 3112. Model animation; 312. Speech unit; 3121. Speech synthesis; 3122. Natural language processing; 32. Assisted shopping submodule; 321. Product location guidance unit; 3211. Shopping route generation function; 3212. Light guidance function; 3213. Electronic map guidance function; 322. Product recommendation unit; 323. Product search unit; 324. Product display unit; 33. Shopping feedback submodule; 331. Product quality Quantity feedback unit; 332, expected product input unit; 34, self-service settlement submodule; 341, code scanning payment unit; 342, biometric payment unit; 343, contactless payment unit; 4, product quality monitoring module; 41, product expiration warning submodule; 42, storage environment abnormal alarm submodule; 5, cloud platform; 51, price tag management module; 52, data analysis module; 521, shopping feedback data analysis submodule; 522, product search data analysis submodule; 523, personalized recommendation submodule; 5231, bill data analysis unit ; 5232, customer behavior analysis unit; 52321, browsing behavior analysis function; 52322, stay behavior analysis function; 52323, selection behavior analysis function; 5233, recommendation plan generation unit; 524, inventory management submodule; 5241, sales data analysis unit; 5242, inventory data analysis unit; 5243, decision support unit; 6, electronic price tag module; 61, wireless communication submodule; 62, display submodule; 63, data processing submodule; 64, power submodule; 7, data security and privacy protection module. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Please see attached Figure 1 -Attached Fig.12, an embodiment provided by the present invention: a multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network, comprising a data acquisition module 1, a shopping process management module 2, an intelligent service module 3, a commodity quality monitoring module 4, a cloud platform 5, an electronic price tag module 6 and a data security and privacy protection module 7, the data acquisition module 1 respectively establishes data connections with the shopping process management module 2, the intelligent service module 3 and the commodity quality monitoring module 4, the cloud platform 5 respectively establishes data connections with the shopping process management module 2, the intelligent service module 3, the commodity quality monitoring module 4, the electronic price tag module 6 and the data security and privacy protection module 7, the shopping process management module 2 Establish a data connection with the intelligent service module 3; the data acquisition module 1 includes a visual sensor 11, a weight sensor 12, an acoustic sensor 13, a temperature and humidity sensor 14 and a sound sensor 15, the visual sensor 11 includes a high-definition camera 111 and a 3D camera 112, the high-definition camera 111 and the 3D camera 112 are used to obtain image data, the weight sensor 12 is used to obtain weight data, the acoustic sensor 13 obtains location data, the temperature and humidity sensor 14 is used to obtain environmental temperature and humidity data, and the sound sensor 15 is used to obtain sound data; the shopping process management module 2 includes a customer identification submodule 21, a bill creation submodule 22, a shopping behavior judgment submodule The shopping behavior judgment submodule 23 includes a product taking behavior judgment unit 231, a product returning behavior judgment unit 232, and an abnormal behavior judgment unit 233. The customer recognition submodule 21 is used for face recognition. The bill creation submodule 22 is used to generate an initial bill. The shopping behavior judgment submodule 23 is used to analyze shopping behavior. The bill update submodule 24 is used to update the initial bill data. The intelligent service module 3 includes an AI virtual digital human submodule 31, an assisted shopping submodule 32, a shopping feedback submodule 33, and a self-service settlement submodule 34. The AI ​​virtual digital human submodule 31 includes a visual unit 311 and a voice unit 312. Unit 312, the visual unit 311 includes an appearance model 3111 and a model animation 3112, the speech unit 312 includes speech synthesis 3121 and natural language processing 3122, the AI ​​virtual digital human submodule 31 is used for interaction, the shopping assistance submodule 32 is used for shopping assistance, the shopping feedback submodule 33 is used for shopping experience feedback, the self-service settlement submodule 34 is used for bill settlement, the visual unit 311 is used for visual display of the digital human, the appearance model 3111 is a digital human model, the model animation 3112 is a digital human interactive animation, the natural language processing 3122 is used for parsing natural language, and the speech synthesis 3121 is used for converting text into speech;The auxiliary shopping submodule 32 includes a product location guidance unit 321, a product recommendation unit 322, a product search unit 323 and a product display unit 324. The product location guidance unit 321 includes a shopping route generation function 3211, a light guidance function 3212 and an electronic map guidance function 3213. The product location guidance unit 321 is used to guide the shopping route, the product recommendation unit 322 is used to recommend products, the product search unit 323 is used to provide product search services, and the product display unit 324 is used to display basic product information. The shopping route generation function 3211 is used to plan the shopping route, the light guidance function 3212 is used to provide route guidance services, and the electronic map guidance function 3213 is used to provide route guidance services; the shopping feedback submodule 33 includes a product quality feedback unit 331 and a desired product entry unit 332. The self-service settlement submodule 34 includes a code scanning payment unit 341, a biometric payment unit 342 and a contactless payment unit 343. The product quality feedback unit 331 is used to enter product quality evaluation. The desired commodity input unit 332 is used to input the name of the commodity desired to be purchased; the commodity quality monitoring module 4 includes a commodity expiration warning submodule 41 and a storage environment abnormality alarm submodule 42, the commodity expiration warning submodule 41 is used to issue a commodity expiration reminder, and the storage environment abnormality alarm submodule 42 is used to issue a storage environment abnormality reminder; the cloud platform 5 includes a price tag management module 51 and a data analysis module 52, the data analysis module 52 includes a shopping feedback data analysis submodule 521, a commodity search data analysis submodule 522, a personalized recommendation submodule 523 and an inventory management submodule 524, the price tag management module 51 is used to remotely manage electronic price tags, the data analysis module 52 is used to analyze data using an AI algorithm, the shopping feedback data analysis submodule 521 is used to count shopping feedback data within a set time period, the commodity search data analysis submodule 522 is used to count commodity search data within a set time period, the personalized recommendation submodule 523 is used to generate a personalized commodity recommendation plan, and the inventory management submodule 524 is used for an inventory management plan;The personalized recommendation submodule 523 includes a bill data analysis unit 5231, a customer behavior analysis unit 5232, and a recommendation scheme generation unit 5233. The customer behavior analysis unit 5232 includes a browsing behavior analysis function 52321, a stay behavior analysis function 52322, and a selection behavior analysis function 52323. The inventory management submodule 524 includes a sales data analysis unit 5241, an inventory data analysis unit 5242, and a decision support unit 5243. The bill data analysis unit 5231 is used to statistically analyze the customer's bill data, the customer behavior analysis unit 5232 is used to analyze the browsing, stay, and selection time in the shopping behavior, and the recommendation scheme generation unit 5233 is used to comprehensively analyze the customer's bill data. The analysis results generate a product recommendation plan, the sales data analysis unit 5241 is used to analyze product sales data, the inventory data analysis unit 5242 is used to analyze product inventory data, and the decision support unit 5243 is used to generate an inventory management plan based on the comprehensive analysis results; the electronic price tag module 6 includes a wireless communication submodule 61, a display submodule 62, a data processing submodule 63 and a power submodule 64, and the data processing submodule 63 establishes data connections with the wireless communication submodule 61, the display submodule 62 and the power submodule 64 respectively, the wireless communication submodule 61 is used to realize wireless communication, the display submodule 62 is used to display data, the data processing submodule 63 is a control center, and the power submodule 64 is used to supply power. ;

[0030] Working principle: Data is collected using the data acquisition module 1, specifically: the high-definition camera 111 and the 3D camera 112 in the visual sensor 11 are used to collect image data, the weight sensor 12 is used to collect product weight data, the acoustic wave sensor 13 is used to collect customer location data, the temperature and humidity sensor 14 is used to detect the temperature and humidity data of the product storage environment, and the sound sensor 15 is used to collect customer voice information; the data acquisition module 1 is sent to the shopping process management module 2, the intelligent service module 3 and the product quality monitoring module 4, and the data of the shopping process management module 2, the intelligent service module 3 and the product quality monitoring module 4 are uploaded to the cloud platform 5 for storage. After the customer is successfully identified by the customer identification submodule 21, the customer can enter the store through the gate. At this time, the bill creation submodule 22 will create an initial bill for the customer. The initial bill contains customer information, order number information and product information. When the customer is shopping, the shopping behavior judgment submodule 23 will analyze the customer's behavior, and use the product taking behavior judgment unit 231 to judge whether the customer has taken the product, and use the product returning behavior judgment unit 232 to judge whether the customer has returned the product. The abnormal behavior judgment unit 233 is used to judge whether the customer's action is abnormal, specifically: the convolutional neural network algorithm is used to identify the customer's action of taking and putting goods, based on the movement trajectory of the goods, the starting position and the ending position are analyzed to determine the category of the goods, and according to the movement trajectory of the goods, the customer's selection behavior is judged to be taking the goods or putting the goods back. Specifically, by calculating the starting coordinates and the ending coordinates of the movement trajectory of the goods, it is judged whether the goods are moved from the near goods area to the far goods area or in the opposite direction. At the same time, it is also monitored whether the weight of the goods has changed. For example, if the customer takes away the goods, the weight data of the goods collected by the weight sensor 12 at that location will become zero, and if the customer puts the goods back, the weight sensor 12 at that location will re-detect the weight data of the goods. Even if the customer puts it in other locations, the corresponding weight sensor 12 can still collect the weight data of the goods. Therefore, after the data analysis results based on the visual sensor 11, the weight sensor 12 and the acoustic wave sensor 13 are classified, and then the results are fused, the goods can be accurately tracked and identified. The result fusion here can use weighted average to determine the final judgment result, specifically: final =ω V1 ×P V1 +ω V2 ×P V2 +ω W ×P W +ω S ×P S , where ω V1 +ω V2 +ω W +ω S =1,P V1 is the judgment result of the product based on visual data, and the weight is ω V1, P V2 is the judgment result of customer behavior based on visual data, and the weight is ω V2 , P W is the judgment result of the product based on weight data, and the weight is ω W , P S is the judgment result of the customer's location based on the sound wave data, and the weight is ω SWhen the customer takes the product and the weight data detected by the corresponding weight sensor 12 is zero, the bill update submodule 24 adds the product information to the initial bill. When the customer puts the product back and the corresponding weight sensor 12 detects that the added weight data is consistent with the product data, the bill update submodule 24 deletes the product information from the initial bill. When the customer puts the product back but the corresponding weight sensor 12 detects that the added weight data is inconsistent with the product data, the abnormal behavior judgment unit 233 judges that the customer has abnormal behavior and reminds the customer that the product cannot be put back. During the shopping process, the customer can use the intelligent service module 3 Assisted shopping, specifically: using the sound sensor 15 to use the customer's voice information to achieve interaction with the AI ​​virtual digital human sub-module 31, or accessing the intelligent service module 3 through the mobile phone to obtain the services of the assisted shopping sub-module 32, the shopping feedback sub-module 33 and the self-service settlement sub-module 34. The customer can add goods to the purchase and confirm the bill on the mobile phone, and can ask the AI ​​virtual digital human sub-module 31 for the location of the goods. The product location guidance unit 321 will formulate a shopping route through the shopping route generation function 3211 according to the storage location of the goods. According to the shopping route, the light guidance function 3212 uses the indicator light to guide the route, or the electronic map guidance function 32 13 provides map guidance on the mobile phone. The indicator light can be realized by the electronic price tag module 6. Different customers are guided by lights of different colors. During the guidance process, the customer location data is obtained by the acoustic wave sensor 13. After completing the shopping, the customer can check out by himself through the code scanning payment unit 341, the biometric payment unit 342 or the contactless payment unit 343. The purchased goods can be evaluated through the product quality feedback unit 331, and the desired goods can be entered through the desired product entry unit 332. Among them, the visual unit 311 is used to visualize the digital human, the appearance model 3111 is the digital human model, and the model animation 3112 is the digital human interactive animation , the speech unit 312 realizes speech and text interaction with customers through natural language processing 3122, and converts text into speech through speech synthesis 3121. The product recommendation unit 322 is used to provide personalized product recommendations for customers. The product search unit 323 is used to search for products. The product display unit 324 is used to display basic information of the products to customers; the product quality monitoring module 4 monitors the shelf life of the product through the product expiration warning submodule 41, and the storage environment abnormal alarm submodule 42 is used to analyze whether the temperature and humidity data of the product storage environment detected by the temperature and humidity sensor 14 meet the product storage requirements. If it does not meet the requirements, an abnormal storage environment alarm is immediately issued;In the cloud platform 5, the price tag management module 51 is used to remotely modify the information of the electronic price tag module 6. The data analysis module 52 uses the shopping feedback data analysis submodule 521 to count the shopping feedback data within a set time period, and uses the product search data analysis submodule 522 to count the product search data within a set time period. The personalized recommendation submodule 523 integrates the customer's basic information, purchase history, browsing behavior, search keywords and other multi-dimensional data to build a detailed customer portrait and generate a product recommendation plan for the customer. Here, a vector can be used to represent customer characteristics. By calculating the similarity between the customer portrait and the product characteristics, relevant products are recommended to the customer. The calculation formula can use the cosine similarity formula: Where u and v are the characteristic vectors of customers and commodities; an inventory management plan is generated using the inventory management submodule 524, specifically: the bill data analysis unit 5231 statistically analyzes the customer's bill data, the customer behavior analysis unit 5232 statistically analyzes the customer's behavior, and uses the browsing behavior analysis function 52321, the stay behavior analysis function 52322 and the selection behavior analysis function 52323 to analyze the time a customer browses a certain commodity, the stay time at a certain commodity and the time spent selecting a certain commodity; the recommendation plan generation unit 5233 comprehensively analyzes the analysis results of shopping feedback data, commodity search data, customer bill data and customer behavior to formulate a commodity recommendation plan for the customer; the sales data analysis unit 5241 is used to analyze commodity sales data, the inventory data analysis unit 5242 is used to analyze commodity inventory data, and the decision support unit 5243 comprehensively analyzes the analysis results of commodity sales data, commodity inventory data, shopping feedback data and commodity search data, and uses the autoregressive moving average model to predict the demand for different commodities in a period of time in the future, and determines the optimal inventory level based on the demand forecast results and combined with factors such as the supply cycle and cost of the commodity. , analyzing sales data and customer browsing and purchasing behaviors, it can be determined which products are slow-moving and which are best-selling products, thereby generating an inventory management plan; wherein, the wireless communication submodule 61 is used to realize wireless communication of the electronic price tag module 6, the display submodule 62 is used to display data, the data processing submodule 63 is the control center of the electronic price tag module 6, the power submodule 64 is used to power the electronic price tag module 6, the data security and privacy protection module 7 is used to ensure the security and compliance of customer data, and the shopping process management module 2, the intelligent service module 3 and the product quality monitoring module 4 are implemented on the edge computing device; the shopping process management module 2 uses a convolutional neural network for image recognition, specifically: preprocessing the image data, including image denoising, cropping, scaling, normalization and other operations to improve the quality and consistency of the data, and then extracting features from the preprocessed image, including action features and product packaging features, based on the extracted features, using the trained classification model to identify customer actions and products, combined with the depth information obtained by the 3D camera 112, by analyzing continuous image frames, the position changes of the products on the shelf are tracked in real time.

[0031] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network, comprising a data collection module (1), a shopping process management module (2), an intelligent service module (3), a product quality monitoring module (4), a cloud platform (5), an electronic price tag module (6) and a data security and privacy protection module (7). Features: The data acquisition module (1) establishes data connections with the shopping process management module (2), the intelligent service module (3) and the commodity quality monitoring module (4) respectively; the cloud platform (5) establishes data connections with the shopping process management module (2), the intelligent service module (3), the commodity quality monitoring module (4), the electronic price tag module (6) and the data security and privacy protection module (7) respectively; and the shopping process management module (2) establishes data connection with the intelligent service module (3).

2. According to claim 1, the multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network, Features: The data acquisition module (1) comprises a visual sensor (11), a weight sensor (12), an acoustic wave sensor (13), a temperature and humidity sensor (14) and a sound sensor (15); the visual sensor (11) comprises a high-definition camera (111) and a 3D camera (112); the high-definition camera (111) and the 3D camera (112) are used to obtain image data; the weight sensor (12) is used to obtain weight data; the acoustic wave sensor (13) is used to obtain position data; the temperature and humidity sensor (14) is used to obtain environmental temperature and humidity data; and the sound sensor (15) is used to obtain sound data.

3. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 1, Features: The shopping process management module (2) comprises a customer identification submodule (21), a bill creation submodule (22), a shopping behavior judgment submodule (23) and a bill update submodule (24); the shopping behavior judgment submodule (23) comprises a commodity taking behavior judgment unit (231), a commodity returning behavior judgment unit (232) and an abnormal behavior judgment unit (233); the customer identification submodule (21) is used for face recognition; the bill creation submodule (22) is used for generating an initial bill; the shopping behavior judgment submodule (23) is used for analyzing shopping behavior; and the bill update submodule (24) is used for updating initial bill data.

4. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 1, Features: The intelligent service module (3) comprises an AI virtual digital human submodule (31), an assisted shopping submodule (32), a shopping feedback submodule (33) and a self-service settlement submodule (34); the AI ​​virtual digital human submodule (31) comprises a visual unit (311) and a speech unit (312); the visual unit (311) comprises an appearance model (3111) and a model animation (3112); the speech unit (312) comprises speech synthesis (3121) and natural language processing (3122); the AI ​​virtual digital human submodule (31) is used for interaction; the assisted shopping submodule (32) is used for assisted shopping; the shopping feedback submodule (33) is used for feedback on shopping experience; the self-service settlement submodule (34) is used for settling bills; the visual unit (311) is used for visually displaying a digital human; the appearance model (3111) is a digital human model; the model animation (3112) is a digital human interactive animation; the natural language processing (3122) is used for parsing natural language; and the speech synthesis (3121) is used for converting text into speech.

5. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 4, Features: The auxiliary shopping submodule (32) includes a product location guidance unit (321), a product recommendation unit (322), a product search unit (323) and a product display unit (324). The product location guidance unit (321) includes a shopping route generation function (3211), a light guidance function (3212) and an electronic map guidance function (3213). The product location guidance unit (321) is used to guide the shopping route, the product recommendation unit (322) is used to recommend products, the product search unit (323) is used to provide product search services, the product display unit (324) is used to display basic product information, the shopping route generation function (3211) is used to plan the shopping route, the light guidance function (3212) is used to provide route guidance services, and the electronic map guidance function (3213) is used to provide route guidance services.

6. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 4, Features: The shopping feedback submodule (33) comprises a product quality feedback unit (331) and a desired product input unit (332); the self-service settlement submodule (34) comprises a code scanning payment unit (341), a biometric payment unit (342) and a contactless payment unit (343); the product quality feedback unit (331) is used to input product quality evaluation, and the desired product input unit (332) is used to input the name of the desired product.

7. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 1, Features: The commodity quality monitoring module (4) comprises a commodity near-expiry warning submodule (41) and a storage environment abnormality alarm submodule (42); the commodity near-expiry warning submodule (41) is used to issue a commodity near-expiry reminder, and the storage environment abnormality alarm submodule (42) is used to issue a storage environment abnormality reminder.

8. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 1, Features: The cloud platform (5) comprises a price tag management module (51) and a data analysis module (52); the data analysis module (52) comprises a shopping feedback data analysis submodule (521), a commodity search data analysis submodule (522), a personalized recommendation submodule (523) and an inventory management submodule (524); the price tag management module (51) is used for remotely managing electronic price tags; the data analysis module (52) is used for analyzing data using an AI algorithm; the shopping feedback data analysis submodule (521) is used for statistically analyzing shopping feedback data within a set time period; the commodity search data analysis submodule (522) is used for statistically analyzing commodity search data within a set time period; the personalized recommendation submodule (523) is used for generating personalized commodity recommendation plans; and the inventory management submodule (524) is used for inventory management plans.

9. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 8, Features: The personalized recommendation submodule (523) includes a bill data analysis unit (5231), a customer behavior analysis unit (5232) and a recommendation scheme generation unit (5233). The customer behavior analysis unit (5232) includes a browsing behavior analysis function (52321), a stay behavior analysis function (52322) and a selection behavior analysis function (52323). The inventory management submodule (524) includes a sales data analysis unit (5241), an inventory data analysis unit (5242) and a decision support unit (5243). The bill data analysis unit (5231) is used to statistically analyze the bill data of customers. The customer behavior analysis unit (5232) is used to analyze the browsing, stay and selection time in shopping behavior. The recommendation scheme generation unit (5233) is used to generate a commodity recommendation scheme based on the comprehensive analysis results. The sales data analysis unit (5241) is used to analyze commodity sales data. The inventory data analysis unit (5242) is used to analyze commodity inventory data. The decision support unit (5243) is used to generate an inventory management scheme based on the comprehensive analysis results.

10. The multimodal fusion intelligent unmanned retail system based on computer vision convolutional neural network according to claim 1, Features: The electronic price tag module (6) comprises a wireless communication submodule (61), a display submodule (62), a data processing submodule (63) and a power submodule (64), and the data processing submodule (63) respectively establishes data connections with the wireless communication submodule (61), the display submodule (62) and the power submodule (64), the wireless communication submodule (61) is used to realize wireless communication, the display submodule (62) is used to display data, the data processing submodule (63) is a control center, and the power submodule (64) is used to supply power.

Images