A shopping cart and a smart recognition and checkout method based on the shopping cart

Abstract

This application discloses a shopping cart and a smart recognition and checkout method based on the shopping cart, relating to the field of smart shopping technology. The method includes: acquiring a binding signal sent by a user terminal; acquiring user data and shopping cart status information fed back by a positioning communication module and a cloud server based on the binding signal; receiving a product placement signal sent by a bottom weighing module, and obtaining product placement information based on the product placement signal, image data collected by a visual sensor, and a deep learning model; generating a first product list based on the product placement information; acquiring a checkout signal sent by the user terminal, and completing the checkout based on the user data, the shopping cart status information, a multimodal smart recognition module, the checkout signal, and the first product list. By combining the bottom weighing module, the visual sensor, and the deep learning model, product information can be more accurately identified, achieving high-precision synchronous identification of product type, quantity, and weight.

Description

Technical Field

[0001] This application relates to the field of smart shopping technology, and in particular to a shopping cart and a smart recognition and checkout method based on the shopping cart. Background Technology

[0002] With the rapid development of smart technology and shopping carts, various smart shopping cart technologies have emerged in the field of supermarket smart upgrades. Their ability to identify products directly affects the accuracy of determining the purchased items and the reliability of the checkout process.

[0003] Existing visual recognition solutions primarily rely on a single camera for image recognition. When products obscure each other, lighting conditions change, or products look similar, misidentification or missed identification can easily occur. Although some solutions supplement with weight sensors for verification, the weight data is only used for simple comparison and is not deeply integrated with visual recognition. This results in low accuracy in recognizing weighed goods (such as bulk fruits and vegetables) and difficulty in accurately matching the correspondence between "what kind of product corresponds to what weight." Existing shopping cart visual recognition solutions also rely heavily on a single camera for image recognition, leading to insufficient accuracy and a tendency for misidentification or missed identification. Summary of the Invention

[0004] The main purpose of this application is to provide a shopping cart and a smart recognition and checkout method based on the shopping cart, which aims to solve the technical problem that existing shopping cart visual recognition solutions mainly rely on a single camera for image recognition, resulting in insufficient recognition accuracy.

[0005] To achieve the above objectives, this application proposes a smart recognition and checkout method based on a shopping cart. The method is applied to an in-vehicle smart terminal of the shopping cart. The in-vehicle smart terminal is connected to a multimodal smart recognition module and a positioning communication module via wired connections. The in-vehicle smart terminal is connected to a cloud server via a network and to a user terminal via Bluetooth. The edge computing unit includes a deep learning model. The method includes: Obtain the binding signal sent by the user terminal; Based on the binding signal, user data and shopping cart status information fed back by the positioning communication module and the cloud server are obtained; The system receives a product placement signal from the bottom weighing module and obtains product placement information based on the product placement signal, image data collected by the vision sensor, and the deep learning model. A first product list is generated based on the product information provided. The system acquires the checkout signal sent by the user terminal and completes the checkout process based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

[0006] In one embodiment, the step of obtaining the product placement information based on the product placement signal, the image data collected by the visual sensor, and the deep learning model includes: The weight waveform of the placed product is obtained based on the product placement signal; Acquire the image sequence of the placed product fed back by the top vision sensor, the side vision sensor, and the fill light; The weight waveform and the image sequence are matched using a deep learning model to obtain the product information.

[0007] In one embodiment, the in-vehicle intelligent terminal is connected to the anti-theft warning module via a wired connection. After the step of generating the first product list based on the added product information, the method further includes: If a product removal signal is received from the bottom weighing module, the product removal information fed back by the vision sensor is obtained; If the retrieved product information is in the first product list, then based on the retrieved product information, the retrieved product location fed back by the positioning communication module and the visual sensor is obtained; and based on the retrieved product location, the anti-theft warning module is controlled to trigger the corresponding warning process.

[0008] In one embodiment, the step of controlling the anti-theft alarm module to trigger a corresponding alarm process based on the location of the retrieved goods includes: If the location of the retrieved product is within a preset range, then the retrieved product information is deleted from the first product list; If the location of the retrieved goods is not within the preset range, the anti-theft warning module is controlled to trigger a graded alarm process, which includes a first-level alarm and a second-level alarm. The Level 1 alarm includes: Control the touch screen to display a prompt box and trigger a preset voice message; Send a reminder message and a correction window displaying a preset time to the user terminal; If a correction signal is received from the multimodal intelligent recognition module within the preset time, the first-level alarm is cancelled. If an uncorrected signal is received from the multimodal intelligent recognition module within the preset time, a level two alarm will be triggered.

[0009] In one embodiment, the vehicle-mounted intelligent terminal is connected to the staff terminal via Bluetooth, and the secondary alarm includes: The shopping cart number and location are obtained based on the shopping cart status information; Obtain abnormal image fragments fed back by the multimodal intelligent recognition module, and generate abnormal event information based on the shopping cart location, the shopping cart number, and the abnormal image fragments; The abnormal event information is sent to the staff terminal, and the audible and visual alarm is activated.

[0010] In one embodiment, the step of completing the checkout based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list includes: Based on the checkout signal, obtain the shopping cart product information fed back by the multimodal intelligent recognition module; Verification is performed based on the product information in the shopping cart and the first product list; If the verification passes, a payment bill will be generated based on the first product list; Send the payment bill to the user terminal; Based on the payment bill, the user terminal sends a payment signal; A second product list is generated based on the payment signal and the first product list; The user data and shopping cart status information are updated according to the second product list to obtain the updated user data and updated shopping cart status information; Complete the checkout process based on the updated user data and the updated shopping cart status information.

[0011] In one embodiment, after the step of obtaining user data and shopping cart status information through the positioning communication module and the cloud server, the method further includes: Acquire the search signal from the user terminal or the touch screen, and determine the product coordinates based on the search signal; The shopping cart location is obtained based on the shopping cart status information; A navigation map is generated based on the shopping cart location and the product coordinates.

[0012] Furthermore, to achieve the above objectives, this application also proposes a shopping cart, which includes: The vehicle body includes a basket, a frame, handrails, casters, and handles; The vehicle-mounted intelligent terminal is installed in the middle of the crossbar of the handrail, and the vehicle-mounted intelligent terminal includes at least an edge computing unit, a touch screen, a network communication module and a Bluetooth communication module; A multimodal intelligent recognition module, wherein the multimodal intelligent recognition module is arranged and installed in multiple directions around the basket; A positioning and communication module is installed on the vehicle body and the basket. An anti-theft warning module is installed on the vehicle body, and the anti-theft warning module includes at least an audible and visual alarm. The in-vehicle intelligent terminal is used to acquire the binding signal sent by the user terminal; Based on the binding signal, user data and shopping cart status information fed back by the positioning communication module and the cloud server are obtained; The system receives a product placement signal from the bottom weighing module and obtains product placement information based on the product placement signal, image data collected by the vision sensor, and a deep learning model. A first product list is generated based on the product information provided. The system acquires the checkout signal sent by the user terminal and completes the checkout process based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

[0013] The multimodal intelligent recognition module includes: A top vision sensor, which is mounted on the top bracket of the basket; A side vision sensor is mounted on the upper part of the side wall of the basket; A bottom weighing module is installed between the bottom of the basket and the frame. A supplementary light is arranged and installed around the top vision sensor and the side vision sensor.

[0014] The positioning and communication module includes: A positioning tag is installed on the bottom of the vehicle body and is connected to a positioning base station via a wireless communication protocol; A near-field recognition module is installed at the entrance of the basket.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: 1) By using a multimodal fusion model, the candidate list of visually recognized products is spatiotemporally aligned and matched with weight change features. The type, quantity, and precise weight of the products are output at once, and the shopping list and cumulative amount are automatically updated, thus achieving more accurate identification of shopping products.

[0016] 2) By combining the bottom weighing module, visual sensors, and the judgment logic of the anti-theft alarm, the anti-theft of the smart shopping cart in the supermarket has been upgraded, taking into account both the user experience and the security needs of the supermarket.

[0017] 3) By using UWB (Ultra-Wideband) positioning technology, the user experience of finding the products they need is optimized, and the security requirements of supermarket smart shopping carts are also improved. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the intelligent recognition and checkout method based on a shopping cart in this application. Figure 2 This is the overall assembly drawing of the shopping cart for this application; Figure 3 This is a schematic diagram of a touch screen on a shopping cart provided in an embodiment of the intelligent recognition and checkout method based on a shopping cart according to this application; Figure 4 This is a schematic diagram of the weighing module at the bottom of the shopping cart provided in an embodiment of the intelligent recognition and checkout method based on the shopping cart in this application; Figure 5 This is a schematic diagram of the top visual sensor in the multimodal intelligent recognition module on the shopping cart, provided in an embodiment of the intelligent recognition and checkout method based on the shopping cart of this application. Figure 6 This is a flowchart illustrating the intelligent checkout process provided in an embodiment of the intelligent recognition and checkout method based on a shopping cart according to this application. Figure 7 This is a flowchart illustrating Embodiment 2 of the intelligent recognition and checkout method based on a shopping cart in this application. Figure 8 This is a flowchart illustrating Embodiment 3 of the intelligent recognition and checkout method based on a shopping cart in this application; Figure 9 This is a flowchart illustrating an embodiment of the intelligent shopping process based on a shopping cart for intelligent identification and checkout in this application. Figure 10 This is a schematic diagram of the module structure of the shopping cart in an embodiment of this application; Explanation of icon numbers: The vehicle body 10, the vehicle-mounted intelligent terminal 20, the multimodal intelligent recognition module 30, the positioning and communication module 40, and the anti-theft warning module 50.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: Obtain a binding signal sent by the user terminal; based on the binding signal, obtain user data and shopping cart status information fed back by the positioning communication module and the cloud server; receive a product placement signal sent by the bottom weighing module, and obtain product placement information based on the product placement signal, image data collected by the visual sensor, and a deep learning model; generate a first product list according to the product placement information; obtain a checkout signal sent by the user terminal, and complete the checkout based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

[0025] Because existing visual recognition solutions primarily rely on a single camera for image recognition, they are prone to misidentification or missed identification when products obscure each other, lighting conditions change, or products look similar. Although some solutions supplement with weight sensors for verification, the weight data is only used for simple comparison and is not deeply integrated with visual recognition, resulting in low recognition accuracy for weighed products (such as bulk fruits and vegetables) and difficulty in accurately matching the correspondence between "what kind of product corresponds to what weight". Existing shopping cart visual recognition solutions mainly rely on a single camera for image recognition, resulting in insufficient recognition accuracy and a tendency to misidentify or miss identification.

[0026] This application provides a solution that utilizes a bottom weighing module, a visual sensor, and a deep learning model to collect real-time information about items placed in the shopping cart, achieving accurate identification of the placed items. Based on this information, a first item list is generated. By establishing this first item list and combining it with algorithmic logic, subsequent steps such as checkout and anti-theft alarm processes are implemented, enabling an efficient, safe, and intelligent supermarket shopping experience.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or in-vehicle intelligent terminal capable of performing the above functions. The following description uses an in-vehicle intelligent terminal on a shopping cart as an example to illustrate this embodiment and the subsequent embodiments. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the corresponding device.

[0028] Based on this, embodiments of this application provide a smart recognition and checkout method based on a shopping cart, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent recognition and checkout method based on a shopping cart according to this application. The method is applied to an in-vehicle intelligent terminal of a shopping cart. The in-vehicle intelligent terminal is connected to a multimodal intelligent recognition module and a positioning communication module via wired connections. The in-vehicle intelligent terminal is connected to a cloud server via a network and to a user terminal via Bluetooth. The edge computing unit includes a deep learning model.

[0029] In addition, this application also provides a shopping cart, as shown in the embodiments. Figure 2 , Figure 2 This is the overall assembly drawing of the shopping cart for this application.

[0030] The shopping cart includes: a cart body, comprising a basket, frame, handlebars, casters, and handles; an in-vehicle intelligent terminal, installed in the middle of the crossbar of the handlebars, the in-vehicle intelligent terminal including at least an edge computing unit, a touch screen, a network communication module, and a Bluetooth communication module; a multimodal intelligent recognition module, arranged and installed in multiple directions around the basket; a positioning and communication module, installed on the cart body and the basket; and an anti-theft warning module, installed on the cart body, the anti-theft warning module including at least an audible and visual alarm.

[0031] The multimodal intelligent recognition module includes: a top vision sensor mounted on the top bracket of the basket; a side vision sensor mounted on the upper part of the side wall of the basket; a bottom weighing module mounted between the bottom of the basket and the frame; and a supplementary light arranged and installed around the top vision sensor and the side vision sensor.

[0032] The positioning and communication module includes: a positioning tag installed on the bottom of the vehicle body, which is connected to a positioning base station via a wireless communication protocol; and a near-field identification module installed at the entrance of the vehicle basket.

[0033] In this embodiment, the intelligent recognition and checkout method based on the shopping cart includes steps S10 to S50: Step S10: Obtain the binding signal sent by the user terminal.

[0034] It should be noted that the user terminal includes at least a mobile phone. The usage scenario can be as follows: the user scans the QR code on the shopping cart armrest with his mobile phone (or a mini-program on the mobile phone) to complete the binding of the shopping cart with the user, and a welcome screen will be displayed on the touch screen. After the user completes the binding, a binding signal will be generated, and the vehicle's smart terminal can obtain the binding signal sent by the user terminal.

[0035] The touchscreen display is multifunctional, allowing users to complete tasks such as linking items to their shopping cart, navigating to desired products, and viewing real-time balances. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the touch screen display on the shopping cart in this application.

[0036] Step S20: Based on the binding signal, obtain user data and shopping cart status information fed back by the positioning communication module and the cloud server.

[0037] It should be noted that user data typically includes information such as the user's ID (identifier), nickname, points, and coupons; while shopping cart status information typically includes information such as shopping cart ID, usage status, and initial position of the shopping cart.

[0038] Understandably, after a user binds their shopping cart, the user's terminal sends user data to the cloud server. The UWB positioning tag in the positioning communication module communicates with the UWB positioning base station preset in the supermarket via UWB pulse signals. The location information of the shopping cart is collected by the base station and transmitted to the cloud server, and then sent to the vehicle-mounted smart terminal. The cloud server usually saves user data and shopping cart status information.

[0039] In addition, the supermarket is equipped with UWB positioning base stations. UWB positioning tags connect to these base stations via the UWB air interface protocol. UWB positioning primarily relies on time difference of arrival (TDOA) or two-way time of flight. Two-way ranging method: The UWB positioning tag on the shopping cart exchanges signals multiple times with a fixed UWB base station within the supermarket. By measuring the signal's flight time and multiplying it by the speed of light, the precise distance between the UWB positioning tag and the base station is calculated. At least three base stations are required. The tag's two-dimensional coordinates can then be calculated using triangulation. Time difference of arrival method: The base stations maintain high-precision clock synchronization. When the UWB positioning tag sends a signal, multiple UWB base stations receive and record the absolute arrival time of the signal. The tag's position is calculated based on the time difference.

[0040] After obtaining user data and shopping cart status information through the positioning communication module and the cloud server, the method may further include: obtaining the search signal of the user terminal or the touch screen, determining the product coordinates based on the search signal; obtaining the shopping cart location based on the shopping cart status information; and generating a navigation map based on the shopping cart location and the product coordinates.

[0041] Understandably, when users need to find products, location-based navigation can provide a better shopping experience. UWB positioning tags send signals in real time, which are received and the UWB positioning base station calculates the shopping cart's location, then feeds it back to the in-vehicle smart terminal via a cloud server. For example, customers can search for products on a touchscreen or their mobile phone. Based on the shopping cart's current location and the product location information fed back from the cloud (pre-entered shelf coordinates), the in-vehicle smart terminal plans the optimal navigation route and dynamically displays directional arrows on the touchscreen.

[0042] Step S30: Receive the product placement signal sent by the bottom weighing module, and obtain product placement information based on the product placement signal, the image data collected by the vision sensor, and the deep learning model.

[0043] It should be noted that when the bottom weighing module detects an increase in weight (a preset threshold can be set; when the increased weight exceeds the preset threshold, such as setting the preset threshold to 20g), it sends a signal to place the product. The vehicle's intelligent terminal will identify the type, quantity, and weight of the product based on the product placement signal, visual sensors, and deep learning models. For example, for weighed products, the weight can be obtained directly, while for piece-counted products, the weight is used as a logical verification to obtain the type and quantity of piece-counted products.

[0044] In addition, when the vision sensor is performing identification work, the near-field identification module will also assist in identifying products with RFID (Radio Frequency Identification) tags, enhancing the reliability of identification.

[0045] For easier understanding, please refer to Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of the bottom weighing module on the shopping cart in this application. Figure 5 This is a schematic diagram of the top visual sensor in the multimodal intelligent recognition module of the shopping cart in this application.

[0046] Step S40: Generate a first product list based on the product information provided.

[0047] It should be noted that the first product list is usually the list of products to be settled. After obtaining the product information, the product information is added to the list of products to be settled and displayed in the touch screen shopping list. The cumulative amount is calculated and updated in real time.

[0048] Furthermore, the theoretical weight is determined based on the first product list. For products that are easy to count, the theoretical weight is the product of the number of identified products and the pre-stored "standard weight per unit" for each product. For loose products, the vehicle-mounted smart terminal does not rely on the pre-stored "standard weight per unit" but directly uses the actual incremental weight measured by the weighing module at the bottom when the product is put into the shopping cart as the theoretical weight of the batch of loose products, that is, the weight difference before and after being put in. For loose products of different sizes, the theoretical weight does not come from the pre-stored "standard weight per unit" but is the real-time weighing value each time it is put in, and the weight is dynamically updated and adjusted as some are taken out.

[0049] Step S50: Obtain the checkout signal sent by the user terminal, and complete the checkout based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

[0050] Specifically, based on the checkout signal, the system obtains the shopping cart product information fed back by the multimodal intelligent recognition module; verifies the shopping cart product information against the first product list; if the verification passes, it generates a payment bill based on the first product list; sends the payment bill to the user terminal; receives the payment signal sent by the user terminal based on the payment bill; generates a second product list based on the payment signal and the first product list; updates the user data and the shopping cart status information based on the second product list to obtain updated user data and updated shopping cart status information; and completes the checkout based on the updated user data and updated shopping cart status information.

[0051] It should be noted that the second item list is usually a list of items already checked out. For example, when a customer clicks the "One-Click Checkout" button on the shopping cart touchscreen or mobile app, the vision sensor quickly scans all items in the cart and compares them with the "Items to be Checked List" to confirm that no items have been missed or over-checked. Simultaneously, the onboard smart terminal reads the real-time total weight from the bottom weighing module and verifies it against the theoretical weight of the items to be checked list. If the weight difference is within a preset range, the verification passes, and the process continues. If the verification fails, the customer is prompted to "please confirm that all items have been correctly placed, or re-check." After successful verification, a payment is generated based on the items to be checked list. The payment bill typically includes the unit price, quantity, weight, and total amount of each item. The bill is sent to the user's terminal and touch screen. The user can complete the payment through a mobile app or touch screen. After successful payment, the status of all items in the pending items list is updated to "checked out" and copied to the checked items list. At the same time, the pending items list is cleared, points for this purchase are calculated, coupons are redeemed, and shopping details, points changes, and other information are synchronized to the mobile app and stored in the shopping history. The shopping cart status is automatically updated to "idle". If the shopping cart is not returned to its original position for a long time, staff can be notified to retrieve it. Meanwhile, the user profile is updated in the cloud.

[0052] Specifically, please refer to Figure 6 For ease of understanding, Figure 6 This is a flowchart of the smart checkout process based on a shopping cart.

[0053] This embodiment uses a multimodal fusion model to perform spatiotemporal alignment and correlation matching between the visually recognized product candidate list and weight change features, outputting the type, quantity, and precise weight of the products in one go, and automatically updating the shopping list and cumulative amount, thus achieving more accurate identification of shopping products.

[0054] This embodiment provides a smart identification and checkout method based on a shopping cart. It acquires a binding signal sent by a user terminal; based on the binding signal, it acquires user data and shopping cart status information fed back by a positioning communication module and a cloud server; it receives a product placement signal sent by a bottom weighing module, and obtains product placement information based on the product placement signal, image data collected by a visual sensor, and a deep learning model; it generates a first product list based on the product placement information; it acquires a checkout signal sent by the user terminal, and completes checkout based on the user data, the shopping cart status information, the multimodal smart identification module, the checkout signal, and the first product list. By combining the bottom weighing module, the visual sensor, and the deep learning model, it can more accurately identify product information and achieve high-precision synchronous identification of product type, quantity, and weight.

[0055] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 7 Step S30 includes steps S301 to S303: Step S301: Receive the product placement signal sent by the bottom weighing module, and obtain the weight waveform of the placed product based on the product placement signal.

[0056] It should be noted that the weight waveform refers to the curve of the electrical signal output by the bottom weighing module changing over time as the weight of the object being measured changes.

[0057] Step S302: Acquire the image sequence of the placed product fed back by the top vision sensor, the side vision sensor, and the fill light.

[0058] It should be noted that an image sequence refers to a collection of images arranged in chronological order; it is essentially a raw video in its original format without compression or encoding.

[0059] Specifically, the top and side vision sensors continuously capture images (e.g., 30 frames per second for 2 seconds) to obtain image sequences during the product placement process, and the supplementary lighting will automatically turn on according to the lighting conditions.

[0060] Step S303: The weight waveform and the image sequence are matched using a deep learning model to obtain the product information.

[0061] It should be noted that the deep learning model is pre-installed in the edge computing unit of the in-vehicle intelligent terminal. It processes image sequences and weight waveforms simultaneously to identify the type, quantity, and weight of the goods, thereby obtaining the information of the goods.

[0062] This embodiment acquires the weight waveform of the placed item based on the item placement signal; it acquires image sequences of the placed item from the top visual sensor, the side visual sensor, and the supplementary light; and it uses a deep learning model to match the weight waveform and the image sequences to obtain the placed item information. By combining the deep learning model, weight waveform, and image sequences, the visually recognized item candidate list is spatiotemporally aligned and correlated with weight change features, outputting the type, quantity, and precise weight of the placed item in a single step, thus improving the accuracy of item recognition.

[0063] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 8Step S40 is followed by steps S41 to S42: Step S41: If a product removal signal is received from the bottom weighing module, obtain the product removal information fed back by the vision sensor.

[0064] It should be noted that when the bottom weighing module detects a decrease in weight, it will send a product removal signal to the vehicle's smart terminal, and at the same time trigger the vision sensor to capture an image of the removed product and identify the product information (product type and quantity).

[0065] Step S42: If the retrieved product information is in the first product list, then based on the retrieved product information, obtain the retrieved product location fed back by the positioning communication module and the visual sensor; and based on the retrieved product location, control the anti-theft alarm module to trigger the corresponding alarm process.

[0066] It should be noted that the in-vehicle intelligent terminal checks whether the removed item exists in the pending checkout list. If the removed item is not in the pending checkout list, it is determined that the removed item exists in the already checked items list, and the item is removed normally without triggering an alarm. This situation only occurs when a customer removes a paid item after checkout (e.g., for inspection or return) and is not within the scope of anti-theft monitoring. If the removed item is in the pending checkout list, the location of the removed item is further determined using UWB positioning and visual sensors, and based on the location of the removed item, the anti-theft alarm module is controlled to trigger the corresponding alarm process.

[0067] Further, if the location of the retrieved item is within a preset range, the retrieved item information is deleted from the first item list; if the location of the retrieved item is not within the preset range, the anti-theft warning module is controlled to trigger a tiered alarm process, which includes a first-level alarm and a second-level alarm. The first-level alarm includes: controlling the touch screen to pop up a prompt box and triggering a preset voice; sending a reminder and a correction window displaying a preset time to the user terminal; if a correction signal is received from the multimodal intelligent recognition module within the preset time, the first-level alarm is canceled; if a still uncorrected signal is received from the multimodal intelligent recognition module within the preset time, the second-level alarm is triggered. The second-level alarm includes: obtaining the shopping cart number and shopping cart location based on the shopping cart status information; obtaining abnormal image fragments fed back by the multimodal intelligent recognition module, generating abnormal event information based on the shopping cart location, the shopping cart number, and the abnormal image fragments; sending the abnormal event information to the staff terminal and activating the sound and light alarm.

[0068] Understandably, the preset range is usually set on supermarket shelves. If it's determined that an item has been returned to the shelf, it's considered a normal abandonment, and the item is removed from the checkout list, updating the checkout list and its theoretical weight (reducing the corresponding weight). This situation does not trigger an alarm. If it's determined that an item has not been returned to the shelf, it's considered an abnormal removal, triggering a tiered alarm process: The first-level alarm usually alerts the user, specifically through a pop-up notification on the vehicle's display screen accompanied by a voice message: "Hello, the item you took has not yet been checked out. Please put it back in your shopping cart or confirm your purchase." Simultaneously, the mobile app receives the same notification, providing a preset correction window (usually set at 15-30 seconds). Correction judgment: If within the correction window period, multimodal... If the intelligent recognition module detects that the item has been returned (increased weight and visually confirmed), it sends a correction signal to the vehicle's intelligent terminal and automatically cancels the alarm, restoring the checkout list. If, within the correction window, the vehicle's intelligent terminal receives an uncorrected signal from the multimodal intelligent recognition module, it triggers a secondary alarm. Secondary alarms usually require employee intervention. Specifically, abnormal event information (shopping cart number, shopping cart location, and abnormal item image fragments) is sent to the employee's terminal via the cloud. Simultaneously, the audible and visual alarm on the shopping cart is activated with low-frequency flashing and beeping. In extreme cases, the electronic lock of the anti-theft warning module can be activated to lock the wheels. In addition, abnormal item image fragments are usually videos or photos of the user abnormally taking items.

[0069] Specifically, please refer to Figure 9 For ease of understanding, Figure 9 This is a flowchart of a smart shopping process based on a shopping cart.

[0070] This embodiment uses a combination of a bottom weighing module and a multimodal intelligent recognition module to determine the status of the retrieved goods. Based on the status of the retrieved goods, it determines whether to trigger a tiered alarm process. The first-level alarm sends a friendly reminder to the customer through the shopping cart terminal and the linked mobile phone, and provides a correction window. If the correction fails, the second-level alarm is triggered, pushing abnormal event information to the staff's handheld terminal and the shopping cart's audio and visual warning. Through intelligent detection and tiered alarm processes, the anti-theft method is upgraded, and the user experience is also guaranteed.

[0071] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the intelligent identification and checkout method based on the shopping cart in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0072] This application also provides a shopping cart, please refer to... Figure 10 The shopping cart includes: The vehicle body 10 includes a basket, a frame, a handrail, casters, and a handle.

[0073] The vehicle-mounted intelligent terminal 20 is installed in the middle of the crossbar of the handrail. The vehicle-mounted intelligent terminal 20 includes at least an edge computing unit, a touch screen, a network communication module, and a Bluetooth communication module.

[0074] A multimodal intelligent recognition module 30 is arranged and installed in multiple directions around the basket. The multimodal intelligent recognition module 30 includes: a top vision sensor, which is installed on the top bracket of the basket; a side vision sensor, which is installed on the upper part of the side wall of the basket; a bottom weighing module, which is installed between the bottom of the basket and the frame; and a supplementary light, which is arranged and installed around the top vision sensor and the side vision sensor.

[0075] The positioning and communication module 40 is installed on the vehicle body and the basket; the positioning and communication module 40 includes: a positioning tag installed on the bottom of the vehicle body, the positioning tag being connected to a positioning base station via a wireless communication protocol; and a near-field identification module installed at the entrance of the basket.

[0076] An anti-theft warning module 50 is installed on the vehicle body, and the anti-theft warning module 50 includes at least an audible and visual alarm.

[0077] The in-vehicle intelligent terminal 20 is used to acquire a binding signal sent by a user terminal; based on the binding signal, acquire user data and shopping cart status information fed back by the positioning communication module and the cloud server; receive a product placement signal sent by the bottom weighing module, and obtain product placement information based on the product placement signal, a visual sensor, and a deep learning model; generate a first product list based on the product placement information; acquire a checkout signal sent by the user terminal, and complete the checkout based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

[0078] In one embodiment, the vehicle-mounted intelligent terminal 20 is further configured to acquire the weight waveform of the placed product based on the product placement signal; acquire the image sequence of the placed product fed back by the top vision sensor, the side vision sensor, and the supplementary light; and perform matching processing on the weight waveform and the image sequence through a deep learning model to obtain the placed product information.

[0079] In one embodiment, the vehicle-mounted intelligent terminal 20 is further configured to, if it receives a product removal signal sent by the bottom weighing module, obtain product removal information fed back by the visual sensor; if the product removal information is in the first product list, obtain the product removal location fed back by the positioning communication module and the visual sensor based on the product removal information; and control the anti-theft warning module to trigger a corresponding warning process based on the product removal location.

[0080] In one embodiment, the in-vehicle intelligent terminal 20 is further configured to: delete the retrieved product information from the first product list if the retrieved product location is within a preset range; and control the anti-theft warning module to trigger a tiered alarm process if the retrieved product location is not within the preset range. The tiered alarm process includes a first-level alarm and a second-level alarm. The first-level alarm includes: controlling the touch screen to pop up a prompt box and triggering a preset voice message; sending a reminder message and a correction window displaying a preset time to the user terminal; canceling the first-level alarm if a correction signal is received from the multimodal intelligent recognition module within the preset time; and triggering a second-level alarm if a non-correction signal is received from the multimodal intelligent recognition module within the preset time.

[0081] In one embodiment, the in-vehicle intelligent terminal 20 is further configured to obtain the shopping cart number and shopping cart location based on the shopping cart status information; obtain abnormal image fragments fed back by the multimodal intelligent recognition module; generate abnormal event information based on the shopping cart location, the shopping cart number, and the abnormal image fragments; send the abnormal event information to the staff terminal; and activate the audible and visual alarm.

[0082] In one embodiment, the in-vehicle intelligent terminal 20 is further configured to: acquire shopping cart product information fed back by the multimodal intelligent recognition module based on the checkout signal; perform verification based on the shopping cart product information and the first product list; if the verification passes, generate a payment bill based on the first product list; send the payment bill to the user terminal; receive a payment signal sent by the user terminal based on the payment bill; generate a second product list based on the payment signal and the first product list; update the user data and the shopping cart status information based on the second product list to obtain updated user data and updated shopping cart status information; and complete the checkout based on the updated user data and updated shopping cart status information.

[0083] In one embodiment, the in-vehicle intelligent terminal 20 is further configured to acquire the search signal of the user terminal or the touch screen, determine the product coordinates based on the search signal, acquire the shopping cart position based on the shopping cart status information, and generate a navigation map based on the shopping cart position and the product coordinates.

[0084] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0086] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0087] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A shopping cart, characterized in that, The shopping cart includes: The vehicle body includes a basket, a frame, handrails, casters, and handles; The vehicle-mounted intelligent terminal is installed in the middle of the crossbar of the handrail, and the vehicle-mounted intelligent terminal includes at least an edge computing unit, a touch screen, a network communication module and a Bluetooth communication module; A multimodal intelligent recognition module, wherein the multimodal intelligent recognition module is arranged and installed in multiple directions around the basket; A positioning and communication module is installed on the vehicle body and the basket. An anti-theft warning module is installed on the vehicle body, and the anti-theft warning module includes at least an audible and visual alarm. The in-vehicle intelligent terminal is used to acquire the binding signal sent by the user terminal; Based on the binding signal, user data and shopping cart status information fed back by the positioning communication module and the cloud server are obtained; The system receives a product placement signal from the bottom weighing module and obtains product placement information based on the product placement signal, image data collected by the vision sensor, and a deep learning model. A first product list is generated based on the product information provided. The system acquires the checkout signal sent by the user terminal and completes the checkout process based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

2. The shopping cart as described in claim 1, characterized in that, The multimodal intelligent recognition module includes: A top vision sensor, which is mounted on the top bracket of the basket; A side vision sensor is mounted on the upper part of the side wall of the basket; A bottom weighing module is installed between the bottom of the basket and the frame. A supplementary light is arranged and installed around the top vision sensor and the side vision sensor.

3. The shopping cart as described in claim 1, characterized in that, The positioning and communication module includes: A positioning tag is installed on the bottom of the vehicle body and is connected to a positioning base station via a wireless communication protocol; A near-field recognition module is installed at the entrance of the basket.

4. A smart recognition and checkout method based on a shopping cart, characterized in that, The method is applied to an in-vehicle intelligent terminal of a shopping cart as described in any one of claims 1 to 3, wherein the in-vehicle intelligent terminal is connected to a multimodal intelligent recognition module and a positioning communication module via wired connections, the in-vehicle intelligent terminal is connected to a cloud server via a network, and the in-vehicle intelligent terminal is connected to a user terminal via Bluetooth, the edge computing unit includes a deep learning model, and the method includes: Obtain the binding signal sent by the user terminal; Based on the binding signal, user data and shopping cart status information fed back by the positioning communication module and the cloud server are obtained; The system receives a product placement signal sent by the bottom weighing module and obtains product placement information based on the product placement signal, image data collected by the vision sensor, and the deep learning model. A first product list is generated based on the product information provided. The system acquires the checkout signal sent by the user terminal and completes the checkout process based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list.

5. The method as described in claim 4, characterized in that, The step of obtaining the product placement information based on the product placement signal, the image data collected by the visual sensor, and the deep learning model includes: The weight waveform of the placed product is obtained based on the product placement signal; Acquire the image sequence of the placed product fed back by the top vision sensor, the side vision sensor, and the fill light; The weight waveform and the image sequence are matched using a deep learning model to obtain the product information.

6. The method as described in claim 4, characterized in that, The in-vehicle intelligent terminal is connected to the anti-theft alarm module via a wired connection. After the step of generating the first product list based on the added product information, the system further includes: If a product removal signal is received from the bottom weighing module, the product removal information fed back by the vision sensor is obtained; If the retrieved product information is in the first product list, then based on the retrieved product information, the retrieved product location fed back by the positioning communication module and the visual sensor is obtained; and based on the retrieved product location, the anti-theft warning module is controlled to trigger the corresponding warning process.

7. The method as described in claim 6, characterized in that, The step of controlling the anti-theft alarm module to trigger the corresponding alarm process based on the location of the retrieved goods includes: If the location of the retrieved product is within a preset range, then the retrieved product information is deleted from the first product list; If the location of the retrieved goods is not within the preset range, the anti-theft warning module is controlled to trigger a graded alarm process, which includes a first-level alarm and a second-level alarm. The Level 1 alarm includes: Control the touch screen to display a prompt box and trigger a preset voice message; Send a reminder message and a correction window displaying a preset time to the user terminal; If a correction signal is received from the multimodal intelligent recognition module within the preset time, the first-level alarm is cancelled. If an uncorrected signal is received from the multimodal intelligent recognition module within the preset time, a level two alarm will be triggered.

8. The method as described in claim 7, characterized in that, The vehicle-mounted intelligent terminal connects to the staff terminal via Bluetooth, and the secondary alarm includes: The shopping cart number and location are obtained based on the shopping cart status information; Obtain abnormal image fragments fed back by the multimodal intelligent recognition module, and generate abnormal event information based on the shopping cart location, the shopping cart number, and the abnormal image fragments; The abnormal event information is sent to the staff terminal, and the audible and visual alarm is activated.

9. The method according to any one of claims 4 to 8, characterized in that, The steps for completing the checkout based on the user data, the shopping cart status information, the multimodal intelligent recognition module, the checkout signal, and the first product list include: Based on the checkout signal, obtain the shopping cart product information fed back by the multimodal intelligent recognition module; Verification is performed based on the product information in the shopping cart and the first product list; If the verification passes, a payment bill will be generated based on the first product list; Send the payment bill to the user terminal; Based on the payment bill, the user terminal sends a payment signal; A second product list is generated based on the payment signal and the first product list; The user data and shopping cart status information are updated according to the second product list to obtain the updated user data and updated shopping cart status information; Complete the checkout process based on the updated user data and the updated shopping cart status information.

10. The method according to any one of claims 4 to 9, characterized in that, After the step of obtaining user data and shopping cart status information through the positioning communication module and the cloud server, the method further includes: Acquire the search signal from the user terminal or the touch screen, and determine the product coordinates based on the search signal; The shopping cart location is obtained based on the shopping cart status information; A navigation map is generated based on the shopping cart location and the product coordinates.

Images