A new retail cloud store intelligent experience monitoring system and method

By using panoramic camera modules and processing center modules in new retail convenience stores, the problem that the existing technology cannot effectively monitor the shopping process is solved, and detailed analysis of the shopping process and shelf layout optimization are achieved, improving the improvement of the shopping process Store operation efficiency and shopping experience.

CN117676074BActive Publication Date: 2024-11-08SHENZHEN BOYUE SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311368085.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-20
Publication Date
2024-11-08
Estimated Expiration
2043-10-20

AI Technical Summary

Technical Problem

The monitoring system of the existing new retail convenience stores is mainly used for anti-theft, and it is impossible to effectively monitor and analyze the shopping process of customers in the store, resulting in the failure to fully optimize the store operation efficiency and customer experience.

Method used

Using panoramic camera module and multiple regional cameras to collect video streaming data, through the processing center's encoding, speed detection, area detection, curve generation and associated storage module, analyze customers' purchasing behavior, related transaction details, and optimize shelf layout and shopping Experience.

Benefits of technology

It realizes detailed monitoring and analysis of the customer's purchasing process, helping backend personnel optimize shelf layout, improve store operating efficiency and customer shopping experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117676074B_ABST
    Figure CN117676074B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of new retail, and in particular to a new retail cloud store intelligent experience monitoring system and method, the system comprising: a panoramic camera module; a plurality of regional cameras; a processing center, including a coding module; a speed detection module, for calculating the displacement speed of each customer; a binding module, for binding the current timestamp as the starting timestamp of the customer target; a regional detection module, for analyzing the panoramic video stream data; a curve generation module, for obtaining the speed change curve from the starting timestamp; a calculation module, for calculating the similarity; an interception module, for intercepting the video stream data as the video stream data to be traced back; an associated storage module, for associating and storing the transaction details of the customer target and the video stream data to be traced back. The present application has the effect of monitoring the customer's shopping process, associating the video stream data with a longer shopping time with the transaction details for subsequent analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of new retail, and in particular to a new retail cloud store intelligent experience monitoring system and method. Background Art

[0002] Currently, new retail convenience stores with 24-hour smart self-service and smart salesperson-free stores are equipped with cameras and connected to the back-end platform. Customer service staff can view the information on the display screen of the back-end platform to supervise the store. However, considering the store area and shelf placement, and the limited shooting angle and range of each camera, stores often need multiple cameras to monitor different areas. However, the monitoring data is only at the anti-theft level, and the customer's shopping process in the store has not been sorted out to make subsequent contributions and improvements to store operations. Summary of the invention

[0003] In order to monitor the customer's shopping process, the video stream data of customers with longer shopping time is associated with the transaction details, so that the back-end personnel can analyze the shopping process and make reasonable adjustments and optimizations in the subsequent shelf sorting, providing customers with a faster shopping experience. This application provides a new retail cloud store intelligent experience monitoring system and method.

[0004] The above-mentioned invention objective of the present application is achieved through the following technical solutions:

[0005] A new retail cloud store intelligent experience monitoring system, including:

[0006] Panoramic camera module, used to collect panoramic video stream data in the store;

[0007] Multiple regional cameras, each of which is used to collect regional video stream data of an associated in-store area. A store has multiple in-store areas.

[0008] Processing center, including:

[0009] The encoding module is used to analyze the panoramic video stream data, mark the customer target and assign the customer a code;

[0010] A speed detection module, used to calculate the displacement speed of each customer based on each consecutive frame of images in the panoramic video stream data;

[0011] A binding module, used for binding the current timestamp as the starting timestamp of a certain customer target if the displacement speed of the customer target is lower than a preset value;

[0012] The area detection module is used to analyze the panoramic video stream data, mark the area in the store where the customer target is located, and record the departure timestamp of the customer target leaving the area in the store;

[0013] A curve generation module, used for obtaining a speed change curve of the customer target within each continuous preset time span from a starting time stamp;

[0014] A calculation module, used for calculating the similarity between the speed change curve and each preset curve in the preset curve set, and when there is a similarity greater than a preset value, taking the timestamp corresponding to the starting point of the speed change curve as the interception timestamp;

[0015] An interception module is used to intercept the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back;

[0016] The associated storage module is used to associate and store the transaction details of the customer target and the video stream data to be traced back.

[0017] In a preferred example, the present application can be further configured as: an area detection module is also used to analyze the panoramic video stream data, and when a customer target enters a certain area in the store, a start signal is sent to the corresponding area camera.

[0018] In a preferred example, the present application can be further configured as follows: it also includes a backend server, which is equipped by backend customer service personnel. If it is detected that a customer target enters another store area from the current store area, the regional video stream data collected by the regional camera associated with the current store area is switched to the regional video stream data collected by the regional camera associated with the other store area and pushed to the backend server in real time for display.

[0019] In a preferred example, the present application can be further configured as follows: the preset curve is pre-collected experimental curve data to characterize whether the customer is lingering in front of the shelf.

[0020] In a preferred example, the present application can be further configured as: an area detection module is also used to analyze the panoramic video stream data, and when there is no customer target in a certain area of ​​the store, a shutdown signal is sent to the corresponding area camera.

[0021] In a preferred example, the present application can be further configured as follows: the preset time span is set to 2 to 10 seconds.

[0022] In a preferred example, the present application can be further configured as follows: the preset time span is set to 3 to 5 seconds.

[0023] The second object of the invention is achieved by the following technical solutions:

[0024] A new retail cloud store intelligent experience monitoring method, comprising:

[0025] Analyze the panoramic video stream data, mark customer targets and assign customer codes;

[0026] The displacement speed of each customer is calculated based on each consecutive frame of images in the panoramic video stream data;

[0027] If the displacement speed of a certain customer target is lower than a preset value, the current timestamp is bound as the starting timestamp of the customer target;

[0028] Analyze the panoramic video stream data, mark the store area where the customer target is located, and record the departure timestamp of the customer target leaving the store area;

[0029] Obtaining a speed change curve of the customer target within each preset time span starting from the start timestamp;

[0030] Calculating the similarity between the speed change curve and each preset curve in the preset curve set, and when there is a similarity greater than a preset value, taking the timestamp corresponding to the starting point of the speed change curve as the interception timestamp;

[0031] Intercept the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back;

[0032] The transaction details of the customer target and the video stream data to be traced back are associated and stored.

[0033] In summary, the present application includes at least one of the following beneficial technical effects:

[0034] By monitoring the customer's shopping process and associating the video stream data of customers' longer shopping time with the transaction details, back-end personnel can analyze the shopping process and obtain the products that take a long time to purchase. In addition, each store area can be differentiated for association, and the shelf merchandise placement in each store area can be analyzed separately. Combined with the store's weekly and monthly purchase data, reasonable adjustments and optimizations can be made in the subsequent shelf sorting, providing customers with a faster shopping experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the connection of various modules of the new retail cloud store intelligent experience monitoring system in one embodiment of the present application;

[0036] Figure 2 It is a flow chart for implementing the new retail cloud store intelligent experience monitoring method in one embodiment of the present application. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure.

[0039] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0040] Figure 1 : is a schematic diagram of the connection of each module of the new retail cloud store intelligent experience monitoring system in one embodiment of the present application, such as Figure 1 As shown, the new retail cloud store intelligent experience monitoring system includes a panoramic camera module, multiple regional cameras and a processing center, wherein the panoramic camera module is installed at the top center of the store, and the panoramic camera module is used to collect panoramic video stream data in the store; multiple regional cameras, multiple cameras are installed in various places in the store, and the multiple regional cameras are respectively used to collect regional video stream data of the associated in-store areas to monitor different in-store areas, that is, the in-store area is associated with a regional camera, and the store is divided into multiple in-store areas. For example, if it is a convenience store, it can be divided into a cash register, a water cabinet channel, a daily necessities channel, a snack channel, a snack end rack, a water cabinet, a cigarette cabinet, etc., depending on the different store configurations of the physical store, and no specific limitation is made here.

[0041] The new retail cloud store intelligent experience monitoring system also includes a backend server, which is equipped by backend customer service personnel. If a customer target is detected to enter another store area from the current store area, the regional video stream data collected by the regional camera associated with the current store area will be switched to the regional video stream data collected by the regional camera associated with another store area and pushed to the backend server for display in real time to realize the unattended function.

[0042] It is understandable that, for the same in-store area, multiple regional cameras may monitor the same in-store area from different angles, and the regional cameras are used to collect regional video stream data of the associated in-store area.

[0043] The processing center includes an encoding module, a speed detection module, a binding module, a region detection module, a curve generation module, a calculation module, a capture module and an associated storage module.

[0044] The encoding module is used to analyze the panoramic video stream data, mark the customer target and assign the customer code. Specifically, the panoramic video stream data can be analyzed and the customer target can be marked through the model. When marking the customer target (human body) in the image, the marked image is analyzed through a machine learning model such as a convolutional neural network to obtain a rectangular frame of the customer target (human body) and assign a corresponding customer code.

[0045] The speed detection module is used to calculate the displacement speed of each customer based on each consecutive frame of images in the panoramic video stream data.

[0046] Specifically, the speed of the object is calculated based on the principle of optical flow method, that is, the relative movement between pixels of multiple consecutive frames. The number of frames can be set to 30 frames, and the displacement speed of each customer can be calculated based on each consecutive number of frames in the panoramic video stream data. The ratio of the target pixel distance to the time span between the first frame and the last frame in the consecutive number of frames is calculated, and then calculated based on a preset mapping relationship with the real-world speed.

[0047] The binding module is used to bind the current timestamp as the customer's pickup timestamp if the customer's displacement speed is lower than a preset value; wherein the preset value can be set to 1m / s. Lower than the preset value indicates that the customer is about to purchase the product.

[0048] The area detection module is used to analyze the panoramic video stream data, mark the area in the store where the customer target is located, and record the departure timestamp of the customer target leaving the area in the store.

[0049] Specifically, each frame of the panoramic video stream data is divided into corresponding store areas to obtain different pixel areas corresponding to the store areas one by one, and then when the customer target leaves the corresponding pixel area, the departure timestamp of the customer target leaving the store area can be recorded.

[0050] The area detection module is also used to analyze the panoramic video stream data. When a customer enters a certain area in the store, a start signal is sent to the corresponding area camera. When there is no customer in a certain area in the store, a shutdown signal is sent to the corresponding area camera to achieve smart energy saving in the store.

[0051] The curve generation module is used to obtain the speed change curve of the customer target within each continuous preset time span starting from the starting time stamp.

[0052] The preset time span may be 2 to 10 seconds, preferably 3 to 5 seconds, and the speed change curve of the customer target within each 3 to 5 second span is obtained from the start time stamp, and the speed change curve is a two-dimensional coordinate curve of speed-time.

[0053] The calculation module is used to calculate the similarity between the speed change curve and each preset curve in the preset curve set. When the similarity is greater than a preset value, the timestamp corresponding to the starting point of the speed change curve is used as the interception timestamp.

[0054] Among them, the similarity can be judged by calculating the mean variance or the Euclidean distance / cosine similarity. The preset value can be set to 80%. When the similarity between the speed change curve and any preset curve in the preset curve set is greater than 80%, the timestamp corresponding to the starting point of the speed change curve is used as the interception timestamp.

[0055] Specifically, the preset curve is pre-collected experimental curve data, that is, the speed change curve of the customer or experimenter actually wandering in front of the shelf looking for goods or selecting goods during the aforementioned preset time span, which can then characterize whether the customer is wandering in front of the shelf looking for goods or selecting goods, etc., to distinguish it from the customer taking a certain item more quickly.

[0056] The interception module is used to intercept the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back.

[0057] In this way, the regional video stream data of a single in-store area can be intercepted for subsequent regional analysis.

[0058] The associated storage module is used to associate and store the transaction details of the customer target, the video stream data to be traced back and the corresponding in-store area.

[0059] Specifically, there is a preset video table of commodity placement problems, which is a pre-set table for querying videos of commodity placement problems. That is, the transaction details of the customer target and the video stream data to be traced back are used as the values ​​in the key-value pair and recorded in advance through the table. Later, only the key in the key-value pair needs to be used to find the transaction details and video stream data to be traced back of the corresponding customer target. In the video table of commodity placement problems, the correspondence between the in-store area and the transaction details and video stream data to be traced back is pre-recorded. In this case, the in-store area is the key in the key-value pair, and the transaction details and video stream data to be traced back are the values ​​in the key-value pair. Therefore, the in-store area can be used as the key to find the corresponding transaction details and video stream data to be traced back.

[0060] In this way, by monitoring the customer's shopping process and associating the video stream data of customers' longer shopping time with the transaction details, it is convenient for back-end personnel to analyze the shopping process and obtain the products with longer shopping process. In addition, each store area can be differentiated for association, and the shelf merchandise placement in each store area can be analyzed separately. Combined with the store's weekly and monthly purchase data, reasonable adjustments and optimizations can be made in the subsequent shelf sorting, providing customers with a faster shopping experience.

[0061] This application also provides a new retail cloud store intelligent experience monitoring method, refer to Figure 2 ,include:

[0062] S1. Analyze the panoramic video stream data, mark the customer target and assign the customer code;

[0063] S2. Calculate the displacement speed of each customer according to each consecutive frame of images in the panoramic video stream data;

[0064] S3. If the displacement speed of a certain customer target is lower than a preset value, the current timestamp is bound as the starting timestamp of the customer target;

[0065] S4, analyzing the panoramic video stream data, marking the store area where the customer target is located, and recording the departure timestamp of the customer target leaving the store area;

[0066] S5, obtaining a speed change curve of the customer target within each preset time span starting from the start timestamp;

[0067] S6, calculating the similarity between the speed change curve and each preset curve in the preset curve set, and when there is a similarity greater than a preset value, taking the timestamp corresponding to the starting point of the speed change curve as the interception timestamp;

[0068] S7, intercepting the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back;

[0069] S8. Associate and store the transaction details of the customer target and the video stream data to be traced back.

[0070] For the specific definition of the new retail cloud store intelligent experience monitoring method, please refer to the definition of the new retail cloud store intelligent experience monitoring system above, which will not be repeated here. Each step of the above new retail cloud store intelligent experience monitoring method can be implemented in whole or in part by software, hardware and their combination.

[0071] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0073] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0074] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0075] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0076] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. New retail cloud store intelligent experience monitoring system, characterized by: include: Panoramic camera module, used to collect panoramic video stream data in the store; Multiple regional cameras, each of which is used to collect regional video stream data of an associated in-store area. A store has multiple in-store areas. Processing center, including: The encoding module is used to analyze the panoramic video stream data, mark the customer target and assign the customer a code; A speed detection module, used to calculate the displacement speed of each customer based on each consecutive frame of images in the panoramic video stream data; A binding module, used for binding the current timestamp as the starting timestamp of a certain customer target if the displacement speed of the customer target is lower than a preset value; The area detection module is used to analyze the panoramic video stream data, mark the area in the store where the customer target is located, and record the departure timestamp of the customer target leaving the area in the store; A curve generation module, used for obtaining a speed change curve of the customer target within each continuous preset time span from a starting time stamp; A calculation module, used for calculating the similarity between the speed change curve and each preset curve in the preset curve set, and when there is a similarity greater than a preset value, taking the timestamp corresponding to the starting point of the speed change curve as the interception timestamp; An interception module is used to intercept the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back; The associated storage module is used to associate and store the transaction details of the customer target and the video stream data to be traced back.

2. The new retail cloud store intelligent experience monitoring system according to claim 1, characterized in that: The area detection module is also used to analyze the panoramic video stream data. When a customer target enters a certain area in the store, a start signal is sent to the corresponding area camera.

3. The new retail cloud store intelligent experience monitoring system according to claim 1, characterized in that: It also includes a backend server, which is equipped for backend customer service personnel. If it is detected that a customer target enters another in-store area from a current in-store area, the regional video stream data collected by the regional camera associated with the current in-store area is switched to the regional video stream data collected by the regional camera associated with the other in-store area and pushed to the backend server in real time for display.

4. The new retail cloud store intelligent experience monitoring system according to claim 1, characterized in that: The preset curve is experimental curve data collected in advance to represent whether the customer is wandering in front of the shelf.

5. The new retail cloud store intelligent experience monitoring system according to claim 1, characterized in that: The area detection module is also used to analyze the panoramic video stream data and send a shutdown signal to the corresponding area camera when there is no customer target in a certain area of ​​the store.

6. The new retail cloud store intelligent experience monitoring system according to claim 1, characterized in that: The preset time span is set to 2 to 10 seconds.

7. The new retail cloud store intelligent experience monitoring system according to claim 6, characterized in that: The preset time span is set to 3 to 5 seconds.

8. A new retail cloud store intelligent experience monitoring method, characterized in that: include: Analyze the panoramic video stream data, mark customer targets and assign customer codes; The displacement speed of each customer is calculated based on each consecutive frame of images in the panoramic video stream data; If the displacement speed of a certain customer target is lower than a preset value, the current timestamp is bound as the starting timestamp of the customer target; Analyze the panoramic video stream data, mark the store area where the customer target is located, and record the departure timestamp of the customer target leaving the store area; Obtaining a speed change curve of the customer target within each preset time span starting from the start timestamp; Calculating the similarity between the speed change curve and each preset curve in the preset curve set, and when there is a similarity greater than a preset value, taking the timestamp corresponding to the starting point of the speed change curve as the interception timestamp; Intercept the video stream data from the interception timestamp to the exit timestamp in the regional video stream data of the regional camera corresponding to the area in the store as the video stream data to be traced back; The transaction details of the customer target and the video stream data to be traced back are associated and stored.