A visual and gyro sensor fusion intelligent retail snapshot method and device

Abstract

The application discloses a kind of visual and gyro sensor fusion's intelligent retail snapshot method, belongs to intelligent retail technical field.The application includes cabinet door opening and closing state identification, AI snapshot camera every certain time to its external picture perception collection to identify cabinet door opening and closing state;MEMS gyroscope calibration, identify cabinet door is closed, MEMS gyroscope carries out zero point offset calibration;Retail cabinet scene snapshot, when MEMS gyroscope detects that cabinet door angle reaches the angle determined, AI snapshot camera carries out snapshot to the interior scene of retail cabinet.The application fuses visual and gyro sensor, by improving the function of existing snapshot camera, to complete the closed auxiliary detection of cabinet door, to realize the zero point offset calibration of MEMS gyroscope, after cabinet door is opened, angle integration is done again, reliable angle data and scene photo can be obtained, solve the data pollution problem.

Description

Technical Field

[0001] This invention belongs to the field of intelligent retail technology, specifically relating to an intelligent retail capture method and device that integrates visual and gyroscope sensing. Background Technology

[0002] As artificial intelligence is increasingly applied in smart retail, display analysis devices for smart freezers are becoming more widespread. By installing devices on the freezer door handles, when a user opens or closes the freezer door, the smart freezer will actively capture photos of the interior scene when a preset angle is reached, and transmit the photos back to the backend for analysis of the freezer's display details.

[0003] Typically, the method for monitoring the movement angle of a cabinet door involves using a gyroscope sensor to collect data in dimensions such as angular velocity and acceleration. The angle is then obtained by integrating the angular velocity over time. Figure 1 As shown, MEMS gyroscopes have advantages such as small size, light weight, high reliability, and easy system integration, and have a wide range of applications. However, MEMS sensors are prone to zero-point drift due to factors such as power supply and ambient temperature. Even when the gyroscope is in a naturally stationary state, the cumulative error over a long period (greater than 5 minutes) will produce angular errors. For example, if the zero-point drift produces an angular velocity offset of 0.1 dps at a certain moment, then integrating this error over 5 minutes will result in an angle shift of 30 degrees. This leads to uncontrollable results in the angle monitoring of the freezer, causing the camera to fail to capture normal, analyzable photos.

[0004] Currently, such as Figure 2 As shown, smart freezer manufacturers consider adding a door magnetic sensor to the door before shipping to calibrate and zero the zero-point offset data generated by the gyroscope. However, the introduction of the door magnetic sensor not only increases the system's operating cost but also increases the complexity of system integration.

[0005] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0006] The present invention mainly addresses the technical problems existing in the prior art, and provides an intelligent retail capture method and device that integrates vision and gyroscope sensing.

[0007] The above-mentioned technical problems of the present invention are mainly solved by the following technical solution: a smart retail capture method integrating vision and gyroscope sensing, comprising:

[0008] Cabinet door opening and closing status recognition: The AI ​​capture camera periodically collects and perceives external images to identify the cabinet door opening and closing status.

[0009] MEMS gyroscope calibration: when the cabinet door is identified as closed, the MEMS gyroscope performs zero-point offset calibration.

[0010] In retail cabinet scene capture, when the MEMS gyroscope detects that the cabinet door angle has reached a predetermined angle, the AI ​​capture camera captures the scene inside the retail cabinet.

[0011] Preferably, the AI-powered image capture camera perceives and collects external images every minute.

[0012] Preferably, when the MEMS gyroscope detects that the cabinet door angle reaches 45°, the AI ​​capture camera captures the scene inside the retail cabinet.

[0013] Preferably, the moving speed of the cabinet door is no higher than 300° / s, and the shutter speed of the AI ​​capture camera is 20ms.

[0014] Preferably, the resolution of the external image captured by the AI ​​capture camera is 216×384.

[0015] As a preferred method, the sigmoid activation function is used for binary classification of the output layer to identify the open and closed state of the cabinet door.

[0016] This invention also provides an intelligent retail snapshot device that integrates vision and gyroscope sensing, comprising:

[0017] Retail counters are used to display items;

[0018] AI-powered capture cameras are installed on the doors of retail cabinets. They capture images of the interior of the cabinets and periodically collect external images to identify the open and closed status of the doors.

[0019] MEMS gyroscopes are installed on the doors of retail cabinets and connected to AI capture cameras. This includes zero-point offset calibration when the cabinet door is closed and real-time monitoring of the cabinet door angle. When the cabinet door angle reaches a predetermined angle, the AI ​​capture camera is triggered to capture the scene inside the retail cabinet.

[0020] The beneficial effects of this invention are as follows:

[0021] 1. This invention integrates vision and gyroscope sensing. By improving the function of existing snapshot cameras, it completes the auxiliary detection of cabinet door closure, thereby realizing the zero-point offset calibration of MEMS gyroscope. After the cabinet door is opened, angle integration is performed to obtain reliable angle data and scene photos, thus solving the data pollution problem.

[0022] 2. This invention can achieve angle calibration well without adding secondary installation, which reduces the complexity of the device and improves its reliability.

[0023] 3. This invention can not only capture photos with the best angle, but also avoid the pollution of end data caused by invalid display data, resulting in direct economic benefits by avoiding the waste of traffic fees, storage fees and display analysis computing power.

[0024] 4. Compared with existing door magnetic sensors, this invention is economical and efficient, and its implementation in the production of retail cabinets is simple and does not add extra production difficulty, making industrialization easier and more convenient. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a retail cabinet based on a MEMS gyroscope and a snapshot camera in the existing technology;

[0026] Figure 2 This is a schematic diagram of a retail cabinet based on an external door magnetic sensor in the existing technology;

[0027] Figure 3 This is a schematic diagram of the structure of the present invention;

[0028] Figure 4 This is a flowchart illustrating one aspect of the image capture method of the present invention;

[0029] Figure 5 This is a schematic diagram of the AI ​​capture camera of the present invention capturing a cabinet door closing;

[0030] Figure 6 This is a schematic diagram of the AI ​​capture camera of the present invention capturing the opening of a cabinet door;

[0031] Figure 7 This is a flowchart of a binary classification method based on deep learning for image classification according to the present invention;

[0032] Figure 8 This is a curve graph normalized by the Sigmoid method of this invention.

[0033] In the image: 1. Retail cabinet; 2. Snap camera; 3. MEMS gyroscope; 4. Door magnetic sensor; 5. AI snap camera. Detailed Implementation

[0034] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0035] Example: An intelligent retail image capture device that integrates vision and gyroscope sensing, such as... Figure 3 As shown, the device includes a retail cabinet 1, an AI capture camera 5, and a MEMS gyroscope 3. The retail cabinet 1 contains display items, and the AI ​​capture camera 5 and the MEMS gyroscope 3 are both installed on the cabinet door of the retail cabinet 1, such as the cabinet door handle.

[0036] When the door of retail cabinet 1 reaches a predetermined angle, the AI ​​capture camera 5 captures the scene inside retail cabinet 1 and transmits the image back to the backend controller so that the controller can perform array analysis and decision-making on the items displayed in retail cabinet 1.

[0037] Furthermore, the AI ​​capture camera 5 has an AI image detection function to recognize the opening and closing status of the cabinet door. It can collect external images from the AI ​​capture camera 5 at regular intervals to identify the opening and closing status of the cabinet door of the retail cabinet 1 during this period.

[0038] MEMS gyroscope 3 is used to monitor the cabinet door angle in real time. When the cabinet door angle reaches a predetermined angle, it triggers AI capture camera 5 to capture the scene inside retail cabinet 1. At the same time, when the cabinet door is closed, zero-point offset calibration is performed to make the angle detected by MEMS gyroscope 3 relative to 0°.

[0039] This invention also provides an intelligent retail image capture method that integrates vision and gyroscope sensing, such as... Figure 4 As shown, it includes:

[0040] Cabinet door opening and closing status recognition: AI capture camera 5 periodically collects external images to identify the cabinet door opening and closing status.

[0041] MEMS gyroscope 3 calibration: when the cabinet door is identified as closed, MEMS gyroscope 3 performs zero-point offset calibration.

[0042] When the MEMS gyroscope 3 detects that the cabinet door angle has reached a predetermined angle, the AI ​​capture camera 5 captures the scene inside the retail cabinet 1.

[0043] As a preferred embodiment, the AI ​​capture camera 5 acquires external images every minute and uses the sigmoid activation function for binary classification in the output layer to identify the opening and closing status of the cabinet door. If the cabinet door is detected to be closed within this minute, the MEMS gyroscope 3 calibrates the zero-point offset generated during this period; therefore, the aforementioned 0.1 dps zero-point offset error will be cleared during normal door opening, and the angle detection of the MEMS gyroscope 3 will be set to 0° relative to the door.

[0044] Preferably, when the MEMS gyroscope 3 detects that the cabinet door angle reaches 45°, it sends a command to the AI ​​capture camera 5 to capture the scene inside the retail cabinet 1. Typically, the cabinet door's movement speed will not exceed 300° / s, so the capture shutter speed of the AI ​​capture camera 5 can be set to 20ms. Using the median speed, the maximum possible angle error is 3°, which is sufficient for capturing the scene inside the retail cabinet 1. Of course, a shutter speed response time of less than 20ms is even better, allowing for better angle image applications.

[0045] By incorporating AI algorithms into the AI ​​capture camera 5, enabling it to have AI image detection capabilities, and replacing external devices such as the external door magnetic sensor 4, the offset caused by environmental influences of the MEMS gyroscope 3 can be eliminated.

[0046] like Figure 3 As shown, the detection of the cabinet door opening and closing only needs to be completed inside the AI ​​capture camera 5. When the cabinet door is detected to be closed, the MEMS gyroscope 3 completes the offset elimination, thereby ensuring the angle detection of the retail cabinet 1 and capturing a complete image, so as to carry out display analysis and decision-making.

[0047] like Figure 4 As shown, when retail cabinet 1 is closed, even if the MEMS gyroscope 3 experiences zero-point offset due to environmental influences, the zero-point offset calibration can be successfully completed by combining the actual door status. In practical application scenarios, such as... Figure 5 and Figure 6 As shown, the difference between the open and closed cabinet doors is very obvious. Therefore, setting the resolution of the external image of the AI ​​capture camera 5 to 216×384 is sufficient to meet the feature extraction requirements. This not only reduces the computing overhead of the AI ​​capture camera 5's CPU, but also ensures good real-time performance.

[0048] Since there are only two features, namely "cabinet door open" and "cabinet door closed," only binary classification of the features is needed. Therefore, the sigmoid activation function is used in the output layer for binary classification, and the data can be normalized between (0,1). The specific formula is as follows:

[0049]

[0050] In the above formula, x represents the category of the cabinet door being open or closed, S(x) is the probability of the cabinet door being open or closed, and e is the natural constant.

[0051] like Figure 7 As shown, binary classification is a deep learning method based on image classification, and its workflow includes convolution, pooling, and fully connected layers.

[0052] like Figure 8 As shown, the curve is normalized by the Sigmoid method.

[0053] Finally, it should be noted that the above embodiments are merely representative examples of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention should be considered within the protection scope of the present invention.

Claims

1. A smart retail image capture method integrating vision and gyroscope sensing, characterized in that, include: Cabinet door opening and closing status recognition: AI capture camera (5) periodically collects external images to identify the cabinet door opening and closing status. MEMS gyroscope (3) calibration: when the cabinet door is closed, MEMS gyroscope (3) performs zero-point offset calibration. When the MEMS gyroscope (3) detects that the cabinet door angle has reached a predetermined angle, the AI ​​capture camera (5) captures the scene inside the retail cabinet (1).

2. The intelligent retail snapshot method integrating vision and gyroscope sensing according to claim 1, characterized in that, The AI ​​capture camera (5) captures images of its external environment every minute.

3. The intelligent retail image capture method integrating vision and gyroscope sensing according to claim 1, characterized in that, When the MEMS gyroscope (3) detects that the cabinet door angle reaches 45°, the AI ​​capture camera (5) captures the scene inside the retail cabinet (1).

4. The intelligent retail image capture method integrating vision and gyroscope sensing according to claim 1, characterized in that, The cabinet door moves at a speed not exceeding 300° / s, and the AI ​​capture camera (5) has a capture shutter speed of 20ms.

5. The intelligent retail image capture method integrating vision and gyroscope sensing according to claim 1, characterized in that, The resolution of the external image of the AI ​​capture camera (5) is 216×384.

6. The intelligent retail snapshot method integrating vision and gyroscope sensing according to claim 1, characterized in that, The sigmoid activation function is used for binary classification of the output layer to identify the open and closed state of the cabinet door.

7. A smart retail image capture device integrating vision and gyroscope sensing, characterized in that, include: Retail counter (1) for displaying items; AI capture camera (5) is installed on the door of the retail cabinet (1) to capture the scene inside the retail cabinet (1) and to collect external images at regular intervals to identify the opening and closing status of the cabinet door. The MEMS gyroscope (3) is installed on the door of the retail cabinet (1) and connected to the AI ​​capture camera (5). It includes zero-point offset calibration when the cabinet door is closed and real-time monitoring of the cabinet door angle. When the cabinet door angle reaches a predetermined angle, the AI ​​capture camera (5) is triggered to capture the scene inside the retail cabinet (1).

Images