A method, electronic device and storage medium for locating a changed area in a shopping cart

By combining a homography transformation model of depth map and RGB image in a smart shopping cart and using image capture triggered by a weight sensor, the problem of rapid and accurate localization of changing areas in the shopping cart under complex lighting conditions was solved, achieving efficient changing area detection.

CN118691856BActive Publication Date: 2024-11-08NANJING IEEMOO INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411147216.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-08
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

In smart shopping carts, existing methods struggle to quickly and accurately locate changing areas within the cart under complex lighting conditions, especially with limited computing power, where existing methods are affected by shadows and require a large amount of computation.

Method used

By combining the depth map and RGB image of the shopping cart, a well-fitted homography transformation model is obtained. Image capture is performed using the weight sensor trigger condition. Combining the feature information of the depth image and RGB image, the depth image is used to locate the change area. Abnormal states are eliminated by cross-union ratio and similarity calculation, thus achieving accurate location of the change area.

Benefits of technology

Under limited computing power, it achieves rapid and accurate positioning of changing areas within the shopping cart, has strong environmental anti-interference capabilities, low computational load, is simple and efficient, and can eliminate abnormal scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691856B_ABST
    Figure CN118691856B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, electronic device and storage medium for locating a changed area in a shopping cart. According to a depth map and an RGB image of a shopping cart, a fitted homography transformation model is obtained; before the output value of a weight sensor changes, a first depth map of the shopping cart is obtained; after the output value of the weight sensor changes, a second depth map of the shopping cart is obtained, and a second RGB image of the shopping cart is obtained at the same time; according to the first depth map and the second depth map of the shopping cart, a changed area of ​​the depth map is obtained; the changed area of ​​the depth map is mapped to the second RGB image of the shopping cart using a fitted homography transformation model, and a changed area W1 on the second RGB image is obtained. The present invention has a small amount of calculation, a strong environmental anti-interference ability, is almost not affected by ambient light, is simple and efficient, easy to implement, and can effectively exclude some abnormal scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method, electronic equipment and storage medium for locating a changed area in a shopping cart, and belongs to the technical field of image processing. Background Art

[0002] The smart shopping cart is a new type of supermarket shopping trolley. In addition to the function of loading goods like an ordinary shopping cart, it also supports users to scan codes to shop by themselves, and quickly check out and leave the supermarket by themselves.

[0003] Based on its own hardware devices such as front-facing camera, dual cameras, weight sensor, etc., and in conjunction with technical means such as lighting equipment and Bluetooth devices in the store, the smart shopping cart can collect all consumption behavior data of customers from entering the store to leaving the store, including users' shopping tracks, product pick-up and placement records, regional shopping time and other information.

[0004] For smart shopping carts, using the camera on the shopping cart to detect the changed area in the cart is directly related to the accuracy of self-service checkout. Currently, there are usually the following methods to detect the changed area in the shopping cart.

[0005] 1. Method 1 uses differential processing on the RGB images of the previous and next frames to directly calculate the change area between the current frame image and the previous frame image.

[0006] 2. Method 2 uses deep learning based on RGB images to directly detect the items in the shopping cart. Similarly, if it is necessary to locate the changed area of ​​the previous and next frames of the video, first complete the target detection and then calculate the changed area.

[0007] 3. Method 3 uses a calculation method based on optical flow to perform operations on each frame of the image. By analyzing the pixel motion in the image sequence, the motion vector field of each pixel is estimated. By analyzing the motion vector field, the moving objects and their positions in the image are detected, and finally the changed area is located.

[0008] The existing methods have the following defects:

[0009] 1. Method 1 is simple to operate, and the changed area can be obtained by directly processing two pictures. However, this method has great limitations. In the complex lighting environment of the supermarket, heavy shadows are often produced due to spotlights and other light sources. This method cannot effectively filter out such interference.

[0010] 2. Method 2 is based on neural network positioning, which has high accuracy and strong anti-interference ability. However, this method requires a lot of computing power support and cannot be well implemented on smart shopping carts.

[0011] 3. Method 3 calculates moving objects based on optical flow and finally determines the changing area of ​​the shopping cart. This method has slightly stronger anti-interference ability than method 1, but it is still affected by shadows. At the same time, it needs to process the motion vector of each pixel, which also requires considerable computing power.

[0012] Based on the above shortcomings of the existing methods, the present invention aims to solve the problem of how to quickly and accurately locate the changing areas inside the shopping cart during use in the complex lighting environment of the supermarket under the condition of limited computing power. Summary of the invention

[0013] Purpose: In order to overcome the deficiencies in the prior art, the present invention provides a method, electronic device and storage medium for locating a changed area in a shopping cart.

[0014] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0015] In a first aspect, a method for locating a change area in a shopping cart specifically includes:

[0016] Step 1: According to the depth map and RGB image of the shopping cart, obtain the fitted homography transformation model.

[0017] Step 2: Before the output value of the weight sensor changes, obtain a first depth map of the shopping cart; after the output value of the weight sensor changes, obtain a second depth map of the shopping cart and simultaneously obtain a second RGB image of the shopping cart; obtain a changed area of ​​the depth map based on the first depth map and the second depth map of the shopping cart.

[0018] Step 3: Map the changed area of ​​the depth map to the second RGB image of the shopping cart using the fitted homography transformation model to obtain the changed area W1 on the second RGB image.

[0019] As a preferred solution, it also includes:

[0020] Step 4: When the output value of the weight sensor changes again, a third depth map of the shopping cart is obtained, and a third RGB image of the shopping cart is obtained at the same time; a second change area of ​​the depth map is obtained according to the second depth map and the third depth map of the shopping cart.

[0021] Step 5: Map the second changed area of ​​the depth map to the third RGB image using the homography transformation model to obtain the changed area W2 and the changed area W3 on the third RGB image, calculate the intersection and union ratio and similarity between the changed area W1 and the changed area W2, and the changed area W1 and the changed area W3, and obtain the new changed area based on the intersection and union ratio and the similarity.

[0022] As a preferred solution, it also includes:

[0023] Step 6: Position anchor frames for the changed area W1 and the new changed area.

[0024] As a preferred solution, the step 1 specifically includes:

[0025] Step 1.1, obtain the depth map of the shopping cart, normalize the depth map of the shopping cart and convert it into a grayscale image corresponding to the depth map.

[0026] Step 1.2, preprocessing the grayscale image corresponding to the depth image to obtain a preprocessed grayscale image corresponding to the depth image.

[0027] Step 1.3, binarize the grayscale image corresponding to the preprocessed depth image to obtain a binary image corresponding to the depth image.

[0028] Step 1.4, obtain the RGB image of the shopping cart, perform color detection on the RGB image of the shopping cart, and obtain the edge image of the shopping cart frame.

[0029] Step 1.5, preprocessing the edge image of the shopping cart frame to obtain a preprocessed edge image of the shopping cart frame, binarizing the preprocessed edge image of the shopping cart frame to obtain a binary image corresponding to the edge image of the shopping cart frame.

[0030] Step 1.6, expand the resolution of the binary image corresponding to the depth map, and reduce the resolution of the binary image corresponding to the edge image of the shopping cart frame, to obtain a binary image corresponding to the depth map and a binary image corresponding to the edge image of the shopping cart frame with the same resolution.

[0031] Step 1.7, perform corner point detection on the binary image corresponding to the depth map with the same resolution and the binary image corresponding to the edge image of the shopping cart frame, respectively, select the outermost corner points of the depth map and the outermost corner points of the RGB image based on the coordinates of the corner points, substitute the outermost corner points of the depth map and the outermost corner points of the RGB image into the homography transformation model, and obtain a fitted homography transformation model.

[0032] As a preferred solution, the homography transformation model expression is as follows:

[0033]

[0034] in, is the coordinate of the corner point on the RGB image, is the coordinate of the corner point on the depth image, is the scale transformation, is the rotation angle, It is the offset between the x and y axes in the Cartesian coordinate system.

[0035] As a preferred solution, the step 2 specifically includes:

[0036] Step 2.1: Before the output value of the weight sensor changes, obtain a first depth map of the shopping cart.

[0037] Step 2.2: When the output value of the weight sensor changes, obtain a second depth map and a second RGB image of the shopping cart.

[0038] Step 2.3: Perform differential processing on the first depth map of the shopping cart and the second depth map of the shopping cart to obtain a depth difference map.

[0039] Step 2.4: Normalize the depth difference map and convert it into a grayscale image corresponding to the depth difference map.

[0040] Step 2.5: Preprocess the grayscale image corresponding to the depth difference image to obtain a preprocessed grayscale image.

[0041] Step 2.6: Binarize the preprocessed grayscale image to obtain the changed area of ​​the depth image.

[0042] As a preferred solution, the step 5 specifically includes:

[0043] The second changed region of the depth map is mapped to the third RGB image using the homography transformation model to obtain the changed region W2 and the changed region W3 on the third RGB image.

[0044] The DIoU between the changed area W1 and the changed area W2 is calculated as DIoU (W1, W2), and the histogram similarity between the changed area W1 and the changed area W2 is calculated as d (W1, W2).

[0045] The DIoU between the change region W1 and the change region W3 is calculated as DIoU (W1, W3), and the histogram similarity between the change region W1 and the change region W3 is calculated as d (W1, W3).

[0046] If DIoU (W1, W2) is greater than the first threshold, the changed region W2 is deleted.

[0047] If DIoU (W1, W3) is less than the first threshold and d (W1, W3) is greater than the second threshold, the changed area W3 is retained and the original product is output to have moved.

[0048] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the changed area W2 is retained and the original product is output to have moved.

[0049] If DIoU (W1, W3) is greater than the first threshold, the changed region W3 is deleted.

[0050] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the changed area W2 is retained and the original product is output to have moved.

[0051] If DIoU(W1, W3) is less than the first threshold and d(W1, W2) is less than the second threshold, the changed area W3 is retained and the new product is added to the output.

[0052] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is less than the second threshold, the changed area W2 is retained and the new product is added to the output.

[0053] If DIoU (W1, W3) is less than the first threshold and d (W1, W2) is greater than the second threshold, the changed area W3 is retained and the original product is output to have moved.

[0054] In a second aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for locating a changing area in a shopping cart as described in any one of the first aspects.

[0055] According to a third aspect, a computer device includes:

[0056] Memory, used to store instructions.

[0057] The processor is used to execute the instructions so that the computer device performs the operations of the method for locating a changing area in a shopping cart as described in any one of the first aspects.

[0058] Beneficial effects: The present invention provides a method, electronic device and storage medium for locating a changed area in a shopping cart. The weight sensor of the smart shopping cart is used as a trigger condition, and a depth camera and an RGB camera are used to capture the shopping cart before and after the weight changes, respectively, to obtain a depth image and an RGB image, respectively; since the depth image is not affected by visible light, the depth image is used to locate the goods in the shopping cart; the RGB image has rich feature information, and some abnormal states are excluded based on the feature information; the depth image and the RGB image are combined to accurately locate the changed area of ​​the previous and next frames in the shopping cart.

[0059] The present invention has small calculation amount, strong environmental anti-interference ability, is almost unaffected by ambient light, is simple and efficient, easy to implement, and can effectively eliminate some abnormal scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is a flow chart of a method for locating a change area in a shopping cart according to the present invention.

[0061] Figure 2 This is a physical photo diagram of a smart shopping cart.

[0062] Figure 3 It is a schematic diagram of the process of obtaining the homography matrix model of the present invention.

[0063] Figure 4 A photo schematic diagram of a depth map of a shopping cart.

[0064] Figure 5 A photo diagram of an RGB image of a shopping cart.

[0065] Figure 6 is a schematic diagram of corner point detection, where Figure 6 (a) is the depth Figure 2 Schematic diagram of corner point detection of value map, Figure 6 (b) in the figure is RGB Figure 2 Schematic diagram of corner point detection in value map.

[0066] Figure 7 is a schematic diagram of the area of ​​interest of the vehicle body, where Figure 7 (a) is a schematic diagram of the vehicle body area of ​​interest in the depth map. Figure 7 (b) is a schematic diagram of the vehicle body area of ​​interest in the RGB image.

[0067] Figure 8 A schematic diagram of the process flow for obtaining the change area of ​​the present invention.

[0068] Fig. 9 is the depth map before and after the weight change, where Fig. 9 (a) is the depth map before the weight change. Fig. 9 (b) in the figure is the depth map after weight change.

[0069] Fig.10 This is a schematic diagram of the RGB image change area positioning before and after, where: Fig.10 (a) is a schematic diagram before locating the changed area of ​​the RGB image. Fig.10 (b) is a schematic diagram of the RGB image after the change area is located.

[0070] Fig.11 Depth difference map before and after weight change.

[0071] Fig.12 A schematic diagram of an embodiment of the RGB image change area before weight change.

[0072] Fig.13 A schematic diagram of an embodiment of the change area of ​​the RGB image after the weight changes.

[0073] Fig.14 It is a schematic diagram of another embodiment of the RGB image change area before the weight changes.

[0074] Fig.15 It is a schematic diagram of another embodiment of the RGB image change area after the weight changes. DETAILED DESCRIPTION

[0075] The following is a clear and complete description of the technical solutions in the examples of the present invention in conjunction with the accompanying drawings in the examples of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present invention.

[0076] The present invention will be further described below in conjunction with specific embodiments.

[0077] Embodiment 1:

[0078] This embodiment introduces a method for locating a changed area in a shopping cart. The method locates the changed area in a smart shopping cart by using RGB images and depth images taken before and after the weight change. Figure 1 As shown, the following steps are included:

[0079] Step 1: According to the depth map and RGB image of the shopping cart, obtain the fitted homography transformation model.

[0080] Furthermore, the step 1 specifically includes:

[0081] Step 1.1, obtain the depth map of the shopping cart, normalize the depth map of the shopping cart and convert it into a grayscale image corresponding to the depth map.

[0082] Step 1.2, preprocessing the grayscale image corresponding to the depth image to obtain a preprocessed grayscale image corresponding to the depth image.

[0083] Step 1.3, binarize the grayscale image corresponding to the preprocessed depth image to obtain a binary image corresponding to the depth image.

[0084] Step 1.4, obtain the RGB image of the shopping cart, perform color detection on the RGB image of the shopping cart, and obtain the edge image of the shopping cart frame.

[0085] Step 1.5, preprocessing the edge image of the shopping cart frame to obtain a preprocessed edge image of the shopping cart frame, binarizing the preprocessed edge image of the shopping cart frame to obtain a binary image corresponding to the edge image of the shopping cart frame.

[0086] Step 1.6, expand the resolution of the binary image corresponding to the depth map, and reduce the resolution of the binary image corresponding to the edge image of the shopping cart frame, to obtain a binary image corresponding to the depth map and a binary image corresponding to the edge image of the shopping cart frame with the same resolution.

[0087] Step 1.7, perform corner point detection on the binary image corresponding to the depth map with the same resolution and the binary image corresponding to the edge image of the shopping cart frame, respectively, select the outermost corner points of the depth map and the outermost corner points of the RGB image based on the coordinates of the corner points, substitute the outermost corner points of the depth map and the outermost corner points of the RGB image into the homography transformation model, and obtain a fitted homography transformation model.

[0088] Step 2: Before the output value of the weight sensor changes, obtain a first depth map of the shopping cart; after the output value of the weight sensor changes, obtain a second depth map of the shopping cart and simultaneously obtain a second RGB image of the shopping cart; obtain a changed area of ​​the depth map based on the first depth map and the second depth map of the shopping cart.

[0089] Furthermore, the step 2 specifically includes:

[0090] Step 2.1: Before the output value of the weight sensor changes, obtain a first depth map of the shopping cart.

[0091] Step 2.2: When the output value of the weight sensor changes, obtain a second depth map and a second RGB image of the shopping cart.

[0092] Step 2.3: Perform differential processing on the first depth map of the shopping cart and the second depth map of the shopping cart to obtain a depth difference map.

[0093] Step 2.4: Normalize the depth difference map and convert it into a grayscale image corresponding to the depth difference map.

[0094] Step 2.5: Preprocess the grayscale image corresponding to the depth difference image to obtain a preprocessed grayscale image.

[0095] Step 2.6: Binarize the preprocessed grayscale image to obtain the changed area of ​​the depth image.

[0096] Step 3: Map the changed area of ​​the depth map to the second RGB image using the fitted homography transformation model to obtain the changed area W1 on the second RGB image.

[0097] Furthermore, the method also includes: Step 4: when the output value of the weight sensor changes again, obtaining a third depth map of the shopping cart and simultaneously obtaining a third RGB image of the shopping cart; and obtaining a second change area of ​​the depth map according to the second depth map and the third depth map of the shopping cart.

[0098] Step 5: Map the second changed area of ​​the depth map to the third RGB image using the homography transformation model to obtain the changed area W2 and the changed area W3 on the third RGB image, calculate the intersection and union ratio and similarity between the changed area W1 and the changed area W2, and the changed area W1 and the changed area W3, and obtain the new changed area based on the intersection and union ratio and the similarity.

[0099] Furthermore, the step 5 specifically includes:

[0100] The second changed region of the depth map is mapped to the third RGB image using the homography transformation model to obtain the changed region W2 and the changed region W3 on the third RGB image.

[0101] The DIoU between the changed area W1 and the changed area W2 is calculated as DIoU (W1, W2), and the histogram similarity between the changed area W1 and the changed area W2 is calculated as d (W1, W2).

[0102] The DIoU between the change region W1 and the change region W3 is calculated as DIoU (W1, W3), and the histogram similarity between the change region W1 and the change region W3 is calculated as d (W1, W3).

[0103] If DIoU (W1, W2) is greater than the first threshold, the changed region W2 is deleted.

[0104] If DIoU (W1, W3) is less than the first threshold and d (W1, W3) is greater than the second threshold, the changed area W3 is retained and the original product is output to have moved.

[0105] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the changed area W2 is retained and the original product is output to have moved.

[0106] If DIoU (W1, W3) is greater than the first threshold, the changed region W3 is deleted.

[0107] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the changed area W2 is retained and the original product is output to have moved.

[0108] If DIoU(W1, W3) is less than the first threshold and d(W1, W2) is less than the second threshold, the changed area W3 is retained and the new product is added to the output.

[0109] Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is less than the second threshold, the changed area W2 is retained and the new product is added to the output.

[0110] If DIoU (W1, W3) is less than the first threshold and d (W1, W2) is greater than the second threshold, the changed area W3 is retained and the original product is output to have moved.

[0111] Furthermore, the method further includes: step 6: positioning anchor frames of the changed region W1 and the new changed region.

[0112] Embodiment 2:

[0113] This embodiment introduces a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for locating a changed area in a shopping cart as described in any one of Embodiments 1 is implemented.

[0114] Embodiment 3:

[0115] This embodiment introduces a computer device, including:

[0116] Memory, used to store instructions.

[0117] The processor is used to execute the instructions so that the computer device performs the operations of a method for locating a changing area in a shopping cart as described in any one of Embodiment 1.

[0118] Embodiment 4:

[0119] This embodiment introduces a method for locating a changed area in a shopping cart and describes a specific application scenario of a smart shopping cart.

[0120] like Figure 2 As shown, Figure 2 The figure shows a real photo of a smart shopping cart, in which a smart terminal for running the method of the present invention is arranged on the armrest of the frame, and two camera lenses are arranged on the smart terminal, which are respectively used to obtain the depth image and RGB image in the shopping cart. In addition, a weight sensor is arranged at the bottom of the shopping cart, and the weight sensor is used to obtain the weight information of the goods in the shopping cart. The weight sensor communicates with the smart terminal through the wiring in the frame, and the smart terminal also communicates with the host computer through a wireless network.

[0121] like Figure 3 As shown, the method of the present invention needs to obtain a fitted homography transformation model in advance, which specifically includes:

[0122] Get the depth map of the shopping cart, such as Figure 4 As shown in FIG. 1 , the depth map of the shopping cart is normalized and converted into a grayscale image corresponding to the depth map, and the pixel range is 0 to 255.

[0123] The grayscale image corresponding to the depth image is preprocessed. The preprocessing includes: removing noise on the grayscale image by filtering to obtain a relatively clean grayscale image, adjusting the contrast of the grayscale image by image enhancement to enhance the grayscale values ​​of darker areas in the image, and obtaining a grayscale image corresponding to the preprocessed depth image.

[0124] In one embodiment, the image enhancement adopts gamma change, which can improve the overall brightness of the image and highlight the details, wherein the parameter gamma value of the gamma change is set to 1 / 2.

[0125] The grayscale image corresponding to the preprocessed depth image is binarized to obtain a binary image corresponding to the depth image.

[0126] Get the RGB image of the shopping cart, such as Figure 5 As shown, color detection is performed on the RGB image of the shopping cart to obtain the edge image of the shopping cart frame.

[0127] The edge image of the shopping cart frame is preprocessed, and the preprocessing method includes: removing detected abnormal points by image morphology and calculating connected domains to obtain a preprocessed edge image of the shopping cart frame, and binarizing the preprocessed edge image of the shopping cart frame to obtain a binary image corresponding to the edge image of the shopping cart frame.

[0128] The resolution of the binary image corresponding to the depth map is expanded, and the resolution of the binary image corresponding to the edge image of the shopping cart frame is reduced, so as to obtain a binary image corresponding to the depth map and a binary image corresponding to the edge image of the shopping cart frame with the same resolution. In one embodiment, the present invention uses trilinear interpolation to expand or reduce, and the resolution size is uniformly adjusted to 1 / 2 of the resolution size of the RGB original image through the above two operations.

[0129] Corner point detection is performed on the binary image corresponding to the depth map with the same resolution and the binary image corresponding to the edge image of the shopping cart frame, and the outermost corner points of the depth map and the outermost corner points of the RGB image are selected based on the coordinates of the corner points.

[0130] In one embodiment, the present invention uses Harris corner point detection to obtain the corner points of the shopping cart frame of the depth image and the RGB image, and obtains a large number of corner points through the corner point detection algorithm, such as Figure 6 As shown, Figure 6 (a) is the depth Figure 2 The corner points of the value graph, Figure 6 (b) is RGB Figure 2 The corner points of the value map are selected. When selecting the corner points, the four outermost points are selected in combination with the coordinates of each corner point. The region of interest on the vehicle body is determined by combining the final located corner points with the boundaries of the lower left and lower right corners of the image. Figure 7 As shown, Figure 7 (a) shows the region of interest of the vehicle body displayed under the depth map. Figure 7 (b) shows the region of interest of the vehicle body displayed under the RGB image.

[0131] In one embodiment, the corner point detection algorithm of the present invention is a corner point corresponding function R, which is used to determine whether a small area has a corner point. The calculation formula of the corner point corresponding function R is as follows:

[0132]

[0133] Where det(M) is the determinant of the matrix M, trace(M) is the trace of the matrix M (i.e., the sum of the diagonal elements of the matrix), α is a constant with a value of 0.05. When R>threshold, it is judged that there are corner points. Here, according to experiments, the threshold value is set to 7, which is more appropriate.

[0134] Substitute the outermost corner points of the depth map and the outermost corner points of the RGB image into the homography transformation model to obtain a fitted homography transformation model. In one embodiment, the present invention obtains 4 corner points in the depth map and the RGB image respectively, substitutes the coordinates of the 4 corner points into the homography transformation model, obtains the parameters of the homography transformation model, substitutes the parameters back into the homography transformation model, and obtains a fitted homography transformation model. The calculation formula of the homography transformation model is as follows:

[0135]

[0136] in, is the coordinate of the corner point on the RGB image, are the coordinates of the corner points on the depth image, and the above are known variables. is the scale transformation, is the rotation angle, is the offset between the x and y axes in the Cartesian coordinate system. The above parameters are all parameters to be solved.

[0137] like Figure 8 As shown, the present invention uses the depth maps of the two frames before and after the weight change to obtain the changed area of ​​the depth map, and maps the changed area of ​​the depth map to the second RGB image using the fitted homography transformation model to obtain the changed area W1 on the second RGB image. Specifically, it includes:

[0138] like Fig. 9 As shown in (a) in FIG. 8 , before the output value of the weight sensor changes, a first depth map of the shopping cart is obtained.

[0139] like Fig. 9 As shown in (b) in FIG. 1 , when the output value of the weight sensor changes, the second depth map of the shopping cart is obtained, such as Fig.10 As shown in (a) in FIG. 8 , a second RGB image of the shopping cart is obtained.

[0140] like Fig.11 As shown, the first depth map of the shopping cart and the second depth map of the shopping cart are differentially processed to obtain a depth difference map.

[0141] The depth difference map is normalized and converted into a grayscale image corresponding to the depth difference map.

[0142] The grayscale image corresponding to the depth difference image is preprocessed to obtain a preprocessed grayscale image.

[0143] The preprocessing includes: removing the noise on the grayscale image by filtering to obtain a relatively clean grayscale image, adjusting the contrast of the grayscale image by image enhancement to enhance the grayscale values ​​of the darker areas in the image, and obtaining a grayscale image corresponding to the preprocessed depth image.

[0144] The preprocessed grayscale image is binarized to obtain the changed area of ​​the depth image.

[0145] The coordinates of each pixel point in the change area of ​​the depth map are mapped to the second RGB image using the fitted homography transformation model to obtain the corresponding coordinates of each pixel point on the second RGB image, such as Fig.10 As shown in (b), the change area W1 on the second RGB image is obtained according to the corresponding coordinates of each pixel point on the second RGB image, and then the anchor frame is positioned according to the change area W1 on the second RGB image.

[0146] The above processing results of the method of the present invention are for application scenario 1: the newly added goods or items that have not been scanned are located in the RGB image through local calculation, and the image of the newly added goods or items that have not been scanned is obtained. The above algorithm does not use a neural network algorithm, but only processes the pixels in the image, so the calculation amount is small and the calculation speed is fast.

[0147] Furthermore, the method further includes: Step 4: when the output value of the weight sensor changes again, obtaining a third depth map of the shopping cart and simultaneously obtaining a third RGB image of the shopping cart; obtaining a second change area of ​​the depth map according to the second depth map and the third depth map of the shopping cart, specifically including:

[0148] like Fig.12As shown in the figure, the changed area W1 on the second RGB image of the shopping cart obtained by the method in step 2 and step 3. When the output value of the weight sensor changes again, the shopping cart will change in weight output value due to the addition of goods or the movement of goods in the shopping cart. When the weight output value stabilizes again, it is the situation after the output value changes again. At this time, the depth map and RGB image of the shopping cart are obtained as the third depth map of the shopping cart and the third RGB image of the shopping cart.

[0149] Using the second depth map of the shopping cart and the third depth map of the shopping cart, repeat the method of step 2 to obtain the second changed area of ​​the depth map. If the items in the second depth map of the shopping cart and the third depth map of the shopping cart have not increased or decreased, or the positions of the items have not changed, that is, the second depth map and the third depth map are the same, then when the second depth map of the shopping cart and the third depth map of the shopping cart are subjected to differential processing, the second changed area of ​​the depth map will not appear.

[0150] Step 5: Map the second changed area of ​​the depth map to the third RGB image using the homography transformation model to obtain the changed area W2 and the changed area W3 on the third RGB image, calculate the intersection and union ratio and similarity between the changed area W1 and the changed area W2, and between the changed area W1 and the changed area W3, and obtain the new changed area based on the intersection and union ratio and the similarity. Specifically including:

[0151] The second changed region of the depth map is mapped to the third RGB image using the homography transformation model to obtain the changed region W2 and the changed region W3 on the third RGB image.

[0152] Calculate the DIoU of the change area W1 and the change area W2 as DIoU (W1, W2). When DIoU (W1, W2) is greater than the threshold, it means that the change area W1 completely overlaps with the change area W2, indicating that the product has changed from existence to non-existence. Output that there is no product or item in the change area W2, and delete the change area W2.

[0153] The DIoU of the change area W1 and the change area W3 is calculated as DIoU (W1, W3). When DIoU (W1, W3) is less than the threshold, it means that the change area W1 and the change area W3 do not overlap. Then the histogram similarity of the change area W1 and the change area W3 is calculated as d (W1, W3). When d (W1, W3) is greater than the threshold, it means that the feature values ​​of the products in the change area W1 and the change area W2 are the same. The product in the output change area W3 is the original product, the change area W3 is retained and the anchor frame is positioned.

[0154] The above processing results of the method of the present invention are for application scenario 2: the goods in the smart shopping cart are moved in the shopping cart, resulting in a change in the weight value of the weight sensor. The changed area W2 and the changed area W3 on the third RGB image after the movement are as follows: Fig.13 As shown, the changed area W2 is the location of the original product, which completely overlaps with the changed area W1, so it can be determined that there is no product here, and the changed area W2 is deleted.

[0155] The change area W3 does not overlap with the change area W1, but the histogram similarity between the change area W3 and the change area W1 is the same, indicating that the goods in the change area W3 are the same as those in the change area W1. Therefore, it can be concluded that the goods in the change area W1 have moved, and the area after the move is the change area W3, and the change area W3 is re-anchored.

[0156] Furthermore, the method further includes: Step 4: when the output value of the weight sensor changes again, obtaining a third depth map of the shopping cart and simultaneously obtaining a third RGB image of the shopping cart; obtaining a second change area of ​​the depth map according to the second depth map and the third depth map of the shopping cart, specifically including:

[0157] like Fig.14 As shown in the figure, the changed area W1 on the second RGB image of the shopping cart obtained by the method in step 2 and step 3. When the output value of the weight sensor changes again, the shopping cart will change in weight output value due to the addition of goods or the movement of goods in the shopping cart. When the weight output value stabilizes again, it is the situation after the output value changes again. At this time, the depth map and RGB image of the shopping cart are obtained as the third depth map of the shopping cart and the third RGB image of the shopping cart.

[0158] Using the second depth map of the shopping cart and the third depth map of the shopping cart, repeat the method of step 2 to obtain the second changed area of ​​the depth map. If the items in the second depth map of the shopping cart and the third depth map of the shopping cart have not increased or decreased, or the positions of the items have not changed, that is, the second depth map and the third depth map are the same, then when the second depth map of the shopping cart and the third depth map of the shopping cart are subjected to differential processing, the second changed area of ​​the depth map will not appear.

[0159] Step 5: Map the second changed area of ​​the depth map to the third RGB image using the homography transformation model to obtain the changed area W2 and the changed area W3 on the third RGB image, calculate the intersection and union ratio and similarity between the changed area W1 and the changed area W2, and between the changed area W1 and the changed area W3, and obtain the new changed area based on the intersection and union ratio and the similarity. Specifically including:

[0160] The DIoU of the change area W1 and the change area W2 is calculated as DIoU (W1, W2). When DIoU (W1, W2) is less than the threshold, it means that the change area W1 and the change area W2 do not overlap. Then the histogram similarity of the change area W1 and the change area W2 is calculated as d (W1, W3). When d (W1, W3) is less than the threshold, it means that the feature values ​​of the products in the change area W1 and the change area W2 are different. The products in the change area W2 are output as new products, and the change area W2 is retained and the anchor frame is positioned.

[0161] The DIoU of the change area W1 and the change area W3 is calculated as DIoU (W1, W3). When DIoU (W1, W3) is less than the threshold, it means that the change area W1 and the change area W3 do not overlap. Then the histogram similarity of the change area W1 and the change area W3 is calculated as d (W1, W3). When d (W1, W3) is greater than the threshold, it means that the feature values ​​of the products in the change area W1 and the change area W2 are the same. The product in the output change area W3 is the original product, the change area W3 is retained and the anchor frame is positioned.

[0162] The above processing results of the method of the present invention are for application scenario 3: the goods in the smart shopping cart are moved in the shopping cart, and new goods are added to the smart shopping cart, resulting in a change in the weight value of the weight sensor. The changed area W2 and the changed area W3 on the third RGB image after the movement and the addition of the new goods are as follows: Fig.15 As shown, the changed area W2 is the location of the newly added product, which does not overlap with the changed area W1 at all, and the histogram similarity is also different. Therefore, it can be judged that there is a new product here, and the changed area W2 is anchored.

[0163] The change area W3 does not overlap with the change area W1, but the histogram similarity between the change area W3 and the change area W1 is the same, indicating that the goods in the change area W3 are the same as those in the change area W1. Therefore, it can be concluded that the goods in the change area W1 have moved, and the area after the move is the change area W3, and the change area W3 is re-anchored.

[0164] Furthermore, when two change areas appear in the difference map, DIoU is used here to calculate the distance between the two target areas, and the overlap, distance, and scale of the two change areas are considered at the same time, DIoU (Distance-IoU), IoU (Intersection over Union). The DIoU calculation formula is as follows:

[0165]

[0166]

[0167] Among them, α is the weight function, v is used to measure the similarity of aspect ratio, A is the original change area, and B is the new change area. The calculation formula is:

[0168]

[0169]

[0170] The relative position of the changed area can be determined by the above formula to determine whether the changed area has shifted.

[0171] At the same time, in order to determine whether the changed area is a newly purchased product, it is necessary to perform a similarity comparison on the RGB images of the changed area. Based on limited hardware computing power, the histogram similarity of the image is used as a calculation method to measure whether two images are similar. The specific calculation formula is as follows:

[0172]

[0173] In the above formula, H1 and H2 are histograms, and Img1 and Img2 are change area images.

[0174] The histogram similarity uses the Buffett coefficient to measure the correlation of the histogram. The relevant formula is as follows:

[0175]

[0176] The above formula is the definition of Bhattacharyya distance, where p and q are two discrete probability distributions, 0≤BC≤1 and 0≤DB≤∞. The specific calculation of histogram similarity is as follows:

[0177]

[0178] Histogram Bhattacharyya distance value [0,1], its value is 1 if it matches perfectly, and 0 if it does not match perfectly. In the present invention, DIoU is related to Bhattacharyya distance 0.5 is used as the threshold.

[0179] After obtaining the changed area through the method of the present invention, from the business level, if the user does not scan the code and puts it directly, or puts in personal items, the area can be located to show the user a picture and interact with the user, reminding the user to scan the code or asking the user whether it is a personal item. From the algorithm level, the above information can be uploaded after the changed area is located, providing a basis for the host computer to perform subsequent identification and verification of the goods in the changed area.

[0180] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for locating a change area in a shopping cart, characterized in that: Specifically include: Step 1: Obtain the fitted homography transformation model based on the depth map and RGB image of the shopping cart; Step 2: before the output value of the weight sensor changes, obtain a first depth map of the shopping cart; after the output value of the weight sensor changes, obtain a second depth map of the shopping cart and simultaneously obtain a second RGB image of the shopping cart; obtain a change area of ​​the depth map according to the first depth map and the second depth map of the shopping cart; Step 3: Map the changed area of ​​the depth map to the second RGB image of the shopping cart using the fitted homography transformation model to obtain the changed area W1 on the second RGB image; The step 1 specifically includes: Step 1.1, obtain the depth map of the shopping cart, normalize the depth map of the shopping cart and convert it into a grayscale image corresponding to the depth map; Step 1.2, preprocessing the grayscale image corresponding to the depth image to obtain a grayscale image corresponding to the preprocessed depth image; Step 1.3, binarizing the grayscale image corresponding to the preprocessed depth image to obtain a binary image corresponding to the depth image; Step 1.4, obtain the RGB image of the shopping cart, perform color detection on the RGB image of the shopping cart, and obtain the edge image of the shopping cart frame; Step 1.5, preprocessing the edge image of the shopping cart frame to obtain a preprocessed edge image of the shopping cart frame, and binarizing the preprocessed edge image of the shopping cart frame to obtain a binary image corresponding to the edge image of the shopping cart frame; Step 1.6, expanding the resolution of the binary image corresponding to the depth image, and reducing the resolution of the binary image corresponding to the edge image of the shopping cart frame, to obtain a binary image corresponding to the depth image and a binary image corresponding to the edge image of the shopping cart frame with the same resolution; Step 1.7, perform corner point detection on the binary image corresponding to the depth map with the same resolution and the binary image corresponding to the edge image of the shopping cart frame, respectively, select the outermost corner points of the depth map and the outermost corner points of the RGB image based on the coordinates of the corner points, substitute the outermost corner points of the depth map and the outermost corner points of the RGB image into the homography transformation model, and obtain a fitted homography transformation model.

2. A method for locating a change area in a shopping cart according to claim 1, characterized in that: Also includes, Step 4: When the output value of the weight sensor changes again, a third depth map of the shopping cart is obtained, and a third RGB image of the shopping cart is obtained at the same time; a second change area of ​​the depth map is obtained according to the second depth map and the third depth map of the shopping cart; Step 5: Map the second changed area of ​​the depth map to the third RGB image using the homography transformation model to obtain the changed area W2 and the changed area W3 on the third RGB image, calculate the intersection and union ratio and similarity between the changed area W1 and the changed area W2, and the changed area W1 and the changed area W3, and obtain the new changed area based on the intersection and union ratio and the similarity.

3. A method for locating a change area in a shopping cart according to claim 2, characterized in that: The method further includes: step 6: positioning anchor frames of the changed area W1 and the new changed area.

4. A method for locating a change area in a shopping cart according to claim 1, characterized in that: The homography transformation model expression is as follows: Among them, (x ′ ,y ′ ) is the coordinate of the corner point on the RGB image, (x, y) is the coordinate of the corner point on the depth image, s is the scale transformation, θ is the rotation angle, t x ,t y It is the offset between the x and y axes in the Cartesian coordinate system.

5. A method for locating a change area in a shopping cart according to any one of claims 1 to 3, characterized in that: The step 2 specifically includes: Step 2.1: Before the output value of the weight sensor changes, obtain a first depth map of the shopping cart; Step 2.2: When the output value of the weight sensor changes, obtain a second depth map and a second RGB image of the shopping cart; Step 2.3: Perform differential processing on the first depth map of the shopping cart and the second depth map of the shopping cart to obtain a depth difference map; Step 2.4: Normalize the depth difference map and convert it into a grayscale image corresponding to the depth difference map; Step 2.5: pre-process the grayscale image corresponding to the depth difference image to obtain a pre-processed grayscale image; Step 2.6: binarize the pre-processed grayscale image to obtain a changed area of ​​the depth image.

6. A method for locating a change area in a shopping cart according to any one of claims 2 to 3, characterized in that: The step 5 specifically includes: Mapping the second changed region of the depth map to the third RGB image using a homography transformation model to obtain a changed region W2 and a changed region W3 on the third RGB image; Calculate the DIoU between the change area W1 and the change area W2 as DIoU(W1, W2), and calculate the histogram similarity between the change area W1 and the change area W2 as d(W1, W2); Calculate the DIoU between the change area W1 and the change area W3 as DIoU(W1, W3), and calculate the histogram similarity between the change area W1 and the change area W3 as d(W1, W3); If DIoU(W1, W2) is greater than the first threshold, delete the changed area W2; If DIoU(W1, W3) is less than the first threshold and d(W1, W3) is greater than the second threshold, the change area W3 is retained and the original product is output to have moved; Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the change area W2 is retained and the original product is output to have moved; If DIoU(W1, W3) is greater than the first threshold, delete the changed area W3; Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is greater than the second threshold, the change area W2 is retained and the original product is output to have moved; If DIoU(W1, W3) is less than the first threshold, and d(W1, W2) is less than the second threshold, keep the changed area W3 and output the added new product; Alternatively, if DIoU(W1, W2) is less than the first threshold and d(W1, W2) is less than the second threshold, the changed area W2 is retained and the new product is added to the output; If DIoU(W1, W3) is less than the first threshold and d(W1, W2) is greater than the second threshold, the change area W3 is retained and the original product is output to be moved.

7. A method for locating a change area in a shopping cart according to claim 6, characterized in that: The first threshold is set to 0.5, and the second threshold is set to 0.

5.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for locating a changing area in a shopping cart as claimed in any one of claims 1 to 7 is implemented.

9. A computer device, characterized in that: include: A memory for storing instructions; The processor is configured to execute the instructions so that the computer device performs the operations of the method for locating a changing area in a shopping cart as claimed in any one of claims 1 to 7.