A product identification method, apparatus, electronic device, and readable storage medium

Abstract

This application discloses a product identification method, apparatus, electronic device, and readable storage medium. The method includes: acquiring an RGB image of a shelf captured by a depth camera, wherein the shelf contains at least one product; extracting features from the RGB image captured by the depth camera to obtain a first region, the first region containing only one product; acquiring the physical height of the first region based on depth information collected by the depth camera; determining the specifications of the product in the first region based on the physical height; and obtaining a product name with specifications based on the product's specifications. This application determines the product specifications by physical height. Since different product specifications correspond to different physical heights, using a highly accurate physical height can better determine the product specifications. Depth information includes distance information for each pixel, and calculating the physical height based on depth information can effectively improve the accuracy of the physical height calculation. Finally, the product name with specifications is obtained, effectively improving the accuracy of product identification.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a product recognition method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] When conducting product statistics, statisticians identify and count the products in the photos. However, for some products, the appearance is similar or even identical, with only differences in specifications. The differences between products occupy a very small area in the overall product photo, making it difficult to identify the subtle differences in specifications visually alone, resulting in low accuracy in product identification. Summary of the Invention

[0003] In view of this, this application provides a product identification method, apparatus, electronic device, and readable storage medium to solve the problem of low product identification accuracy due to specifications.

[0004] To achieve the above objectives, the following solution is proposed:

[0005] A product identification method, the method comprising:

[0006] Acquire an RGB image of a shelf, taken by a depth camera, on which at least one item is displayed;

[0007] After feature extraction from the RGB image captured by the depth camera, a first region is obtained, which contains only one product.

[0008] Based on the depth information collected by the depth camera, the physical height of the first region is obtained;

[0009] The specifications of the goods in the first area are determined based on the physical height;

[0010] The product name containing the product specifications is obtained by using the product specifications.

[0011] Optionally, obtaining the physical height of the first region based on the depth information collected by the depth camera includes:

[0012] The first distance from the actual product area to the depth camera is obtained based on the correspondence between the RGB image captured by the depth camera and the depth information;

[0013] The physical height of the first region is calculated based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance.

[0014] Optionally, the focal length includes the focal length f in the x-axis direction. x and the focal length f in the y-axis direction y The step of calculating the physical height of the first region based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance includes:

[0015] Obtain the f x and the f y The mean f;

[0016] Obtain the coordinates of the top center point (x) of the first region based on the coordinates of the first region. m y1) and the coordinates of the bottom center point (x m ,y2);

[0017] According to the formula:

[0018] w xt =(x m -c x )×c z ÷f

[0019] w yt =(y1-c y )×c z ÷f

[0020] w xb =(x m -c x )×c z ÷f

[0021] w yb =(y2-c y )×c z ÷f

[0022] Calculate the physical world coordinates (w) of the top center point of the first region. xt w yt ) and the physical world coordinates of the bottom center point (w xb w yb );

[0023] According to the (w) xt w yt ) and the (w xb w yb Calculate the physical height of the first region.

[0024] Optionally, it also includes: obtaining the category of the goods in the first region;

[0025] Determining the specifications of goods in the first region based on the physical height includes:

[0026] Determine at least one specification corresponding to the category of goods in the first region, wherein the specification corresponds to the height of the goods;

[0027] The physical height is compared with the product height corresponding to the at least one specification, and the specification of the product in the first region is determined based on the comparison result.

[0028] Optional, also includes:

[0029] A first image is acquired using a camera other than the depth camera, wherein the resolution of the first image is higher than the resolution of the RGB image captured by the depth camera;

[0030] After feature extraction from the first image, a second region is obtained, which contains only one product.

[0031] Identify the category of goods in the second region;

[0032] Determine whether the first region and the second region are regions containing the same product. If so, establish a correspondence between the first region and the second region, and determine the category of the product in the second region as the category of the product in the first region.

[0033] Determining the specifications of goods in the first region based on the physical height includes:

[0034] The specifications of the goods in the first area are determined based on the physical height and the category of the goods in the first area.

[0035] Optionally, determining whether the first area and the second area are areas containing the same product includes:

[0036] Extract feature points from the first image and feature points from the RGB image captured by the depth camera;

[0037] The feature points of the first image are matched with the feature points of the RGB image captured by the depth camera;

[0038] After a successful match, the second region contained in the first image is mapped onto the RGB image captured by the depth camera;

[0039] If the overlap rate between the first region in the RGB image and the mapped second region meets a preset requirement, then the first region and the second region are determined to be regions containing the same product.

[0040] Optionally, it also includes: identifying the products in the first region of the RGB image captured by the depth camera to obtain the category of the products in the first region;

[0041] Determining the specifications of goods in the first region based on the physical height includes:

[0042] The specifications of the goods in the first area are determined based on the physical height and the category of the goods in the first area.

[0043] A product identification device, the device comprising:

[0044] An image acquisition unit is used to acquire an RGB image of a shelf taken by a depth camera, the shelf having at least one item;

[0045] The feature extraction unit is used to extract features from the RGB image captured by the depth camera to obtain a first region, wherein the first region contains only one product.

[0046] The height acquisition unit is used to acquire the physical height of the first area based on the depth information collected by the depth camera;

[0047] A specification determination unit is used to determine the specifications of the goods in the first area based on the physical height;

[0048] A name determination unit is used to obtain a product name containing the specifications of the product.

[0049] An electronic device, comprising a memory and a processor;

[0050] The memory is used to store programs;

[0051] The processor is used to execute the program to implement each step of any of the above-mentioned product identification methods.

[0052] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described product identification methods.

[0053] This application discloses a product identification method, apparatus, electronic device, and readable storage medium. The method includes: capturing an RGB image of a shelf using a depth camera; extracting features from the RGB image to obtain a first region containing only one product; obtaining the physical height of the first region and determining the product's specifications based on the physical height; and finally determining the product name based on the specifications. Since products of different specifications have different physical heights, this application can determine the product's specifications using the physical height. Depth information contains distance information for each pixel; calculating the physical height based on depth information can effectively improve the accuracy of the physical height calculation. Therefore, based on a highly accurate physical height, the product's specifications can be determined more accurately, resulting in a product name with specifications. Thus, this method can effectively improve the accuracy of product identification. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a product identification method provided in this application embodiment;

[0056] Figure 2 A flowchart illustrating another product identification method provided in this application embodiment;

[0057] Figure 3 A flowchart illustrating yet another product identification method provided in this application embodiment;

[0058] Figure 4 A flowchart illustrating yet another product identification method provided in this application embodiment;

[0059] Figure 5 This is a schematic diagram of the structure of a commodity identification device provided in an embodiment of this application;

[0060] Figure 6 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] like Figure 1 As shown in the figure, this application provides a product identification method, which includes:

[0063] S10. Obtain an RGB image of the shelf taken by a depth camera, wherein the shelf has at least one item;

[0064] S11. After extracting features from the RGB image captured by the depth camera, a first region is obtained, which contains only one product.

[0065] S12. Based on the depth information collected by the depth camera, obtain the physical height of the first region;

[0066] S13. Determine the specifications of the goods in the first area based on their physical height;

[0067] S14. Obtain the product name with specifications by using the product's specifications.

[0068] In this embodiment, a patrol robot or a person can photograph shelves containing at least one product. For example, a patrol robot equipped with a depth camera can move around the warehouse carrying the depth camera to photograph different shelves. In addition to acquiring a planar image of the object, the depth camera can also obtain depth information. There is a correspondence between the planar image and the depth information; the depth information includes the distance information of each pixel in the planar image, which is the distance from the actual product area corresponding to each pixel in the planar image to the depth camera. Since the image contains other redundant image information besides the product image information, and this redundant image information is not conducive to product recognition, this embodiment can use a feature extractor to extract features from the RGB image captured by the depth camera to separate the product image information into a separate part. Specifically, the feature extractor can be a pre-trained product detection feature extractor. After feature extraction from the RGB image captured by the depth camera, at least one first region can be divided for the products contained in the RGB image captured by the depth camera, and each first region contains only one product. The first region can be a rectangular region, whose four sides can connect to the image boundary of the product, making the border of the rectangular region the bounding rectangle of the product image. Optionally, the border of the first region can be a product detection box. For some products, different specifications of products have different physical heights, so this application can determine the specifications of the product in the first region by calculating the physical height of the first region. This embodiment can calculate the physical height of the first region using the distance information corresponding to a pixel in the first region and the optical feature parameters of the depth camera. The aforementioned pixel can be the pixel corresponding to the center of the first region, and the optical feature parameters of the depth camera can include the focal length and the imaging origin. In reality, the physical height of some products placed on the shelf is not easily affected by external forces, such as bottled products. Of course, the physical height of some products placed on the shelf is easily affected by external forces, such as inflatable packaged products. For products whose physical height is not easily affected by external forces (e.g., the physical height of a 500ml bottle of mineral water is 22cm, and the physical height of a 550ml bottle of mineral water is 22.7cm), this embodiment can compare the calculated physical height of the first region with the physical heights corresponding to different product specifications, and determine the product specifications in the first region based on the comparison results. For products whose physical height is easily affected by external forces, the physical height corresponding to different product specifications is a range; for example, the physical height range of a medium-sized bag of potato chips is 15cm to 19cm, and the physical height range of a large bag of potato chips is 22cm to 25cm.This embodiment can determine whether the calculated physical height of the first region falls within the physical height range corresponding to a certain specification. If so, the specification can be determined as the specification of the product in the first region.

[0069] This application provides a product identification method, which may include: using a depth camera to photograph a shelf, obtaining an RGB image of the shelf captured by the depth camera; extracting features from the RGB image to obtain a first region, which contains only one product; obtaining the physical height of the first region and determining the product's specifications based on the physical height; and finally determining the product name based on the specifications. Since products of different specifications have different physical heights, this application can determine the product's specifications using the physical height. Depth information contains distance information for each pixel; calculating the physical height based on depth information can effectively improve the accuracy of the physical height calculation. Therefore, based on a highly accurate physical height, the product's specifications can be determined more accurately, resulting in a product name with specifications. Thus, this method can effectively improve the accuracy of product identification.

[0070] like Figure 2 As shown, in another product identification method provided according to an embodiment of this application, Figure 1 Step S12 shown may include:

[0071] S15. Based on the correspondence between the RGB image captured by the depth camera and the depth information, obtain the first distance from the actual product area corresponding to the first pixel in the first region to the depth camera.

[0072] S16. Calculate the physical height of the first region based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance.

[0073] In this embodiment, the depth camera acquires distance information via structured light. Specifically, a controllable light spot, light strip, or light surface structure is projected through the structured light projector in the depth camera. When obstructed, the light returns. Based on the information carried by the returning structured light, the distance between the obstruction point or the region corresponding to all points on an obstruction surface and the depth camera can be calculated, generating a depth information file. In this embodiment, the depth information file can be a matrix, the same size as the RGB image captured by the depth camera. Each value in the matrix corresponds to the distance information of a pixel in the RGB image captured by the depth camera. A first region can include many pixels. The first pixel can be the center point of the first region, the top midpoint of the first region, or the bottom midpoint of the first region. The coordinates of the first region, the focal length of the depth camera, and the coordinates of the imaging origin can be directly obtained when calculating the physical height. The coordinates of the first region can be represented by the coordinates of a set of diagonal vertices, such as the coordinates of the top left vertex (x1, y1) and the bottom right vertex (x2, y2).

[0074] In another product identification method provided according to an embodiment of this application, the focal length includes the focal length f in the x-axis direction. x and the focal length f in the y-axis direction y , Figure 2 Step S16 shown may include:

[0075] Get f x and f y The mean f;

[0076] Obtain the coordinates of the top center point (x) of the first region based on the coordinates of the first region. m y1) and the coordinates of the bottom center point (x m ,y2);

[0077] According to the formula:

[0078] w xt =(x m -c x )×c z ÷f

[0079] w yt =(y1-c y )×c z ÷f

[0080] w xb =(x m -c x )×c z ÷f

[0081] w yb =(y2-c y )×cz ÷f

[0082] Calculate the physical world coordinates (w) of the top center point of the first region. xt w yt ) and the physical world coordinates of the bottom center point (w xb w yb );

[0083] According to (w) xt w yt ) and (w xb w yb Calculate the physical height of the first region.

[0084] Among them, c z That is, the first distance, (c x c y () represents the coordinates of the origin of the depth camera image. This embodiment can be expressed using the formula:

[0085]

[0086] Calculate f x and f y The mean value f. In this embodiment, it can be based on the coordinates (x1, y1, x2, y2) of the first region and the formula: Get the coordinates of the top center point of the first region (x m y1) and the coordinates of the bottom center point (x m ,y2).

[0087] Since cameras can be placed anywhere in the environment, a reference coordinate system can be chosen in the environment to describe the position of the camera and the position of any object in the environment. This coordinate system is called the physical world coordinate system.

[0088] This embodiment can be achieved through the formula:

[0089]

[0090] The physical height of the first region is calculated. In this embodiment, the distance c between the actual product area corresponding to the center point of the first region and the depth camera is used. z When calculating physical world coordinates, w xt and w xb Since the values ​​are the same, this embodiment can calculate the physical height using only w. yt and w yb Calculate the physical height of the first region.

[0091] In another embodiment, this application can also obtain the distance c between the actual product area corresponding to the top center point of the first region and the depth camera.zt And obtain the distance c between the actual product area corresponding to the bottom center point of the first area and the depth camera. zb Then based on the formula

[0092] w xt =(x m -c x )×c zt ÷f

[0093] w yt =(y1-c y )×c zt ÷f

[0094] w xb =(x m -c x )×c zb ÷f

[0095] w yb =(y2-c y )×c zb ÷f

[0096] Calculate the physical world coordinates (w) of the top center point of the first region. xt w yt ) and the physical world coordinates of the bottom center point (w xb w yb At this point, due to c zt and c zb The values ​​may differ, therefore the calculated value of w at this time will vary. xt and w xb The value may be different. In this case, this embodiment can be expressed by the formula:

[0097]

[0098] Calculate the physical height of the first region.

[0099] like Figure 3 As shown, in another product identification method provided according to an embodiment of this application, the method may further include step S17;

[0100] S17. Obtain the category of the goods in the first area;

[0101] Figure 1 Step S13 shown may specifically include:

[0102] S18. Determine at least one specification corresponding to the category of goods in the first area, and the specification corresponds to the height of the goods;

[0103] S19. Compare the physical height with the product height corresponding to at least one specification, and determine the specification of the product in the first region based on the comparison result.

[0104] The product category can be a product type, which can be determined based on product attributes. For example, product types can include: drinking water, potato chips, toilet paper, and beer. In another embodiment, the product category can include both the product brand and the product type. For example, drinking water of brand A and potato chips of brand B. For certain product types, products of the same specification from different brands may have the same size. A product can include at least one specification. The product category reveals the various specifications a product may have. For example, a bottle of mineral water may have specifications of 300 ml, 450 ml, and 500 ml, and a bag of potato chips may have specifications of 12g, 40g, 70g, 135g, and 300g. Therefore, in determining product specifications, the product category can be determined first to know the specific specifications. In this embodiment, the correspondence between product categories and specific specifications can be obtained from a pre-defined product specification table. Different specifications of goods may correspond to different physical heights. For example, a 500ml bottle of cola is 35cm high, while a 300ml bottle is 15cm high. In this embodiment, the specifications of a product can be determined by its physical height. After determining the category of the goods in the first area, the specific specifications corresponding to the product category can be determined. The physical height calculated based on depth information is compared with the physical heights of goods corresponding to different specifications. Based on the comparison results, the specifications of the goods in the first area can be determined.

[0105] The above comparison process may include:

[0106] The calculated physical height is subtracted from the physical height corresponding to different specifications to obtain at least one difference value;

[0107] If the difference between the calculated physical height and one of the physical heights is the minimum of at least one of the above differences, then the specification corresponding to that physical height is determined as the specification of the product in the first region.

[0108] For example, if the product category in the first area is determined to be Xiaoming mineral water, and the product specification table shows that Xiaoming mineral water has two corresponding sizes: 330 ml and 350 ml, the physical height of the 330 ml product is 16 cm, and the physical height of the 350 ml product is 18.2 cm, then based on the depth information, the physical height of the first area is calculated to be 16.5 cm. Comparing this physical height with the physical heights of the two sizes in the product specification table, the difference between this physical height and the physical height of the 330 ml product is less than the difference between this physical height and the physical height of the 350 ml product. Therefore, it can be considered that the size of Xiaoming mineral water in the first area is 330 ml.

[0109] In another product identification method provided according to an embodiment of this application, the method may further include steps one through six:

[0110] Step 1: Acquire a first image using a camera other than the depth camera. The resolution of the first image is higher than that of the RGB image captured by the depth camera.

[0111] Step 2: After feature extraction from the first image, a second region is obtained, which contains only one product.

[0112] Step 3: Identify the product categories in the second area;

[0113] Step 4: Determine whether the first area and the second area contain the same product. If so, proceed to Step 5.

[0114] Step 5: Establish the correspondence between the first and second regions, and determine the category of goods in the second region as the category of goods in the first region;

[0115] Figure 1 Step S13 shown may specifically include:

[0116] Step Six: Determine the specifications of the goods in the first area based on the physical height and the category of the goods in the first area.

[0117] The depth camera and other cameras can be mounted on the same device or on different devices. When mounted on the same device, they can be adjacent left and right, vertically, or at a certain distance. The mounting positions ensure that the images captured by the depth camera and other cameras have the same viewing angle, facilitating the establishment of a correspondence between regions in the two images. When mounted on different devices, the different devices can be different patrol robots. Each patrol robot can have a fixed movement route. When it stops at a point on its movement route, the patrol robot carrying one camera captures an image of the shelf. The patrol robot carrying the other camera can stop at the same point, select the same shooting angle, and capture another image of the shelf. Since the image contains redundant image information in addition to the product image information, and this redundant image information is not conducive to product recognition, this embodiment can use a feature extractor to extract features from the first image. Specifically, the feature extractor can be a pre-trained product detection feature extractor. After feature extraction from the first image, at least one second region can be divided for the products contained in the first image. Each second region contains only one product. Because the resolution of the RGB image captured by the depth camera is lower than that of the first image, the accuracy of product category recognition is not high. Therefore, this embodiment can determine the product category in the RGB image captured by the depth camera using the first image. The higher the resolution of the first image, the higher the accuracy of product category recognition. This embodiment uses a classifier to determine the product category in the second region of the first image. When using the classifier to identify product categories, the size of the second region in the first image only needs to meet the input standard of the classifier. In this embodiment, the classifier has been trained, and the input standard is 256*256. After establishing a correspondence between the second region of the first image and the first region of the RGB image captured by the depth camera, the product category in the first region of the RGB image captured by the depth camera can be determined. After obtaining the product category in the first region, the specifications of the product in the first region can be determined by the physical height of the first region and the product category in the first region.

[0118] In another product identification method provided according to an embodiment of this application, step six of the above embodiment may include:

[0119] Extract feature points from the first image and the RGB image captured by the depth camera;

[0120] The feature points of the first image are matched with the feature points of the RGB image captured by the depth camera;

[0121] After a successful match, the second region contained in the first image is mapped onto the RGB image captured by the depth camera;

[0122] If the overlap rate between the first region in the RGB image and the mapped second region meets the preset requirements, then the first region and the second region are determined to be regions containing the same product.

[0123] In this embodiment, the correspondence between the first and second regions can be achieved using feature points from two images. Feature points are points containing rich local information, obtained through algorithmic analysis. Feature points can appear at corners or in areas with drastic texture changes in an image. Feature points possess "scale invariance," meaning they have the consistent property of being recognizable across different images. An image's feature point consists of two parts: keypoints and descriptors. Keypoints refer to the location of the feature point in the image; descriptors are typically vectors that describe the information of pixels surrounding the keypoint, designed manually. This embodiment uses the SIFT (Scale-invariant feature transform) algorithm to extract feature points from two images. After obtaining the feature points of the first and second regions, a feature point matching method can be used to match them. Feature point matching involves finding the feature point in the first region that is most similar to a feature point in the second region. The feature point matching method used in this embodiment is a brute-force matching algorithm. The brute-force matching algorithm calculates the distances between a feature point descriptor and all other feature point descriptors, sorts them, and selects the feature point corresponding to the closest feature descriptor as the matching point. Specifically, in this embodiment, when using the brute-force matching algorithm for feature point matching, feature point A in the first image is selected. The distances between the feature descriptor of feature point A and the feature descriptors of all feature points in the RGB image captured by the depth camera are calculated. After sorting the distances, the feature point corresponding to the closest feature descriptor is selected as the matching point for feature point A. Alternatively, in another embodiment, feature point B in the RGB image captured by the depth camera can be selected. The distances between the feature descriptor of feature point B and the feature descriptors of all feature points in the first image are calculated. After sorting the distances, the feature point corresponding to the closest feature descriptor is selected as the matching point for feature point B. After finding matching points for all feature points in the image, region mapping is required. Region mapping can be the process of connecting matching points according to the connection method of feature points. After connecting matching points, a mapped region is formed, which corresponds to the region formed by connecting feature points. The mapping method for the region can be either mapping the second region contained in the first image to the RGB image captured by the depth camera, or mapping the first region in the RGB image captured by the depth camera to the first image. In this embodiment, the second region contained in the first image is mapped to the RGB image captured by the depth camera.Since the position of the second region mapped from the first image is fixed, the overlap rate between the second region mapped from the first image and all first regions in the RGB image captured by the depth camera can be calculated separately. The first region with the highest overlap rate with the second region mapped from the first image is determined as the first region containing the same product as the second region.

[0124] According to another product recognition method provided in the embodiments of this application, the method may further include: recognizing products in a first region of an RGB image captured by a depth camera to obtain the category of products in the first region;

[0125] Figure 1 Step S13 shown may specifically include:

[0126] The specifications of the goods in the first area are determined based on the physical height and the category of the goods in the first area.

[0127] In this embodiment, if the RGB image captured by the depth camera has a high resolution, then another high-resolution image is no longer needed to assist in identifying the product category. This embodiment can identify products using only the RGB image captured by the depth camera. After determining the product category in the first region, this embodiment can determine the specifications of the products in the first region based on the physical height of the first region and the product category.

[0128] like Figure 4 As shown, in another product identification method provided according to an embodiment of this application, the method may further include:

[0129] S40. Obtain an RGB image of the shelf taken by a depth camera, where there is at least one item on the shelf;

[0130] S41. Obtain the first image of the shelf, which has at least one item on it;

[0131] S42. The RGB image captured by the depth camera is sent to the product detection feature extractor for feature extraction to obtain the first region;

[0132] S43. The first image is sent to the commodity detection feature extractor for feature extraction to obtain the second region;

[0133] S44. Calculate the physical height of the first region based on the depth information collected by the depth camera and the optical feature parameters of the depth camera;

[0134] S45. After cropping the second region in the first image, adjust the size of the second region to 256*256 and send it to the classifier to obtain the category of the goods in the second region;

[0135] S46. Extract feature points from the first image and the RGB image captured by the depth camera using the SIFT algorithm;

[0136] S47. Match the feature points of the first image with the feature points of the RGB image captured by the depth camera using a brute-force matching algorithm;

[0137] S48. After successful feature point matching, the first region is mapped onto the first image;

[0138] S49. Calculate the overlap rate between the first region and all second regions on the first image, and select the region with the highest overlap rate as the second region that matches the first region.

[0139] S50. Determine the physical height of the second region based on the matching relationship between the first and second regions;

[0140] S51. Determine the different specifications corresponding to the product categories in the second area through the product specification table;

[0141] S52. Subtract the physical height of the second region from the physical heights corresponding to different specifications to obtain at least one difference value;

[0142] S53. If the difference between the physical height of the second region and one of the physical heights is the minimum of at least one of the above differences, then the specification corresponding to that physical height is determined as the specification of the product in the second region.

[0143] S54. Obtain the product name with specifications based on the product specifications.

[0144] The order of steps S40 and S41, S42 and S43, S44 and S45, and S44-S45 and S46-S49 is not limited. This embodiment uses a mapping method that maps the first region onto the first image. This embodiment can obtain the product name with specified specifications based on the product's specifications and category. For example, if the product category in the second region is known to be "Xiaoming Mineral Water," and the specified size is 500 ml, then the product name would be "Xiaoming Mineral Water 500 ml."

[0145] The commodity identification device provided in the embodiments of this application is described below. The commodity identification device described below and the commodity identification method described above can be referred to in correspondence.

[0146] like Figure 5 As shown in the figure, this application embodiment also provides a product identification device, which may include:

[0147] Image acquisition unit 100 is used to acquire RGB images of a shelf taken by a depth camera, wherein the shelf has at least one product.

[0148] The feature extraction unit 110 is used to extract features from the RGB image captured by the depth camera to obtain a first region, which contains only one product.

[0149] The height acquisition unit 120 is used to acquire the physical height of the first region based on the depth information collected by the depth camera;

[0150] Specification determination unit 130 is used to determine the specifications of goods in the first area based on physical height;

[0151] The name determination unit 140 is used to obtain a product name with specifications based on the product's specifications.

[0152] In another product identification device provided according to an embodiment of this application, the height acquisition unit 120 may include:

[0153] The distance acquisition subunit is used to obtain the first distance from the actual product area to the depth camera corresponding to the first pixel in the first region, based on the correspondence between the RGB image captured by the depth camera and the depth information.

[0154] The calculation subunit is used to calculate the physical height of the first region based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance.

[0155] In another product recognition device provided according to an embodiment of this application, the focal length includes the focal length f in the x-axis direction. x and the focal length f in the y-axis direction y The computational subunit may include:

[0156] Mean acquisition sub-unit, used to obtain f x and f y The mean f;

[0157] The center point coordinate acquisition sub-unit is used to obtain the top center point coordinates (x, y) of the first region based on the coordinates of the first region. m y1) and the coordinates of the bottom center point (x m ,y2);

[0158] According to the formula:

[0159] w xt =(x m -c x )×c z ÷f

[0160] w yt =(y1-cy )×c z ÷f

[0161] w xb =(x m -c x )×c z ÷f

[0162] w yb =(y2-c y )×c z ÷f

[0163] The coordinate calculation sub-unit is used to calculate the physical world coordinates (w) of the top center point of the first region. xt w yt ) and the physical world coordinates of the bottom center point (w xb w yb );

[0164] The height calculation subunit is used to calculate the height based on (w) xt w yt ) and (w xb w yb Calculate the physical height of the first region.

[0165] In another product identification device provided according to an embodiment of this application, the device may further include a category acquisition unit;

[0166] The category acquisition unit is used to acquire the category of the goods in the first area;

[0167] The specification determination unit 130 may specifically include:

[0168] The category-corresponding sub-unit is used to determine at least one specification corresponding to the category of the goods in the first area, and the specification corresponds to the height of the goods;

[0169] The height comparison subunit is used to compare the physical height with the height of the product corresponding to at least one specification, and determine the specification of the product in the first region based on the comparison result.

[0170] In another product identification device provided according to an embodiment of this application, the device may further include:

[0171] The first image acquisition unit is used to acquire a first image through a camera other than the depth camera, wherein the resolution of the first image is higher than the resolution of the RGB image captured by the depth camera.

[0172] The image feature extraction unit is used to extract features from the first image to obtain a second region, which contains only one product.

[0173] Category identification unit, used to identify the category of goods in the second area;

[0174] The relationship determination unit is used to determine whether the first region and the second region are regions containing the same product. If so, the relationship establishment unit is executed.

[0175] The relationship establishment unit is used to establish the correspondence between the first region and the second region, and to determine the category of goods in the second region as the category of goods in the first region;

[0176] The specification determination unit 130 is specifically configured to determine the specifications of the goods in the first area based on the physical height and the category of the goods in the first area.

[0177] In another product identification device according to an embodiment of this application, the relationship determination unit may include:

[0178] The feature point extraction subunit is used to extract feature points from the first image and the RGB image captured by the depth camera;

[0179] The feature point matching subunit is used to match the feature points of the first image with the feature points of the RGB image captured by the depth camera;

[0180] The region mapping subunit is used to map the second region contained in the first image to the RGB image captured by the depth camera after a successful match.

[0181] If the overlap rate between the first region in the RGB image and the mapped second region meets the preset requirements, then the region determination subunit is executed;

[0182] The region determination subunit is used to determine whether the first region and the second region contain the same product.

[0183] In another product identification device provided according to an embodiment of this application, the device may further include a category acquisition unit;

[0184] The category acquisition unit is used to identify the products in the first region of the RGB image captured by the depth camera and obtain the category of the products in the first region;

[0185] The specification determination unit 130 is specifically configured to determine the specifications of the goods in the first area based on the physical height and the category of the goods in the first area.

[0186] This application also provides an electronic device. Figure 6 A hardware block diagram of the electronic device is shown, with reference to... Figure 6 The hardware structure of the electronic device may include: memory 1 and processor 2;

[0187] Memory 1 is used to store programs;

[0188] Processor 2 is used to execute programs to implement the various steps of any of the above-mentioned image recognition methods.

[0189] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the various steps of any of the above-described image recognition methods.

[0190] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] In a typical configuration, the device includes one or more processors (CPUs), memory, and a bus. The device may also include input / output interfaces, network interfaces, etc.

[0192] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0193] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0194] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0195] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0196] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0197] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A product identification method, characterized in that, The method includes: Acquire RGB images of a shelf, taken by a patrol robot or a person using a depth camera, wherein the shelf has at least one item; After extracting features from the RGB image captured by the depth camera using a pre-trained product detection feature extractor, a first region is obtained. The first region contains only one product. The first region is a rectangular region, and the four sides of the rectangular region are connected to the image boundary of the product. The border of the first region is the product detection box. A first image is acquired using cameras other than the depth camera, the resolution of which is higher than the resolution of the RGB image captured by the depth camera. A second region is obtained by feature extraction from the first image, and this second region contains only one product. The category of the product in the second region is identified. It is determined whether the first region and the second region contain the same product; if so, a correspondence is established between the first region and the second region, and the category of the product in the second region is determined to be the same as the category of the product in the first region. The determination of whether the first region and the second region contain the same product includes: extracting feature points from the first image and the RGB image captured by the depth camera using the SIFT algorithm; matching the feature points of the first image and the RGB image captured by the depth camera using a brute-force matching algorithm; after successful matching, connecting the matching points according to the connection method of the feature points to map the second region contained in the first image to the RGB image captured by the depth camera; calculating the overlap rate between each first region in the RGB image captured by the depth camera and the mapped second region, and determining the first region with the highest overlap rate as the region containing the same product as the second region. Based on the depth information collected by the depth camera, the physical height of the first region is obtained; The specifications of the goods in the first region are determined based on the physical height and the category of the goods in the first region; the determination of the specifications of the goods in the first region based on the physical height and the category of the goods in the first region includes: after determining the category of the goods in the first region, subtracting the physical height of the first region from the physical height corresponding to different specifications, thereby obtaining at least one difference value; if the difference between the physical height of the first region and one of the physical heights is the minimum value among the above at least one difference value, then the specification corresponding to that physical height is determined as the specification of the goods in the first region; The product name containing the product specifications is obtained by using the product specifications.

2. The method according to claim 1, characterized in that, The step of obtaining the physical height of the first region based on the depth information collected by the depth camera includes: The first distance from the actual product area to the depth camera is obtained based on the correspondence between the RGB image captured by the depth camera and the depth information; The physical height of the first region is calculated based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance.

3. The method according to claim 2, characterized in that, The focal length includes focal length along the axis focal length along the y-axis The step of calculating the physical height of the first region based on the focal length of the depth camera, the coordinates of the imaging origin of the depth camera, the coordinates of the first region, and the first distance includes: Obtain the and stated mean ; Obtain the coordinates of the top center point of the first region based on the coordinates of the first region. , ) and the coordinates of the bottom center point ( , ); According to the formula: Calculate the physical world coordinates of the top center point of the first region. , ) and the physical world coordinates of the bottom center point ( , ); According to the above ( , ) and the ( , Calculate the physical height of the first region; in, For the first distance, ( , ) represents the coordinates of the imaging origin of the depth camera.

4. The method according to claim 1, characterized in that, Also includes: Obtain the category of the goods in the first region; Determining the specifications of goods in the first region based on the physical height includes: Determine at least one specification corresponding to the category of goods in the first region, wherein the specification corresponds to the height of the goods; The physical height is compared with the product height corresponding to the at least one specification, and the specification of the product in the first region is determined based on the comparison result.

5. The method according to claim 1, characterized in that, Also includes: The product category in the first region of the RGB image captured by the depth camera is obtained by identifying the product in the first region. Determining the specifications of goods in the first region based on the physical height includes: The specifications of the goods in the first area are determined based on the physical height and the category of the goods in the first area.

6. A product identification device, characterized in that, The device includes: An image acquisition unit is used to acquire RGB images of a shelf taken by a patrol robot or a person using a depth camera, wherein the shelf has at least one product. The feature extraction unit is used to extract features from the RGB image captured by the depth camera using a pre-trained product detection feature extractor to obtain a first region, which contains only one product; wherein, the first region is a rectangular region, the four sides of the rectangular region are connected to the image boundary of the product, and the border of the first region is a product detection box; The first image acquisition unit is used to acquire a first image through a camera other than the depth camera, wherein the resolution of the first image is higher than the resolution of the RGB image captured by the depth camera; An image feature extraction unit is used to extract features from the first image to obtain a second region, wherein the second region contains only one product. A category identification unit is used to identify the category of goods in the second area; The relationship determination unit is used to determine whether the first region and the second region are regions containing the same product; if so, the relationship establishment unit is executed. The relationship determination unit is specifically used to extract feature points of the first image and feature points of the RGB image captured by the depth camera using the SIFT algorithm; to match the feature points of the first image and the feature points of the RGB image captured by the depth camera using a brute-force matching algorithm; after a successful match, the matching points are connected according to the connection method of the feature points to map the second region contained in the first image to the RGB image captured by the depth camera; the overlap rate between each first region in the RGB image captured by the depth camera and the mapped second region is calculated respectively, and the first region with the highest overlap rate is determined as the region containing the same product as the second region; The relationship establishment unit is used to establish a correspondence between the first region and the second region, and to determine the category of the goods in the second region as the category of the goods in the first region; The height acquisition unit is used to acquire the physical height of the first area based on the depth information collected by the depth camera; A specification determination unit is used to determine the specifications of the goods in the first area based on the physical height and the category of the goods in the first area; The specification determination unit is specifically used to, after determining the category of goods in the first area, subtract the physical height of the first area from the physical height corresponding to different specifications, thereby obtaining at least one difference value; if the difference between the physical height of the first area and one of the physical heights is the minimum value among the above at least one difference value, then the specification corresponding to that physical height is determined as the specification of the goods in the first area. A name determination unit is used to obtain a product name containing the specifications of the product.

7. An electronic device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the commodity identification method as claimed in any one of claims 1-5.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the commodity identification method as claimed in any one of claims 1-5.

Images