Method and device for determining placement information of target object

Abstract

The present invention discloses a method and device for determining the placement information of a target object. The invention comprises: processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; determining multiple object frames from the target image, and determining the target object frame based on the multiple object frames and the relative position relationship between the multiple object frames and the target price tag image, wherein the object frames other than the target object frame in the multiple object frames are determined as multiple comparison object frames. The method and device are applied in the field of supermarkets and stores, and solve the technical problem in the related art that the inspection of the goods on the shelves by manpower leads to low detection efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and a device for determining placement information of a target object. Background Art

[0002] In the related technologies, current supermarket shelf management mainly relies on manual management. Since the shelves need to be manually inspected before the relevant information of the goods placed on the shelves can be identified, the efficiency is low and the goods on the shelves cannot be replenished in time. Based on the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0003] The main purpose of the present invention is to provide a method and device for determining the placement information of a target object, so as to improve the efficiency of detecting the conditions of goods on a shelf.

[0004] In order to achieve the above-mentioned object, according to one aspect of the present invention, a method for determining the placement information of a target object is provided. The invention comprises: processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; determining multiple object frames from the target image, and determining a target object frame based on the multiple object frames and the relative position relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is an object frame in the multiple object frames; determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein the object frames other than the target object frame in the multiple object frames are determined as multiple comparison object frames.

[0005] Furthermore, the object placement information includes at least one of the following: placement quantity information of the target objects; placement position information of the target objects; vacant position information of the target objects; and misplacement information of the target objects.

[0006] Furthermore, before processing the target image containing the target object and determining the target price tag image, the method includes: calibrating the shelf area and obtaining the target image corresponding to the calibrated shelf area; performing edge extraction on the target image; performing straight line detection on the extracted edges to obtain multiple straight lines; identifying the multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag; and identifying the price tag column through the upper and lower boundaries.

[0007] Furthermore, identifying multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag includes: counting multiple slopes corresponding to the multiple straight lines, and eliminating straight lines with absolute values ​​of slopes greater than or equal to 1 to determine multiple first target straight lines; calculating the slope differences between any two of the multiple first target straight lines; eliminating invalid straight lines to obtain multiple second target straight lines, where invalid straight lines are straight lines with slope differences greater than a preset value; determining the distance values ​​between any two of the multiple second target straight lines; and respectively determining two straight lines corresponding to distance values ​​within a preset range as the upper and lower boundaries corresponding to the price tag column.

[0008] Furthermore, the target image containing the target object is processed to determine the target price tag image, including: enlarging and scanning the image corresponding to the price tag column by a variable-magnification ball camera, and performing quadrilateral detection on the image to obtain a quadrilateral image; extracting text information in the quadrilateral image; if the text information is text related to the target object, determining that the quadrilateral image is the target price tag image.

[0009] Furthermore, if the text information is text related to the target object, after determining that the quadrilateral image is the target price tag image, the method further includes: saving the text information and the coordinates corresponding to the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera.

[0010] Furthermore, determining multiple object frames from the target image includes: using a variable-magnification ball camera to enlarge and scan an image corresponding to the target object placement area, wherein the area between two price tag columns in the target image is determined as the target object placement area; detecting the image corresponding to the target object placement area through a target object detection model, and determining multiple object frames.

[0011] Furthermore, before determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, the method further includes: extracting first text information and first color texture information from the target object frame; extracting a plurality of second text information and a plurality of second color texture information from the plurality of comparison object frames; determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames includes: sequentially comparing the plurality of second text information with the first text information; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; If the first text information does not match, the second color texture information is compared with the first color texture information in turn; if the second color texture information matches the first color texture information, it is determined that the object corresponding to the comparison object box and the target object are the same type of objects; if any one of the second text information and the second color texture information in the comparison object box is empty, the comparison object box is determined to be an empty object box; based on the type difference between the object corresponding to the comparison object box and the target object, at least one of the placement quantity information of the target object, the placement position information of the target object and the position misplacement information of the target object is determined; based on the empty object box, the vacant position information of the target object is determined.

[0012] In order to achieve the above-mentioned object, according to another aspect of the present invention, a device for determining the placement information of a target object is provided. The device comprises: a first determination unit, for processing a target image containing a target object, and determining a target price tag image, wherein the target price tag image is a price tag image including the target object; a second determination unit, for determining a plurality of object frames from the target image, and determining a target object frame according to the plurality of object frames and the relative positional relationship between the plurality of object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is an object frame in the plurality of object frames; a third determination unit, for determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, wherein object frames other than the target object frame in the plurality of object frames are determined as the plurality of comparison object frames.

[0013] Furthermore, the object placement information includes at least one of the following: placement quantity information of the target objects; placement position information of the target objects; vacant position information of the target objects; and misplacement information of the target objects.

[0014] Through the present invention, the following steps are adopted: processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; determining multiple object frames from the target image, and determining a target object frame based on the multiple object frames and the relative position relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is an object frame in the multiple object frames; determining object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein object frames other than the target object frame in the multiple object frames are determined as the multiple comparison object frames. Therefore, the above method is applied to the goods management in the supermarket, and the placement information of the target object in the shelf can be automatically identified, thereby improving the monitoring efficiency of the target object, and further solving the technical problem of low detection efficiency caused by manpower inspection of the goods on the shelf in the overdue goods management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0016] Figure 1 is a flowchart of a method for determining placement information of a target object provided according to an embodiment of the present invention; and

[0017] Figure 2 Schematic diagram of three relative position relationships between a target price tag and a product box provided according to an embodiment of the present invention;

[0018] Figure 3 is a schematic diagram of determining the area corresponding to the target price tag through quadrilateral detection provided by an embodiment of the present invention;

[0019] Figure 4 is a schematic diagram of commodity detection using a detection model provided in an embodiment of the present invention;

[0020] Figure 5 is a schematic diagram of a comparison area determined according to the position of a target price tag provided according to an optional embodiment of the present invention;

[0021] Figure 6 is a flow chart of comparing feature vectors between a target product frame and a comparison object frame provided by an embodiment of the present invention;

[0022] Figure 7 is a schematic diagram of a comparison result of feature vectors between a target product frame and a comparison object frame provided by an embodiment of the present invention;

[0023] Figure 8A schematic diagram showing that a target commodity is out of stock after comparing feature vectors according to an embodiment of the present invention;

[0024] Fig. 9 It is a schematic diagram showing the situation where there is a misplaced product at the placement position of the target product after feature vector comparison;

[0025] Fig.10 is a schematic diagram of a device for determining placement information of a target object provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] According to an embodiment of the present invention, a method for determining placement information of a target object is provided.

[0030] Figure 1 FIG. 1 is a flow chart of a method for determining target object placement information provided according to an embodiment of the present invention. Figure 1 As shown, the invention comprises the following steps:

[0031] Step S101, processing a target image including a target object to determine a target price tag image, where the target price tag image is a price tag image including the target object;

[0032] As described above, a method for determining the placement information of a target object provided in an embodiment of the present application can be specifically applied to supermarket commodity identification, wherein, in one implementation, the object in the embodiment of the present application can be a commodity placed on a supermarket shelf for sale.

[0033] Among them, in the embodiments provided in the present application, monitoring and identifying attributes of goods by collecting images and processing the collected images can greatly improve management efficiency, and a product information database can be established to help supermarkets optimize object types.

[0034] It should be noted that in the embodiments of the present application, only commodities are used as examples of objects for illustration, and those skilled in the art can also flexibly set the objects to other items, etc.

[0035] Specifically, the target price tag image is first determined by processing the captured target image, wherein the target price tag image is an image containing a price tag corresponding to the target object, that is, the target object to be processed can be determined by determining the price tag image of the target object, which is the target product in this embodiment. For example, by processing the target image, it is determined that the target product is Coca-Cola and the price tag image corresponding to Coca-Cola.

[0036] Step S102, determining multiple object frames from the target image, and determining a target object frame based on the multiple object frames and the relative position relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is one of the multiple object frames.

[0037] Specifically, by determining the relative position relationship between multiple product frames and the target price tag in the target product placement area, a target product frame equivalent to the anchor sample is determined. It should be noted that in the embodiment of the present application, the product image contained in the target product frame matches the target price tag by default. In the present embodiment, the size of the image of the product frame is substantially consistent with the image size of the target product. In the present application, the product frames determined in sequence are by default the products arranged in sequence on the shelf. For example, taking Coca-Cola as an example of the target product, a can of Coca-Cola is first determined to be placed in the product frame corresponding to the Coca-Cola by default. The placement of Coca-Cola is determined by processing the shelf image to determine whether corresponding Coca-Cola is placed in other comparison product frames.

[0038] Furthermore, the relative position relationship between the target price tag and the target product is generally divided into three types, such as Figure 2 As shown, based on these relative position relationships, the product frame that is closest to the price tag coordinates is taken as the target product frame (also called the comparison anchor sample). It can be considered that this target product frame matches the product price tag type.

[0039] It should also be noted that in the embodiment of the present application, the relative position relationship is actually set according to the actual situation. Preferably, the product box at the top of the target price tag is determined as the target product box that matches the target price tag. Optionally, if it is detected that the product box at the top of the target price tag is an empty product box, optionally, the product box adjacent to the empty product box can be determined as the target product box.

[0040] Step S103 : determining object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein object frames other than the target object frame among the multiple object frames are determined as the multiple comparison object frames.

[0041] Further, after the target commodity frame is determined, other commodity frames in the determined target commodity placement area are determined as comparison object frames to be compared with the target commodity frame.

[0042] Specifically, the feature vector in the comparison object frame is compared with the feature vector in the target product frame in turn to determine the object placement information of the target product (in this embodiment, the object placement information is the product placement information. It should be noted that the objects in this application change with the application of different scenarios).

[0043] The object placement information includes at least one of the following: placement quantity information of the target object; placement position information of the target object; vacant position information of the target object; and misplacement information of the target object.

[0044] Specifically, by comparing the feature vectors in the comparison object frame with the target product frame, it is possible to determine whether the product in the comparison product frame is of the same type as the target product or whether the comparison frame is an empty product frame, and the product placement information of the target product can be determined by determining the situation of the product in the comparison product frame.

[0045] It should be noted that, in this embodiment, the target object is the target product, and the comparison object frame is the comparison product frame.

[0046] Further, when the products in the comparison product frame and the target product are the same type of products, the placement quantity of the target product is determined to be plus 1, that is, the placement quantity information of the target product is determined.

[0047] Furthermore, the placement information of the target product can be determined by comparing the position information of the product frame.

[0048] Meanwhile, if the products in the comparison product frame are empty, the position corresponding to the comparison product frame is determined to be the vacant position of the target product. Therefore, the vacant position information of the target product can be determined through the empty product frame.

[0049] On the other hand, if the products in the comparison product frame are not the same type as the target product, the position corresponding to the comparison product frame is determined as the misplaced position of the target product, that is, the misplaced position information of the target product is determined.

[0050] Optionally, before processing the target image containing the target object to determine the target price tag image, the method includes:

[0051] Calibrate the shelf area and obtain the target image corresponding to the calibrated shelf area;

[0052] Specifically, before determining the target price tag image, the shelf area needs to be calibrated first. Since the position of the camera is fixed, the shelf area only needs to be calibrated once.

[0053] Extract the edge of the target image; perform straight line detection on the extracted edge to obtain multiple straight lines;

[0054] Secondly, identify the shelf price tag column: including: processing the target image captured by the camera and processing the target image, extracting the edge of the target image, and performing straight line detection on the edge to obtain a number of straight lines.

[0055] Identify multiple straight lines and determine the upper and lower boundaries of the price tag column corresponding to the target price tag;

[0056] Next, multiple straight lines are identified, and the upper boundary and the lower boundary of the price tag column corresponding to the target price tag are determined; the price tag column is identified through the upper boundary and the lower boundary.

[0057] Identify the price tag column by its upper and lower boundaries.

[0058] Finally, the corresponding area of ​​the price tag column in the target image is determined by identifying the upper boundary and the lower boundary of the price tag column.

[0059] Optionally, identifying multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag includes: counting multiple slopes corresponding to the multiple straight lines, and eliminating straight lines with absolute values ​​of slopes greater than or equal to 1 to determine multiple first target straight lines; calculating the slope differences between any two of the multiple first target straight lines; eliminating invalid straight lines to obtain multiple second target straight lines, where invalid straight lines are straight lines with slope differences greater than a preset value; determining the distance values ​​between any two of the multiple second target straight lines; and respectively determining two straight lines corresponding to distance values ​​within a preset range as the upper and lower boundaries corresponding to the price tag column.

[0060] As mentioned above, since the angle of photographing the shelf is generally horizontal, the target image containing the target product is processed to obtain multiple straight lines, and outliers, that is, straight lines with obvious slopes that are too different from others, are removed, and the straight lines with the absolute value of the slope |k|<1 are counted. At the same time, the straight lines outside the shelf calibration area are removed.

[0061] An optional embodiment, since the difference in slopes of the two straight lines corresponding to the upper and lower boundaries of a price tag column is the smallest, the slope difference between any two straight lines among the multiple straight lines is calculated, and when the slope difference is within a very small preset range, the two straight lines corresponding to the slopes are determined as the two upper boundaries and the lower boundary corresponding to a price tag column.

[0062] In another optional embodiment, since the spacing between the straight lines between the price tag columns conforms to the rule of alternating wide and narrow, the multiple second target straight lines are sorted from top to bottom, and the upper and lower boundaries of the label column are determined according to the rule of alternating wide and narrow.

[0063] Optionally, the target image containing the target object is processed to determine the target price tag image, including: enlarging and scanning the image corresponding to the price tag column by a variable-magnification ball camera, and performing quadrilateral detection on the image to obtain a quadrilateral image; extracting text information in the quadrilateral image; if the text information is text related to the target object, determining that the quadrilateral image is the target price tag image.

[0064] Optionally, if the text information is text related to the target object, after determining that the quadrilateral image is the target price tag image, the method further includes: saving the text information and the coordinates corresponding to the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera.

[0065] As mentioned above, since the information of the price tag and the product corresponding to the price tag is determined by image processing means in the present application, in this embodiment, the image corresponding to the price tag column is enlarged and scanned by a zoom ball camera. Since the price tags in supermarkets or stores are generally quadrilaterals, the quadrilateral image is determined by quadrilateral detection, and the quadrilateral image containing text related to the target product is determined as the image of the target price tag corresponding to the target product.

[0066] Furthermore, since the price tag is small, in order to ensure the recognition effect in a large scene, we use a zoom dome camera as the image acquisition device. When scanning the price tag column, the zoom center of the dome camera is the center line of the price tag column, and the scan is enlarged from left to right along the center line. The hough transform and other methods are used for quadrilateral detection. For details, Figure 3As shown, the quadrilateral range is obtained, and text character detection and recognition are performed within this range to extract the text information in the price tag. If no text information related to the product is extracted, the price tag is considered to be a false detection, discarded, and the price tag column is scanned again. If the text information related to the product is extracted, the information is saved, and the coordinates of the center point of the price tag quadrilateral in the absolute coordinate system of the ball camera are saved for backtracking.

[0067] Optionally, determining multiple object frames from the target image includes: using a variable-magnification ball camera to enlarge and scan an image corresponding to the target object placement area, wherein the area between two price tag columns in the target image is determined as the target object placement area; detecting the image corresponding to the target object placement area through a target object detection model, and determining multiple object frames.

[0068] As described above, the boundary of the commodity placement area can be determined by identifying the upper and lower boundaries of the price tag column. Therefore, the area between the lower boundary of the upper price tag column and the upper boundary of the lower price tag column of two adjacent price tag columns is determined as the target commodity placement area.

[0069] Furthermore, after zooming in with a zoom ball camera, scan along the center line of the product area from left to right, and use general detection models such as yolo and frcnn to detect the product. After the product is detected, the position of the product frame in the absolute coordinate system of the ball camera is saved. The schematic diagram of using the detection model for product detection is shown in the following figure. Figure 4 shown.

[0070] Optionally, before determining the object placement information of the target object based on the similarity between the target object frame and the plurality of comparison object frames, the method further includes:

[0071] Extracting first text information and first color texture information from the target object frame;

[0072] Extracting a plurality of second text information and a plurality of second color texture information from a plurality of comparison object frames;

[0073] Based on the similarity between the target object frame and the multiple comparison object frames, determining the object placement information of the target object includes:

[0074] sequentially comparing the plurality of second text messages with the first text message;

[0075] If the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are the same type of objects;

[0076] If the second text information does not match the first text information, then sequentially comparing the second color texture information with the first color texture information;

[0077] If the second color texture information matches the first color texture information, it is determined that the object corresponding to the comparison object frame and the target object are the same type of objects;

[0078] If any one of the second text information and the second color texture information in the comparison object frame is empty, determining that the comparison object frame is an empty object frame;

[0079] Determine at least one of placement quantity information of the target object, placement position information of the target object, and position misplacement information of the target object according to the type difference between the object corresponding to the comparison object frame and the target object;

[0080] Based on the empty object frame, determine the vacant position information of the target object.

[0081] As described above, the information on the placement of goods in the product placement area is obtained by extracting the image feature vectors in the target product frame and the comparison product frame and comparing them in sequence.

[0082] It should be noted that, in the embodiment provided in the present application, it is also necessary to determine the comparison range of the products so as to use the multiple product frames existing in the product comparison range as the comparison product frames.

[0083] Optionally, in an optional embodiment, the target commodity placement area corresponding to the area between two price tags adjacent to the left and right of the target price tag is used as the comparison area. The specific schematic diagram is as follows: Figure 5 shown.

[0084] Optionally, comparing the feature vector of the target product frame with the feature vectors of multiple comparison product frames in sequence to obtain a comparison result, and obtaining the placement information of the target product according to the comparison result includes:

[0085] Comparing the plurality of second text information with the first text information in sequence; if the second text information matches the first text information, determining that the product corresponding to the comparison product box and the target product are the same type of products;

[0086] If the second text information does not match the first text information, then sequentially comparing the second color texture information with the first color texture information;

[0087] If the second color texture information matches the first color texture information, it is determined that the product corresponding to the comparison product frame and the target product are the same type of products;

[0088] As mentioned above, the specific feature vector comparison flow chart is as follows Figure 6 As shown, and the comparison results in general are shown in the following figure Figure 7 As shown, Figure 7 As shown in , the products marked with a check mark are consistent with the type of the target product, and the products marked with a cross are inconsistent with the type of the target product.

[0089] If any one of the second text information and the second color texture information in the comparison product frame is empty, determining that the comparison product frame is an empty product frame;

[0090] According to the difference in type between the product corresponding to the comparison product frame and the target product, determine the placement quantity information of the target product, the placement position information of the target product, and the misplacement information of the target product;

[0091] Based on the empty product frame, determine the vacant position information of the target product.

[0092] As described above, an empty product frame can be determined by extracting the feature vector in the comparison product frame, that is, there is no product at the position corresponding to the product frame.

[0093] Furthermore, when there is Figure 8 In the case shown, Figure 8 This is a schematic diagram showing that the target product is out of stock after comparing the feature vectors. Figure 8 The blank part in the middle corresponds to the empty product box. The blank part indicates that the product is out of stock, which means that the current product is out of stock and needs to be replenished, and the out-of-stock position is reported and processed.

[0094] Furthermore, if the Fig. 9 In the case shown, Fig. 9 This is a schematic diagram showing the situation where there are misplaced items at the target product's placement position after feature vector comparison. The items marked with a cross are misplaced items, that is, there is a mismatch within the product matching range. It is possible that the customer has placed other items here, and a misplacement notification needs to be reported to notify supermarket employees to put them back to the correct location.

[0095] Therefore, the placement quantity, placement position, misplaced position and vacant position of the target product can be determined by comparing the target product frame to determine whether the target product frame belongs to the same type of product.

[0096] A method for determining placement information of a target object provided by an embodiment of the present invention determines a target price tag image by processing a target image including a target object, wherein the target price tag image is a price tag image including the target object; a target object frame is determined from multiple object frames based on multiple object frames determined in the target image, and a relative position relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object; and placement information of the target object is determined based on the target object frame and the similarity between the multiple object frames, wherein the target object placement information is at least one of the following: placement quantity information of the target object, placement position information of the target object, vacant position information of the target object, and misplaced position information of the target object, thereby solving the technical problem in the related art that the inspection of the objects on the shelf by manpower leads to low detection efficiency, and achieving the technical effect of improving the efficiency of goods management.

[0097] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0098] The embodiment of the present invention also provides a device for determining the placement information of a target object. It should be noted that the device for determining the placement information of a target object in the embodiment of the present invention can be used to execute the method for determining the placement information of a target object provided in the embodiment of the present invention. The following introduces a device for determining the placement information of a target object provided in the embodiment of the present invention.

[0099] Fig.10 is a schematic diagram of a device for determining placement information of a target object provided according to an embodiment of the present invention. Fig.10 As shown, the device comprises:

[0100] A first determining unit 101 is used to process a target image including a target object to determine a target price tag image, where the target price tag image is a price tag image including the target object;

[0101] A second determining unit 102 is used to determine a plurality of object frames from the target image, and determine a target object frame according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is one of the plurality of object frames;

[0102] The third determining unit 103 is configured to determine the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein object frames other than the target object frame among the multiple object frames are determined as the multiple comparison object frames.

[0103] Optionally, the object placement information includes at least one of the following: placement quantity information of the target objects; placement position information of the target objects; vacant position information of the target objects; and misplacement information of the target objects.

[0104] Optionally, the device includes: a calibration unit, used to calibrate the shelf area and obtain the target image corresponding to the calibrated shelf area before processing the target image containing the target object and determining the target price tag image; a first extraction unit, used to extract edges of the target image; a detection unit, used to perform straight line detection on the extracted edges to obtain multiple straight lines; a first recognition unit, used to recognize multiple straight lines and determine the upper and lower boundaries of the price tag column corresponding to the target price tag; and a second recognition unit, used to recognize the price tag column through the upper and lower boundaries.

[0105] Optionally, the first identification unit includes: a first elimination module, used to count multiple slopes corresponding to multiple straight lines, and eliminate straight lines with absolute values ​​of slopes greater than or equal to 1 to determine multiple first target straight lines; a calculation module, used to calculate the slope differences between any two of the multiple first target straight lines; a second elimination module, used to eliminate invalid straight lines to obtain multiple second target straight lines, invalid straight lines being straight lines with slope differences greater than a preset value; a first determination unit 101, used to determine the distance values ​​between any two of the multiple second target straight lines; and a second determination unit 102, used to respectively determine two straight lines corresponding to distance values ​​within a preset range as the upper boundary and lower boundary corresponding to the price tag column.

[0106] Optionally, the first determination unit 101 includes: a first scanning module, used to enlarge and scan the image corresponding to the price tag column through a variable magnification ball camera, and perform quadrilateral detection on the image to obtain a quadrilateral image; an extraction module, used to extract text information in the quadrilateral image; a first determination module, used to determine that the quadrilateral image is a target price tag image when the text information is text related to the target object.

[0107] Optionally, the device further comprises: a storage unit for storing the text information and the corresponding coordinates of the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera after determining that the quadrilateral image is the target price tag image if the text information is text related to the target object.

[0108] Optionally, the second determination unit 102 includes: a second scanning module, used to use a variable magnification ball camera to enlarge and scan the image corresponding to the target object placement area, wherein the area between two price tag columns in the target image is determined as the target object placement area; a second determination module, used to detect the image corresponding to the target object placement area through a target object detection model, and determine multiple object frames.

[0109] Optionally, the device further includes: a second extraction unit, used to extract the first text information and the first color texture information in the target object frame before determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames; a third extraction unit, used to extract multiple second text information and multiple second color texture information in the multiple comparison object frames; a third determination unit 103, used to determine the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, including: a first comparison unit, used to compare the multiple second text information with the first text information in sequence; a fourth determination unit, used to determine that the object corresponding to the comparison object frame and the target object are the same type of object when the second text information matches the first text information; a second comparison unit, used to compare the second text information with the first text information in sequence; When the second text information does not match the first text information, the second color texture information is compared with the first color texture information in sequence; a fifth determination unit is used to determine that the object corresponding to the comparison object box and the target object are the same type of objects when the second color texture information matches the first color texture information; a sixth determination unit is used to determine that the comparison object box is an empty object box when any one of the second text information and the second color texture information in the comparison object box is empty; a seventh determination unit is used to determine at least one of the placement quantity information of the target object, the placement position information of the target object and the position misplacement information of the target object based on the type difference between the object corresponding to the comparison object box and the target object; an eighth determination unit is used to determine the vacant position information of the target object based on the empty object box.

[0110] An embodiment of the present invention provides a device for determining placement information of a target object. The device includes a first determination unit 101 for processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object. A second determination unit 102 is used to determine a plurality of object frames from the target image, and determine the target object frame according to the plurality of object frames and the relative position relationship between the plurality of object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is one of the plurality of object frames. A third determination unit 103 is used to determine object placement information of the target object based on the similarity between the target object frame and a plurality of comparison object frames, wherein object frames other than the target object frame in the plurality of object frames are determined as a plurality of comparison object frames. By applying the above method in supermarkets and other stores, the technical problem of low detection efficiency caused by manpower inspection of objects on shelves in the related art is solved, and the technical effect of improving the efficiency of goods management is achieved.

[0111] A device for determining placement information of a target object includes a processor and a memory. The first determination unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0112] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set, and the technical problem of low detection efficiency caused by human inspection of the objects on the shelf in the related technology can be solved by adjusting the kernel parameters.

[0113] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0114] An embodiment of the present invention provides a computer storage medium on which a program is stored. When the program is executed by a processor, a method for determining placement information of a target object is implemented.

[0115] An embodiment of the present invention provides a processor, which is used to run a program, wherein a method for determining placement information of a target object is executed when the program is run.

[0116] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the program: processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; determining multiple object frames from the target image, and determining a target object frame based on the multiple object frames and the relative positional relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is an object frame in the multiple object frames; determining object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein object frames other than the target object frame in the multiple object frames are determined as the multiple comparison object frames.

[0117] Optionally, the object placement information includes at least one of the following: placement quantity information of the target objects; placement position information of the target objects; vacant position information of the target objects; and misplacement information of the target objects.

[0118] Optionally, before processing the target image containing the target object and determining the target price tag image, the method includes: calibrating the shelf area and obtaining the target image corresponding to the calibrated shelf area; performing edge extraction on the target image; performing straight line detection on the extracted edges to obtain multiple straight lines; identifying the multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag; and identifying the price tag column through the upper and lower boundaries.

[0119] Optionally, identifying multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag includes: counting multiple slopes corresponding to the multiple straight lines, and eliminating straight lines with absolute values ​​of slopes greater than or equal to 1 to determine multiple first target straight lines; calculating the slope differences between any two of the multiple first target straight lines; eliminating invalid straight lines to obtain multiple second target straight lines, where invalid straight lines are straight lines with slope differences greater than a preset value; determining the distance values ​​between any two of the multiple second target straight lines; and respectively determining two straight lines corresponding to distance values ​​within a preset range as the upper and lower boundaries corresponding to the price tag column.

[0120] Optionally, the target image containing the target object is processed to determine the target price tag image, including: enlarging and scanning the image corresponding to the price tag column by a variable-magnification ball camera, and performing quadrilateral detection on the image to obtain a quadrilateral image; extracting text information in the quadrilateral image; if the text information is text related to the target object, determining that the quadrilateral image is the target price tag image.

[0121] Optionally, if the text information is text related to the target object, after determining that the quadrilateral image is the target price tag image, the method further includes: saving the text information and the coordinates corresponding to the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera.

[0122] Optionally, determining multiple object frames from the target image includes: using a variable-magnification ball camera to enlarge and scan an image corresponding to the target object placement area, wherein the area between two price tag columns in the target image is determined as the target object placement area; detecting the image corresponding to the target object placement area through a target object detection model, and determining multiple object frames.

[0123] Optionally, before determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, the method further includes: extracting first text information and first color texture information from the target object frame; extracting multiple second text information and multiple second color texture information from the multiple comparison object frames; determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames includes: sequentially comparing the multiple second text information with the first text information; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type If the first text information does not match, the second color texture information is compared with the first color texture information in turn; if the second color texture information matches the first color texture information, it is determined that the object corresponding to the comparison object box and the target object are the same type of objects; if any one of the second text information and the second color texture information in the comparison object box is empty, the comparison object box is determined to be an empty object box; according to the type difference between the object corresponding to the comparison object box and the target object, at least one of the placement quantity information of the target object, the placement position information of the target object and the position misplacement information of the target object is determined; according to the empty object box, the vacant position information of the target object is determined. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0124] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: processing a target image containing a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; determining multiple object frames from the target image, and determining a target object frame based on the multiple object frames and the relative position relationship between the multiple object frames and the target price tag image, wherein an area corresponding to an object frame is used to place a target object, and the target object frame is an object frame in the multiple object frames; determining object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, wherein object frames other than the target object frame in the multiple object frames are determined as multiple comparison object frames.

[0125] Optionally, the object placement information includes at least one of the following: placement quantity information of the target objects; placement position information of the target objects; vacant position information of the target objects; and misplacement information of the target objects.

[0126] Optionally, before processing the target image containing the target object and determining the target price tag image, the method includes: calibrating the shelf area and obtaining the target image corresponding to the calibrated shelf area; performing edge extraction on the target image; performing straight line detection on the extracted edges to obtain multiple straight lines; identifying the multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag; and identifying the price tag column through the upper and lower boundaries.

[0127] Optionally, identifying multiple straight lines and determining the upper and lower boundaries of the price tag column corresponding to the target price tag includes: counting multiple slopes corresponding to the multiple straight lines, and eliminating straight lines with absolute values ​​of slopes greater than or equal to 1 to determine multiple first target straight lines; calculating the slope differences between any two of the multiple first target straight lines; eliminating invalid straight lines to obtain multiple second target straight lines, where invalid straight lines are straight lines with slope differences greater than a preset value; determining the distance values ​​between any two of the multiple second target straight lines; and respectively determining two straight lines corresponding to distance values ​​within a preset range as the upper and lower boundaries corresponding to the price tag column.

[0128] Optionally, the target image containing the target object is processed to determine the target price tag image, including: enlarging and scanning the image corresponding to the price tag column by a variable-magnification ball camera, and performing quadrilateral detection on the image to obtain a quadrilateral image; extracting text information in the quadrilateral image; if the text information is text related to the target object, determining that the quadrilateral image is the target price tag image.

[0129] Optionally, if the text information is text related to the target object, after determining that the quadrilateral image is the target price tag image, the method further includes: saving the text information and the coordinates corresponding to the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera.

[0130] Optionally, determining multiple object frames from the target image includes: using a variable-magnification ball camera to enlarge and scan an image corresponding to the target object placement area, wherein the area between two price tag columns in the target image is determined as the target object placement area; detecting the image corresponding to the target object placement area through a target object detection model, and determining multiple object frames.

[0131] Optionally, before determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames, the method further includes: extracting first text information and first color texture information from the target object frame; extracting multiple second text information and multiple second color texture information from the multiple comparison object frames; determining the object placement information of the target object based on the similarity between the target object frame and the multiple comparison object frames includes: sequentially comparing the multiple second text information with the first text information; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type; if the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are of the same type If the first text information does not match, the second color texture information is compared with the first color texture information in turn; if the second color texture information matches the first color texture information, it is determined that the object corresponding to the comparison object box and the target object are the same type of objects; if any one of the second text information and the second color texture information in the comparison object box is empty, the comparison object box is determined to be an empty object box; based on the type difference between the object corresponding to the comparison object box and the target object, at least one of the placement quantity information of the target object, the placement position information of the target object and the position misplacement information of the target object is determined; based on the empty object box, the vacant position information of the target object is determined.

[0132] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0136] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0137] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0138] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0139] It should also be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, object or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, object or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, object or device that includes the element.

[0140] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0141] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A method for determining placement information of a target object, characterized in that: include: Processing a target image including a target object to determine a target price tag image, wherein the target price tag image is a price tag image including the target object; Determine a plurality of object frames from the target image, and determine a target object frame according to the plurality of object frames and the relative positional relationship between the plurality of object frames and the target price tag image, wherein an area corresponding to one of the object frames is used to place one of the target objects, and the target object frame is one of the plurality of object frames; Determining object placement information of the target object based on similarities between the target object frame and a plurality of comparison object frames, wherein object frames other than the target object frame among the plurality of object frames are determined as a plurality of comparison object frames; Before processing the target image containing the target object to determine the target price tag image, the method includes: Calibrate the shelf area and obtain the target image corresponding to the calibrated shelf area; Performing edge extraction on the target image; Performing straight line detection on the extracted edges to obtain a plurality of straight lines; Identify a plurality of the straight lines, and determine an upper boundary and a lower boundary of the price tag column corresponding to the target price tag; The price label column is identified by the upper boundary and the lower boundary.

2. The method according to claim 1, characterized in that The object placement information includes at least one of the following: The placement quantity information of the target object; Placement location information of the target object; Vacant position information of the target object; The location misplacement information of the target object.

3. The method according to claim 1, characterized in that Identifying the plurality of straight lines and determining the upper boundary and the lower boundary of the price tag column corresponding to the target price tag includes: Counting multiple slopes corresponding to the multiple straight lines, and eliminating straight lines whose absolute values ​​of the slopes are greater than or equal to 1 to determine multiple first target straight lines; Calculating the slope differences between any two of the first target straight lines; Eliminate invalid straight lines to obtain a plurality of second target straight lines, wherein the invalid straight lines are straight lines corresponding to the slope differences being greater than a preset value; Determine the distance values ​​between any two of the plurality of second target straight lines; The two straight lines corresponding to the distance values ​​within the preset range are respectively determined as the upper boundary and the lower boundary corresponding to the price tag column.

4. The method according to claim 1, characterized in that Process the target image containing the target object to determine the target price tag image, including: The image corresponding to the price tag column is enlarged and scanned by a variable-magnification ball camera, and a quadrilateral detection is performed on the image to obtain a quadrilateral image; Extracting text information from the quadrilateral image; If the text information is text related to the target object, the quadrilateral image is determined to be the target price label image.

5. The method according to claim 4, characterized in that If the text information is text related to the target object, after determining that the quadrilateral image is the target price label image, the method further includes: The text information and the coordinates corresponding to the center point of the quadrilateral image in the absolute coordinate system of the zoom ball camera are saved.

6. The method according to claim 1, characterized in that Determining a plurality of object frames from the target image comprises: The image corresponding to the target object placement area is enlarged and scanned by using a variable-magnification dome camera, wherein the area between two price tag columns in the target image is determined as the target object placement area; The image corresponding to the target object placement area is detected by a target object detection model, and a plurality of the object frames are determined.

7. The method according to claim 1, characterized in that Before determining the object placement information of the target object based on the similarity between the target object frame and a plurality of comparison object frames, the method further includes: Extracting first text information and first color texture information from the target object frame; Extracting a plurality of second text information and a plurality of second color texture information from a plurality of the comparison object frames; Determining the object placement information of the target object based on the similarity between the target object frame and a plurality of comparison object frames includes: sequentially comparing the plurality of second text information with the first text information; If the second text information matches the first text information, determining that the object corresponding to the comparison object frame and the target object are the same type of objects; If the second text information does not match the first text information, sequentially comparing the second color texture information with the first color texture information; If the second color texture information matches the first color texture information, determining that the object corresponding to the comparison object frame and the target object are the same type of objects; If any one of the second text information and the second color texture information in the comparison object frame is empty, determining that the comparison object frame is an empty object frame; Determine at least one of placement quantity information of the target object, placement position information of the target object, and position misplacement information of the target object according to the type difference between the object corresponding to the comparison object frame and the target object; According to the empty object frame, the vacant position information of the target object is determined.

8. A device for determining placement information of a target object, characterized in that: include: A first determining unit is used to process a target image including a target object to determine a target price tag image, where the target price tag image is a price tag image including the target object; a second determining unit, configured to determine a plurality of object frames from the target image, and determine a target object frame according to the plurality of object frames and the relative positional relationship between the plurality of object frames and the target price tag image, wherein an area corresponding to one of the object frames is used to place one of the target objects, and the target object frame is one of the plurality of object frames; A third determining unit is used to determine the object placement information of the target object based on the similarity between the target object frame and a plurality of comparison object frames, wherein object frames other than the target object frame among the plurality of object frames are determined as the plurality of comparison object frames; The device is also used to calibrate the shelf area and obtain the target image corresponding to the calibrated shelf area; perform edge extraction on the target image; perform straight line detection on the extracted edges to obtain multiple straight lines; identify the multiple straight lines and determine the upper boundary and lower boundary of the price tag column corresponding to the target price tag; and identify the price tag column through the upper boundary and the lower boundary.

9. The device according to claim 8, characterized in that The object placement information includes at least one of the following: The placement quantity information of the target object; Placement location information of the target object; Vacant position information of the target object; The location misplacement information of the target object.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for determining placement information of a target object as described in any one of claims 1 to 7.

11. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes a method for determining placement information of a target object as described in any one of claims 1 to 7.

Images