Object inventory method and apparatus, medium, device and product

Abstract

Embodiments of the present specification provide an object inventory method, an object inventory device, a computer readable storage medium, an electronic device and a computer program product. The method comprises: performing target detection on an image about a target region to obtain a plurality of detection boxes about objects in the target region, and then needing to identify the category of the object corresponding to each detection box. In the embodiments of the present specification, the actual category corresponding to each detection box is determined based on a seed object of a known category. Further, according to the actual category corresponding to each detection box, the inventory data of the objects in the target region is determined, including: the number of categories of the objects contained in the target region, and / or the number of objects belonging to each category in the target region.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to an object inventory method, an object inventory device, a computer-readable storage medium, an electronic device, and a computer program product. Background Technology

[0002] In related technologies, inventory operations for objects (such as goods of various categories) are based on electronic tags or barcode tags. The goods are counted through unique tags. This solution has low inventory efficiency and cannot achieve automated processing. In addition, it requires high accuracy of manual data collection.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this specification, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this specification is to provide an object inventory method, object inventory apparatus, computer-readable storage medium, electronic device, and computer program product that can improve inventory efficiency to a certain extent.

[0005] Other features and advantages of this specification will become apparent from the following detailed description, or may be learned in part by practice of this specification.

[0006] According to one aspect of this specification, an object inventory method is provided, the method comprising: performing object detection on an image about a target region to obtain N detection boxes about objects, where N is a positive integer; determining the i-th actual category corresponding to the i-th detection box based on a seed object, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known; determining inventory data of the objects in the target region according to the actual categories corresponding to the N detection boxes respectively; wherein the inventory data of the objects in the target region includes: the number of categories of objects contained in the target region, and / or the number of objects in the target region belonging to each category respectively.

[0007] In an exemplary embodiment, based on the foregoing scheme, determining the i-th actual category corresponding to the i-th detection box based on the seed object includes: determining the i-th predicted category corresponding to the i-th detection box through a prediction model; determining whether the i-th predicted category belongs to a preset category group, wherein the preset category group contains all categories corresponding to the seed objects; if the i-th predicted category does not belong to the preset category group, obtaining the seed features corresponding to the seed object that is consistent with the (i-1)-th actual category, and matching the seed features with the image features corresponding to the i-th detection box to obtain a first matching result, where i is an integer greater than 1; and determining the i-th actual category corresponding to the i-th detection box based on the first matching result.

[0008] In an exemplary embodiment, based on the foregoing scheme, determining the i-th actual category corresponding to the i-th detection box according to the first matching result includes: if the first matching result is a successful match, determining the (i-1)-th actual category as the i-th actual category corresponding to the i-th detection box; if the first matching result is a failed match, obtaining the seed feature corresponding to the seed object that is consistent with the (i+1)-th actual category, and matching the seed feature with the image feature corresponding to the i-th detection box to obtain a second matching result; and determining the i-th actual category corresponding to the i-th detection box according to the second matching result.

[0009] In an exemplary embodiment, based on the aforementioned scheme, determining the i-th actual category corresponding to the i-th detection box according to the second matching result includes: if the second matching result is a successful match, determining the (i+1)-th actual category as the i-th actual category corresponding to the i-th detection box.

[0010] In an exemplary embodiment, based on the foregoing scheme, after determining whether the i-th predicted category belongs to the preset category group, the method further includes: if the i-th predicted category belongs to the preset category group, determining the i-th predicted category as the i-th actual category corresponding to the i-th detection box.

[0011] In an exemplary embodiment, based on the foregoing scheme, the step of matching the seed features with the image features corresponding to the i-th detection box to obtain a first matching result includes: matching the handcrafted features of the seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the seed object; matching the depthwise convolutional features of the seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the seed object; and determining the first matching result based on the first matching score and the second matching score between the i-th detection box and the seed object.

[0012] In an exemplary embodiment, based on the aforementioned scheme, determining the i-th actual category corresponding to the i-th detection box based on the seed object includes: determining the recognition classification score corresponding to the i-th detection box through a prediction model, where i is a positive integer not greater than N; obtaining the seed features corresponding to the seed object, and matching the seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box; and determining the i-th actual category corresponding to the i-th detection box based on the recognition classification score and the matching classification score corresponding to the i-th detection box.

[0013] In an exemplary embodiment, based on the aforementioned scheme, the seed objects comprise K objects, and the seed features corresponding to the j-th seed object include: handcrafted features of the j-th seed object and deep convolutional features of the j-th seed object, where K is a positive integer and j is a positive integer not greater than K.

[0014] The above-mentioned matching of seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box includes: matching the handcrafted features of the j-th seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the j-th seed object; matching the depthwise convolutional features of the j-th seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the j-th seed object; determining the matching classification score between the i-th detection box and the j-th seed object based on the first matching score and the second matching score; and determining the matching classification score corresponding to the i-th detection box based on the matching classification scores between the i-th detection box and the K seed objects.

[0015] In an exemplary embodiment, based on the foregoing scheme, after performing target detection on the image to obtain N detection boxes for the object, the method further includes: clustering the N detection boxes in a preset direction to obtain the number of layers in which the object is distributed in the preset direction, and the number of vacancies in the target layer and / or the number of objects in the target layer; determining the number of layers in which the object is distributed in the preset direction, and the number of vacancies in the target layer and / or the number of objects in the target layer as inventory data of the object in the target area.

[0016] In an exemplary embodiment, based on the foregoing scheme, after clustering the N detection boxes in the preset direction, the method further includes: determining M detection boxes belonging to the target layer from the N detection boxes according to the clustering results; determining the j-th actual category corresponding to the j-th detection box, where M is a positive integer and j is any integer between 1 and M, including 1 and M; determining the inventory data of the objects in the target layer according to the actual categories corresponding to the M detection boxes respectively; wherein the inventory data of the objects in the target layer includes: the number of categories of objects contained in the target layer, and / or the number of objects belonging to each category in the target layer.

[0017] According to another aspect of this specification, an object inventory device is provided, the device comprising: a detection frame determination module, a category determination module, and an object inventory module.

[0018] The aforementioned detection box determination module is used to perform target detection on the image of the target region to obtain N detection boxes for the object, where N is a positive integer; the aforementioned category determination module is used to: determine the i-th actual category corresponding to the i-th detection box based on the seed object, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known; and the aforementioned object inventory module is used to: determine the inventory data of the objects in the aforementioned target region according to the actual categories corresponding to the aforementioned N detection boxes respectively; wherein the inventory data of the objects in the aforementioned target region includes: the number of categories of objects contained in the aforementioned target region, and / or, the number of objects in the aforementioned target region belonging to each category respectively.

[0019] According to another aspect of this specification, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the object inventory method as described in the above embodiments.

[0020] According to one aspect of this specification, a computer-readable storage medium is provided that stores instructions which, when executed on a computer or processor, cause the computer or processor to perform the object inventory method as described in the above embodiments.

[0021] According to another aspect of this specification, a computer program product containing instructions is provided that, when the computer program product is run on a computer or processor, causes the computer or processor to perform the object inventory method as described in the above embodiments.

[0022] The object inventory method, object inventory apparatus, computer-readable storage medium, electronic device, and computer program product provided in the embodiments of this specification have the following technical effects:

[0023] In the exemplary embodiments provided in this specification, object detection is first performed on the image of the target region to obtain multiple detection boxes for objects in the target region. Then, the category of the object corresponding to each detection box needs to be identified. In the embodiments of this specification, the actual category corresponding to each detection box is determined based on seed objects of known categories, which helps to improve the accuracy of category determination. Further, based on the actual category corresponding to each detection box, the inventory data of objects in the target region is determined, including: the number of object categories contained in the target region, and / or, the number of objects in the target region belonging to each category. The object inventory scheme provided in the embodiments of this specification can realize automated inventory, and at the same time, since the accuracy of object category identification is guaranteed, it has high inventory efficiency.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification. It is obvious that the drawings described below are merely some embodiments of this specification, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 This is a schematic diagram of a store inventory scenario provided in one embodiment of this specification.

[0027] Figure 2 This is a flowchart illustrating an object inventory method provided in one embodiment of this specification.

[0028] Figure 3 This is a schematic flowchart illustrating a detection frame determination method provided in one embodiment of this specification.

[0029] Figure 4 This is a flowchart illustrating an embodiment of the method for determining the actual category provided in this specification.

[0030] Figure 5a This is a schematic diagram of multiple detection frames provided in one embodiment of this specification.

[0031] Figure 5b This is a schematic diagram of multiple detection frames and their respective categories provided in one embodiment of this specification.

[0032] Figure 6a This is a flowchart illustrating a method for determining the predicted category corresponding to a detection box, as provided in one embodiment of this specification.

[0033] Figure 6b This is a flowchart illustrating a method for determining the prediction category corresponding to a detection box, as provided in another embodiment of this specification.

[0034] Figure 7 This is a flowchart illustrating a matching method between seed features provided in one embodiment of this specification.

[0035] Figure 8 This is a flowchart illustrating a matching method between seed features provided in another embodiment of this specification.

[0036] Figure 9 This is a flowchart illustrating a method for determining the actual category, provided as another embodiment of this specification.

[0037] Figure 10 This is a flowchart illustrating an object inventory method provided in another embodiment of this specification.

[0038] Figure 11 This is a schematic diagram of a product inventory provided for one embodiment of this specification.

[0039] Figure 12 This is a schematic diagram of the structure of an object inventory device provided in one embodiment of this specification.

[0040] Figure 13 This is a schematic diagram of the structure of an object inventory device provided in another embodiment of this specification.

[0041] Figure 14 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this specification clearer, the embodiments of this specification will be described in further detail below with reference to the accompanying drawings.

[0043] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this specification more comprehensive and complete, and to fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of the embodiments described herein. However, those skilled in the art will recognize that the technical solutions described herein may be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., may be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this specification.

[0045] Furthermore, the accompanying drawings are merely illustrative diagrams of this specification and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0046] Another object inventory method provided by related technologies, when applied to the inventory of store goods, first involves a T+1 system inventory, and then a daily summary of product sales data based on the store's daily sales volume. However, this solution cannot achieve real-time performance and lacks responsiveness.

[0047] This specification provides an object inventory method, object inventory apparatus, computer-readable storage medium, electronic device, and computer program product, which can solve the problems existing in the solutions provided by related technologies. The solution provided by this specification is based on computer vision capabilities, providing end-to-end capabilities from visual input of object images to automated object inventory, and has real-time performance and high inventory efficiency.

[0048] In an exemplary embodiment, Figure 1 This is a schematic diagram of a store inventory scenario provided in one embodiment of this specification.

[0049] In stores (including unmanned retail stores and regular stores), there is a need for inventory checks. Timely inventory checks of existing stock and identification of items requiring replenishment enable timely replenishment. Furthermore, it allows for the identification of best-selling products, enabling analysis of consumer shopping habits, shopping paths, and user profiles. Therefore, the object inventory solution provided in this specification's embodiments can achieve both rapid, vision-based product entry processing and automated, periodic inventory checks of shelf stock. This empowers merchants to better manage their stores and maximize profits.

[0050] The object inventory scheme provided in the embodiments of this specification enables store inventory management. Users only need to take images of the corresponding area (e.g., shelves containing goods) using a mobile phone or other device with a camera function. The computing device then uses the images to perform product detection, recognition, and a series of post-processing modules to automatically output the current status of the goods on the shelves, thereby automatically performing inventory management and outputting information such as out-of-stock reminders.

[0051] In an exemplary embodiment, Figure 2 This is a flowchart illustrating an embodiment of the object inventory method provided in this specification, as shown below. Figure 2 As shown in the figure, the embodiment includes: S210-S230.

[0052] In S210, target detection is performed on the image of the target region to obtain N detection boxes about the object, where N is a positive integer.

[0053] In an exemplary embodiment, reference is made to Figure 3 Using a mobile phone or other device with a camera to take images of the corresponding area (such as a shelf containing goods), it is understandable that the images should be taken of the shelf as much as possible to improve image quality and thus improve recognition accuracy.

[0054] In an exemplary embodiment, an image can be captured by a camera such as a mobile phone, and then detection and recognition steps can be performed. The ability to stitch together images from multiple perspectives or angles can also be added, enabling more areas to be automatically inventoried at once, thereby improving the efficiency of object inventory.

[0055] In an exemplary embodiment, the acquired image of the target region is input into a pre-trained object detection model to determine the detection boxes contained in the image. (See reference...) Figure 3The pre-trained object detection model mentioned above can be EfficientNet. During the construction of EfficientNet, NAS (Neural Architecture Search) technology is used, which can simultaneously consider multiple influencing factors such as the resolution of the input image, the depth of the network, and the width of the network. Therefore, the resulting EfficientNet network structure can acquire the latent fine-grained features of the input image. In this embodiment, this is beneficial for accurately obtaining the detection box corresponding to each object.

[0056] In S220, based on the seed object, the i-th actual category corresponding to the i-th detection box is determined, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known.

[0057] In exemplary embodiments, relying solely on machine learning models for prediction cannot effectively guarantee the accuracy of product type identification. However, the solution provided in this specification determines the actual category corresponding to each detection box based on seed objects of known categories, which helps improve the accuracy of category determination. The embodiment of determining the actual category corresponding to each detection box based on seed objects of known categories will be described in detail in the following embodiments.

[0058] In S230, the inventory data of the objects in the target area are determined according to the actual categories corresponding to the N detection boxes.

[0059] In the case of the actual category corresponding to each detection box, the inventory data for the objects within the target area can include: the number of categories of objects contained in the target area, and / or the number of objects in the target area belonging to each category. For example, if the above image is an image of shelf 1, then the number of product categories contained in shelf 1 when the image was taken, and the quantity of products belonging to each category, can be determined.

[0060] In an exemplary embodiment, Figure 4 The flowchart illustrating the actual category determination method provided in one embodiment of this specification can be considered as a specific implementation of S220. For example... Figure 4 As shown in the figure, the embodiment includes: S410-S4100.

[0061] In S410, the i-th predicted category corresponding to the i-th detection box is determined by the prediction model.

[0062] In an exemplary embodiment, reference is made to Figure 5aEight detection boxes related to the product are shown, which can be labeled a1, a2, a3, a4, b1, b2, b3, and b4 from left to right. In this embodiment, a prediction model is used to determine the predicted category of the object corresponding to each detection box.

[0063] For example, refer to Figure 6a Using prediction model 60, the predicted category corresponding to the first detection box "a1" from left to right is determined, for example, "Category A". Similarly, using prediction model 60, the predicted categories corresponding to the second detection box "a2" and the third detection box "a3" from left to right are determined, and they are also "Category A".

[0064] For example, refer to Figure 5a As can be seen, although the first four objects in the captured image share the same category from left to right, the orientation of the first three objects differs from that of the fourth object. Specifically, the first three objects are all facing forward, while the fourth object is facing backward. Therefore, the prediction model described above can predict that the first three objects belong to category A. However, the prediction model alone may not correctly predict the category of the fourth object, meaning it cannot accurately predict the category of the fourth object. For example, refer to... Figure 6b Using prediction model 60, the predicted category corresponding to the fourth detection box "a4" from left to right is determined, for example, "other".

[0065] To address the issue of incorrectly categorizing the fourth object mentioned above, this specification provides embodiments with constraints based on the consistency of adjacent goods, specifically:

[0066] In S420, it is determined whether the i-th predicted category belongs to the preset category group.

[0067] In an exemplary embodiment, the seed object is an object of a pre-known category. For example, in the process of inventorying beverages in a store, the seed object could be Sprite, Coca-Cola, or coconut juice. The preset category group contains all the categories corresponding to the seed objects. If the predicted category output by the preset model does not belong to the preset category group, for example, it does not belong to any of Sprite, Coca-Cola, or coconut juice, its actual category can be determined based on the constraint of consistency between adjacent products. Of course, if the i-th predicted category belongs to the preset category group, it indicates that the prediction accuracy of the prediction model is guaranteed, and no further verification is needed. Therefore, S4100 is executed: the i-th predicted category is determined as the i-th actual category corresponding to the i-th detection box.

[0068] In an exemplary embodiment, if the i-th predicted category does not belong to the aforementioned preset category group, it indicates that the category of the object corresponding to the i-th detection box needs further identification, and S430-S490 are executed. Specifically, S430 provides a method to determine the category of the product corresponding to the i-th detection box based on the consistency between adjacent objects on the first side (e.g., the left side) of the i-th detection box. If this still cannot be determined, S460 provides a method to determine the category of the product corresponding to the i-th detection box based on the consistency between adjacent objects on the second side (e.g., the right side) of the i-th detection box.

[0069] It should be noted that if there are no adjacent objects on the first side (e.g., the left side) of the i-th detection box, i.e., the i-th detection box is the first object from the left, and if it is determined by S420 that its predicted category does not belong to the preset category group, then the method of matching the seed features of each seed object with the image features corresponding to the i-th detection box can be used to determine whether it belongs to the category of a certain seed object. Similarly, if there are no adjacent objects on the second side (e.g., the right side) of the i-th detection box, i.e., the i-th detection box is the first object from the right, and if it is determined by S420 that its predicted category does not belong to the preset category group, then the method of matching the seed features of each seed object with the image features corresponding to the i-th detection box can be used to determine whether it belongs to the category of a certain seed object.

[0070] In S430, the seed features corresponding to the seed object that is consistent with the (i-1)th actual category are obtained, and the above seed features are matched with the image features corresponding to the i-th detection box to obtain the first matching result.

[0071] In an exemplary embodiment, Figure 7 This is a flowchart illustrating a matching method between seed features provided in one embodiment of this specification, which can be considered a specific implementation of S430. For example... Figure 7 As shown in the figure, the embodiment includes: S4302-S4306.

[0072] In S4302, the handcrafted features of the seed object corresponding to the (i-1)th detection box are matched with the handcrafted features corresponding to the ith detection box to obtain a first matching score between the ith detection box and the seed object. In S4304, the depthwise convolutional features of the seed object are matched with the depthwise convolutional features corresponding to the ith detection box to obtain a second matching score between the ith detection box and the seed object.

[0073] For example, to ensure comprehensive matching features and improve matching accuracy, which in turn improves object category recognition accuracy, each seed object's seed features should include at least two aspects. (See reference) Figure 8The seed feature 810 corresponding to the seed object that matches the (i-1)th actual category includes two aspects: handcrafted features and deep convolutional features. Correspondingly, the image feature 820 corresponding to the i-th detection box also includes two aspects: handcrafted features and deep convolutional features.

[0074] Among them, handcrafted features include low-level features and intermediate features. Low-level features mainly refer to features derived from the image itself through simple operations. These include color, texture, shape, and gradient, which are also the foundational features for intermediate feature operations. Well-known low-level features in the image and video field include RGB, LAB, HSV, HOG, LBP, SIFT, SURF, and SILTP (an improved version of LBP), etc. Mid-level features are obtained by multi-feature fusion processing based on low-level features, mainly involving the fusion of diverse colors and textures. For example, LOMO (HSV+SILTP). Mid-level features also refer to features that improve and innovate feature extraction methods, often incorporating more complex data processing theories. Intermediate features are reflected in the extraction strategy and fusion process. Deep convolutional features typically involve designing neural network models to mine deeper and more abstract features of the image. They do not require manual intervention and are less affected by lighting, pose, etc.

[0075] refer to Figure 7 and Figure 8 In S4302, manual feature comparison and matching are performed to obtain a first matching score of 830. In S4304, deep convolutional feature comparison and matching are performed to obtain a second matching score of 840. Further, in S4306, the first matching result 800 is determined based on the first matching score 830 and the second matching score 840 between the i-th detection box and the seed object.

[0076] Understandably, the weights of the first matching score and the second matching score can be set according to actual needs. The matching result obtained by combining the above two aspects can comprehensively reflect the degree of matching between the product corresponding to the i-th detection box and its first (left) adjacent product, which is conducive to improving the matching accuracy and thus improving the accuracy of object category recognition.

[0077] Continue to refer to Figure 4 In S440, is the first matching result a successful match?

[0078] If the first matching result is a successful match, it means that the type of the product corresponding to the i-th detection box is consistent with the type of the product on its first side (left side). It also means that the constraint of the consistency of adjacent products provided by the embodiment of this specification can guarantee the accuracy of the category identification of the i-th detection box. Then, S450 is executed: determine the i-1-th actual category as the i-th actual category corresponding to the i-th detection box.

[0079] For example, refer to Figure 5b The image features corresponding to detection box a4 are matched with the seed features of the seed object (i.e., the beverage "Sprite") corresponding to its first side (left side) detection box a3. A successful first match is obtained, thus determining that the actual category corresponding to detection box a4 is the same as the actual category corresponding to detection box a3, both belonging to category A. This embodiment demonstrates that the type of product corresponding to detection box a4 is consistent with the type of product corresponding to detection box a3, and also shows that the constraint of adjacent product consistency provided in this embodiment can guarantee the accuracy of category recognition for detection box a4.

[0080] If the first matching result fails, it means that the type of the product corresponding to the i-th detection box is inconsistent with the type of the product on its first side (left side). Then, S460 is executed: obtain the seed feature corresponding to the seed object that is consistent with the (i+1)-th actual category, and match the seed feature with the image feature corresponding to the i-th detection box to obtain the second matching result.

[0081] For example, refer again Figure 5a If detection boxes b1-b4 all belong to the same category, but detection box b1 is an occluded item in the image, then the prediction model can predict that the latter three (b2-b4) objects belong to category B. However, the prediction result for detection box b1 using only the above prediction model may not belong to category B, that is, the category corresponding to detection box b1 cannot be correctly predicted. As mentioned above, it is necessary to determine its corresponding category according to the adjacent item consistency constraint mentioned in the embodiments of this specification. Specifically: refer to Figure 5b If the image features corresponding to detection box b1 are matched with the seed features of the seed object (i.e., the beverage "Sprite") corresponding to its first side (left side) detection box a4, and the first matching result is a failure, it is determined that the actual category corresponding to detection box b1 is not the actual category corresponding to detection box a4. Then, S460 needs to be executed, that is, to match the image features corresponding to detection box b1 with the seed features of the seed object (i.e., the beverage "Coca-Cola") corresponding to its second side (right side) detection box b2.

[0082] The specific implementation of S460 is similar to that of S430, and will not be described again here. After the matching process of S460, a second matching result is obtained. Execute S470: Is the second matching result a successful match?

[0083] If the second matching result is a successful match, it means that the type of the product corresponding to the i-th detection box is consistent with the type of the product on its second side (right side). It also means that the constraint of the consistency of adjacent products provided by the embodiment of this specification can guarantee the accuracy of the category identification of the i-th detection box. Then, S480 is executed: determine the i+1-th actual category as the i-th actual category corresponding to the i-th detection box.

[0084] For example, refer to again Figure 5b The image features corresponding to detection box b1 are matched with the seed features of the seed object (i.e., the beverage "Coca-Cola") corresponding to its second (right) detection box b2. A second matching result is obtained, indicating success. Therefore, the actual category corresponding to detection box b1 is determined to be the same as the actual category corresponding to detection box b2, both belonging to category B. This embodiment illustrates that the type of product corresponding to detection box b1 is consistent with the type of product corresponding to detection box b2, and also demonstrates that the constraint of adjacent product consistency provided in this embodiment can guarantee the accuracy of category identification for detection box b1.

[0085] If the second matching result fails, it indicates that the type of the product corresponding to the i-th detection box is inconsistent with the product types on both sides (left and right). In this case, S490 is executed: a reminder message is generated to indicate that the actual category of the object corresponding to the i-th detection box cannot be identified. It is understood that in this situation, the user can determine the product category through other methods, such as based on product labels. Therefore, the method of fusing product labels and visual information in this embodiment can improve the accuracy of the overall solution.

[0086] In an exemplary embodiment, Figure 9 The flowchart illustrating the actual category determination method provided in another embodiment of this specification can be used as another specific implementation of S220.

[0087] It should be noted that, as Figure 4 In the actual category determination scheme shown, if the prediction result based on the machine learning model is not ideal (the predicted category does not belong to the preset type group), the category is then determined based on the consistency of adjacent items (e.g., Figure 5a If the recognition result of detection box a3 is "Type A" Sprite, which belongs to the preset type group, then there is no need to further identify the actual category of the object based on the consistency of adjacent products. Therefore, it has the technical effect of high judgment efficiency. Figure 9 In the actual category determination scheme shown, the recognition result for each detection box comes from two parts: one part is the prediction score of the machine learning model, and the other part is the matching score after matching the seed features corresponding to each seed. Category recognition is performed based on the two scores, which has a high accuracy effect.

[0088] For details, please refer to Figure 9 The embodiment shown in the figure includes: S910-S930.

[0089] In S910, the recognition classification score corresponding to the i-th detection box is determined by the prediction model.

[0090] In S920, the seed features corresponding to the seed object are obtained, and the seed features are matched with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box.

[0091] In an exemplary embodiment, if there are K seed objects, the seed features corresponding to the j-th seed object include: handcrafted features of the j-th seed object and deep convolutional features of the j-th seed object, where K is a positive integer and j is a positive integer not greater than K.

[0092] In this embodiment, the image feature 820 corresponding to the i-th detection box is matched with the seed feature corresponding to each seed. For example, as shown... Figure 8 The diagram shows that each seed feature includes both handcrafted features and deep convolutional features. Specifically, the process of determining the matching classification score between the i-th detection box and the j-th seed object is as follows:

[0093] The handcrafted features of the j-th seed object are matched with the handcrafted features corresponding to the ith detection box to obtain a first matching score between the ith detection box and the j-th seed object; the deep convolutional features of the j-th seed object are matched with the deep convolutional features corresponding to the ith detection box to obtain a second matching score between the ith detection box and the j-th seed object; based on the first matching score and the second matching score between the ith detection box and the j-th seed object, the matching classification score between the ith detection box and the j-th seed object is determined.

[0094] Understandably, the weights of the first matching score and the second matching score can be set according to actual needs. The matching classification score obtained by combining the above two aspects can comprehensively reflect the degree of matching between the product corresponding to the i-th detection box and its j-th seed product (j-th seed object), which is conducive to improving the matching accuracy and thus improving the accuracy of object category recognition.

[0095] And so on, based on the matching classification scores between the i-th detection box and each of the K seed objects, for example, represented as: score1, score2, score3, score4... score k From this, determine the maximum value, score. max For example, score maxThe score3 indicates that the type of the object corresponding to the i-th detection box has the highest similarity to the third seed object. Therefore, score3 is determined as the matching classification score corresponding to the i-th detection box.

[0096] Furthermore, in S930, the i-th actual category corresponding to the i-th detection box is determined based on the recognition classification score corresponding to the i-th detection box and the matching classification score corresponding to the i-th detection box.

[0097] It is understandable that the weights of the recognition classification score and the matching classification score can be set according to actual needs, and this manual does not impose any restrictions on this.

[0098] As mentioned above, Figure 9 In the actual category determination scheme shown, the recognition result for each detection box comes from two parts: one part is the prediction score of the machine learning model, and the other part is the matching score after matching the seed features corresponding to each seed. Category recognition is performed based on the two scores, which has a high accuracy effect.

[0099] In an exemplary embodiment, Figure 10 This is a flowchart illustrating an object inventory method provided in another embodiment of this specification, such as... Figure 10 As shown in the figure, the embodiment includes: S1010-S1040.

[0100] In S1010, the above N detection boxes are clustered in a preset direction to obtain the number of layers in which the objects are distributed in the preset direction, as well as the number of vacancies contained in the target layer and / or the number of objects contained in the target layer.

[0101] In an exemplary embodiment, the N detection boxes obtained in S210 are clustered in the horizontal direction (shelves typically display goods in layers) to obtain the number of layers in which the objects are distributed in the horizontal direction, as well as the number of vacancies contained in the target layer and / or the number of objects contained in the target layer. (See reference...) Figure 11 Based on the inventory data determined in this embodiment: after clustering the detection boxes horizontally, the number of shelves with goods is found to be 4. The clustering results of the detection boxes in each shelf determine that the empty spaces are "empty," meaning the goods are sold out and need to be restocked. For example, Figure 11 The system has no empty spaces in the first layer, one empty space in the second layer, one empty space in the third layer, and three empty spaces in the fourth layer. This allows the system to accurately and in real-time notify users of the layers that need restocking. More specifically, it can also inform users of the exact restocking location based on the location of the empty spaces.

[0102] It is understood that the above-mentioned preset direction is not limited to the horizontal direction, but can also be the vertical direction or a pre-set direction, and this manual does not limit it.

[0103] In S1020, based on the clustering results, M detection boxes belonging to the target layer are determined from the aforementioned N detection boxes. In S1030, the j-th actual category corresponding to the j-th detection box is determined, where M is a positive integer and j is any integer between 1 and M, inclusive. And, in S1040, based on the actual categories corresponding to the aforementioned M detection boxes, the inventory data of the objects in the target layer is determined.

[0104] For a detailed implementation of determining the j-th actual category corresponding to the j-th detection box, please refer to [reference needed]. Figure 4 or Figure 9 The embodiments shown will not be described in detail here.

[0105] After determining the actual category corresponding to each detection box in each layer, the inventory data that can be determined according to this embodiment is: the number of object categories contained in each layer, and / or the number of objects belonging to each category in each layer.

[0106] For example, in an embodiment of store inventory management, the inventory data demo can be output in real time, including two parts. The first part is the automatic inventory output of shelf goods, including: number of shelves, total number of goods, number of goods types, out-of-stock areas, and detailed product statistics for each row. The second part is the statistics for specified goods, which can calculate the quantity and corresponding placement of the goods that the store wants to count in real time.

[0107] As can be seen, the overall framework of the embodiments in this specification includes multiple computer vision technologies to build a complete automated inventory management solution for shelf merchandise, which can efficiently and in real time count shelf merchandise.

[0108] It should be noted that the above figures are merely illustrative of the processes included in the methods according to exemplary embodiments of this specification, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may, for example, be executed synchronously or asynchronously in multiple modules.

[0109] The following are embodiments of the apparatus described in this specification, which can be used to execute the embodiments of the methods described in this specification. For details not disclosed in the apparatus embodiments of this specification, please refer to the embodiments of the methods described in this specification.

[0110] in, Figure 12 This is a schematic diagram of the structure of an object inventory device provided in one embodiment of this specification. Please refer to [link / reference]. Figure 12The object inventory device shown in the figure can be implemented as all or part of an electronic device through software, hardware, or a combination of both. It can also be integrated as an independent module on a server or as an independent module in an electronic device.

[0111] The object inventory device 1200 described in the embodiments of this specification includes: a detection frame determination module 1210, a category determination module 1220, and an object inventory module 1230.

[0112] The detection box determination module 1210 is used to perform target detection on the image of the target region to obtain N detection boxes about the object, where N is a positive integer; the category determination module 1220 is used to: determine the i-th actual category corresponding to the i-th detection box based on the seed object, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known; and the object inventory module 1230 is used to: determine the inventory data of the objects in the target region according to the actual categories corresponding to the N detection boxes respectively; wherein the inventory data of the objects in the target region includes: the number of categories of objects contained in the target region, and / or the number of objects in the target region belonging to each category respectively.

[0113] In an exemplary embodiment, Figure 13 This is a schematic diagram of an object inventory device provided in another embodiment of this specification. Please refer to... Figure 13 :

[0114] In an exemplary embodiment, based on the foregoing scheme, the category determination module 1220 includes: a prediction unit 12202, a determination unit 12204, and a first matching unit 12206.

[0115] The prediction unit 12202 is used to: determine the i-th predicted category corresponding to the i-th detection box through the prediction model; the determination unit 12204 is used to: determine whether the i-th predicted category belongs to a preset category group, wherein the preset category group contains all the categories corresponding to the seed objects; the first matching unit 12206 is used to: when the i-th predicted category does not belong to the preset category group, obtain the seed features corresponding to the seed object that is consistent with the (i-1)-th actual category, and match the seed features with the image features corresponding to the i-th detection box to obtain a first matching result, where i is an integer greater than 1; the determination unit 12204 is also used to: determine the i-th actual category corresponding to the i-th detection box based on the first matching result.

[0116] In an exemplary embodiment, based on the foregoing scheme, the determining unit 12204 is further configured to: determine the (i-1)th actual category as the i-th actual category corresponding to the i-th detection box when the first matching result is a successful match; the first matching unit 12206 is further configured to: obtain the seed feature corresponding to the seed object that is consistent with the (i+1)th actual category when the first matching result is a failed match, and match the seed feature with the image feature corresponding to the i-th detection box to obtain a second matching result; the determining unit 12204 is further configured to: determine the i-th actual category corresponding to the i-th detection box according to the second matching result.

[0117] In an exemplary embodiment, based on the foregoing scheme, the determining unit 12204 is further configured to: determine the (i+1)th actual category as the i-th actual category corresponding to the i-th detection box when the second matching result is a successful match.

[0118] In an exemplary embodiment, based on the foregoing scheme, the determining unit 12204 is further configured to: determine the i-th predicted category as the i-th actual category corresponding to the i-th detection box when the i-th predicted category belongs to the preset category group.

[0119] In an exemplary embodiment, based on the foregoing scheme, the first matching unit 12206 is specifically configured to: match the handcrafted features of the seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the seed object; match the depthwise convolutional features of the seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the seed object; and determine a first matching result based on the first matching score and the second matching score between the i-th detection box and the seed object.

[0120] In an exemplary embodiment, based on the foregoing scheme, the category determination module 1220 further includes a second matching unit 12208.

[0121] Based on the seed object, the prediction unit 12202 is further configured to: determine the recognition classification score corresponding to the i-th detection box through the prediction model, where i is a positive integer not greater than N; the second matching unit 12208 is configured to: obtain the seed features corresponding to the seed object, and match the seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box; and the determining unit 12204 is further configured to: determine the i-th actual category corresponding to the i-th detection box based on the recognition classification score and the matching classification score corresponding to the i-th detection box.

[0122] In an exemplary embodiment, based on the aforementioned scheme, the seed objects comprise K objects, and the seed features corresponding to the j-th seed object include: handcrafted features of the j-th seed object and deep convolutional features of the j-th seed object, where K is a positive integer and j is a positive integer not greater than K.

[0123] The second matching unit 12208 is specifically used for: matching the handcrafted features of the j-th seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the j-th seed object; matching the depthwise convolutional features of the j-th seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the j-th seed object; determining the matching classification score between the i-th detection box and the j-th seed object based on the first and second matching scores; and determining the matching classification score corresponding to the i-th detection box based on the matching classification scores between the i-th detection box and the K seed objects.

[0124] In an exemplary embodiment, based on the foregoing scheme, the above-mentioned device further includes a clustering module 1240.

[0125] The clustering module 1240 is used to: after the detection box determination module 1210 performs target detection on the image and obtains N detection boxes about the object, cluster the N detection boxes in a preset direction to obtain the number of layers in which the object is distributed in the preset direction, and the number of vacancies in the target layer and / or the number of objects in the target layer; the object inventory module 1230 is specifically used to: determine the number of layers in which the object is distributed in the preset direction, and the number of vacancies in the target layer and / or the number of objects in the target layer as inventory data of the object in the target area.

[0126] In an exemplary embodiment, based on the foregoing scheme, the object inventory module 1230 is further specifically configured to: after the clustering module 1240 clusters the N detection boxes in a preset direction, determine, based on the clustering results, M detection boxes belonging to the target layer from the N detection boxes; determine the j-th actual category corresponding to the j-th detection box, where M is a positive integer and j is any integer between 1 and M, including 1 and M; and determine the inventory data of the objects in the target layer based on the actual categories corresponding to the M detection boxes respectively; wherein, the inventory data of the objects in the target layer includes: the number of categories of objects contained in the target layer, and / or, the number of objects belonging to each category in the target layer.

[0127] It should be noted that the object inventory device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the object inventory method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0128] Furthermore, the object inventory apparatus and object inventory method embodiments provided in the above embodiments belong to the same concept. Therefore, for details not disclosed in the apparatus embodiments of this specification, please refer to the object inventory method embodiments described above, which will not be repeated here.

[0129] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0130] This specification also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0131] Figure 14 This is a schematic diagram of the electronic device provided in the embodiments of this specification. Please refer to... Figure 14 As shown, the electronic device 1400 includes a processor 1401 and a memory 1402.

[0132] In this embodiment, processor 1401 is the control center of the computer system and can be a processor of a physical machine or a processor of a virtual machine. Processor 1401 may include one or more processing cores, such as a 4-core processor or a 9-core processor. Processor 1401 can be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 901 may also include a main processor and a coprocessor; the main processor is used to process data in the wake-up state, and the coprocessor is a low-power processor used to process data in the standby state.

[0133] In the embodiments of this specification, the processor 1401 is specifically used for: performing target detection on an image of a target region to obtain N detection boxes for objects, where N is a positive integer; determining the i-th actual category corresponding to the i-th detection box based on a seed object, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known; determining inventory data of the objects in the target region according to the actual categories corresponding to the N detection boxes respectively; wherein, the inventory data of the objects in the target region includes: the number of categories of objects contained in the target region, and / or, the number of objects in the target region belonging to each category respectively.

[0134] Further, the above-mentioned determination of the i-th actual category corresponding to the i-th detection box based on the seed object includes: determining the i-th predicted category corresponding to the i-th detection box through a prediction model; determining whether the i-th predicted category belongs to a preset category group, wherein the preset category group contains all categories corresponding to the seed objects; if the i-th predicted category does not belong to the preset category group, obtaining the seed features corresponding to the seed object that is consistent with the (i-1)-th actual category, and matching the seed features with the image features corresponding to the i-th detection box to obtain a first matching result, where i is an integer greater than 1; and determining the i-th actual category corresponding to the i-th detection box based on the first matching result.

[0135] Further, determining the i-th actual category corresponding to the i-th detection box based on the first matching result includes: if the first matching result is a successful match, determining the (i-1)-th actual category as the i-th actual category corresponding to the i-th detection box; if the first matching result is a failed match, obtaining the seed feature corresponding to the seed object that is consistent with the (i+1)-th actual category, and matching the seed feature with the image feature corresponding to the i-th detection box to obtain a second matching result; and determining the i-th actual category corresponding to the i-th detection box based on the second matching result.

[0136] Furthermore, determining the i-th actual category corresponding to the i-th detection box based on the second matching result includes: if the second matching result is a successful match, determining the (i+1)-th actual category as the i-th actual category corresponding to the i-th detection box.

[0137] Furthermore, the processor 1301 is specifically configured to: after determining whether the i-th predicted category belongs to the preset category group, if the i-th predicted category belongs to the preset category group, determine the i-th predicted category as the i-th actual category corresponding to the i-th detection box.

[0138] Further, the above-mentioned matching of the seed features with the image features corresponding to the i-th detection box to obtain a first matching result includes: matching the handcrafted features of the seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the seed object; matching the depthwise convolutional features of the seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the seed object; and determining the first matching result based on the first matching score and the second matching score between the i-th detection box and the seed object.

[0139] Further, based on the seed object, determining the i-th actual category corresponding to the i-th detection box includes: determining the recognition classification score corresponding to the i-th detection box through a prediction model, where i is a positive integer not greater than N; obtaining the seed features corresponding to the seed object, and matching the seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box; and determining the i-th actual category corresponding to the i-th detection box based on the recognition classification score and the matching classification score corresponding to the i-th detection box.

[0140] Furthermore, the above seed objects include K, and the seed features corresponding to the j-th seed object include: the handcrafted features of the j-th seed object and the deep convolutional features of the j-th seed object. K takes the value of a positive integer, and j takes the value of a positive integer not greater than K.

[0141] The above-mentioned matching of seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box includes: matching the handcrafted features of the j-th seed object with the handcrafted features corresponding to the i-th detection box to obtain a first matching score between the i-th detection box and the j-th seed object; matching the depthwise convolutional features of the j-th seed object with the depthwise convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the j-th seed object; determining the matching classification score between the i-th detection box and the j-th seed object based on the first matching score and the second matching score; and determining the matching classification score corresponding to the i-th detection box based on the matching classification scores between the i-th detection box and the K seed objects.

[0142] Furthermore, the processor 1301 is specifically configured to: after performing target detection on the image to obtain N detection boxes for the object, cluster the N detection boxes in a preset direction to obtain the number of layers in which the object is distributed in the preset direction, and the number of vacancies contained in the target layer and / or the number of objects contained in the target layer; and determine the number of layers in which the object is distributed in the preset direction, and the number of vacancies contained in the target layer and / or the number of objects contained in the target layer as inventory data of the object in the target area.

[0143] Furthermore, the processor 1301 is specifically configured to: after clustering the N detection boxes in the preset direction, determine M detection boxes belonging to the target layer from the N detection boxes based on the clustering results; determine the j-th actual category corresponding to the j-th detection box, where M is a positive integer and j is any integer between 1 and M, including 1 and M; and determine the inventory data of the objects in the target layer based on the actual categories corresponding to the M detection boxes respectively; wherein the inventory data of the objects in the target layer includes: the number of categories of objects contained in the target layer, and / or the number of objects belonging to each category in the target layer.

[0144] Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this specification, the non-transitory computer-readable storage media in memory 1402 is used to store at least one instruction for execution by processor 1401 to implement the methods in the embodiments of this specification.

[0145] In some embodiments, the electronic device 1400 further includes a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a display screen 1404, a camera 1405, and an audio circuit 1406.

[0146] Peripheral interface 1403 can be used to connect at least one input / output (I / O) related peripheral device to processor 1401 and memory 1402. In some embodiments of this specification, processor 1401, memory 1402, and peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments of this specification, any one or two of processor 1401, memory 1402, and peripheral interface 1403 can be implemented on separate chips or circuit boards. This specification does not specifically limit the embodiments in this regard.

[0147] Display screen 1404 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1404 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1401 for processing. In this case, display screen 1404 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments of this specification, there may be one display screen 1404, which serves as the front panel of electronic device 1400; in other embodiments, there may be at least two display screens 1404, respectively disposed on different surfaces of electronic device 1400 or in a folded design; in still other embodiments, display screen 1404 may be a flexible display screen, disposed on a curved or folded surface of electronic device 1400. Furthermore, display screen 1404 may also be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. The display screen 1404 can be made of materials such as liquid crystal display (LCD) and organic light-emitting diode (OLED).

[0148] Camera 1405 is used to acquire images or videos. Optionally, camera 1405 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, virtual reality (VR) shooting, or other fusion shooting functions. In some embodiments of this specification, camera 1405 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0149] The audio circuit 1406 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 1401 for processing. For stereo sound acquisition or noise reduction purposes, there may be multiple microphones, each located in a different part of the electronic device 1400. The microphone may also be an array microphone or an omnidirectional microphone.

[0150] Power supply 1407 is used to supply power to various components in electronic device 1400. Power supply 1407 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 1407 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0151] The block diagrams of the electronic device shown in the embodiments of this specification do not constitute a limitation on the electronic device 1400. The electronic device 1400 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0152] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0153] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described object inventory device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0155] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0156] The above description is merely a specific embodiment of this specification, but the scope of protection of this specification is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this specification should be included within the scope of protection of this specification. Therefore, equivalent variations made in accordance with the claims of this specification are still within the scope of this specification.

Claims

1. An object inventory method, wherein, The method includes: Perform object detection on the image of the target region to obtain N detection boxes about the object, where N is a positive integer; Based on the seed object, determine the i-th actual category corresponding to the i-th detection box, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known. Based on the actual categories corresponding to the N detection boxes, determine the inventory data of the objects within the target area; The inventory data of the objects within the target area includes: the number of object categories contained in the target area, and / or the number of objects belonging to each category in the target area; The step of determining the i-th actual category corresponding to the i-th detection box based on the seed object includes: The prediction model is used to determine the i-th predicted category corresponding to the i-th detection box; Determine whether the i-th predicted category belongs to a preset category group, wherein the preset category group contains all the categories corresponding to the seed objects; If the i-th predicted category does not belong to the preset category group, obtain the seed feature corresponding to the seed object that is consistent with the (i-1)-th actual category, and match the seed feature with the image feature corresponding to the i-th detection box to obtain the first matching result, where i is an integer greater than 1; Based on the first matching result, determine the i-th actual category corresponding to the i-th detection box; The seed features corresponding to the seed object include the handcrafted features of the seed object and the depth convolutional features of the seed object; the image features corresponding to the i-th detection box include the handcrafted features of the i-th detection box and the depth convolutional features of the i-th detection box. The handcrafted features include low-level features extracted from the image itself and mid-level features obtained by multi-feature fusion processing based on the low-level features; the deep convolutional features are obtained by mining based on a neural network model.

2. The method according to claim 1, wherein, The step of determining the i-th actual category corresponding to the i-th detection box based on the first matching result includes: If the first matching result is a successful match, the (i-1)th actual category is determined to be the i-th actual category corresponding to the i-th detection box; If the first matching result is a failure, the seed feature corresponding to the seed object that is consistent with the (i+1)th actual category is obtained, and the seed feature is matched with the image feature corresponding to the i-th detection box to obtain the second matching result; Based on the second matching result, the i-th actual category corresponding to the i-th detection box is determined.

3. The method according to claim 2, wherein, The step of determining the i-th actual category corresponding to the i-th detection box based on the second matching result includes: If the second matching result is a successful match, the (i+1)th actual category is determined to be the i-th actual category corresponding to the i-th detection box.

4. The method according to any one of claims 1 to 3, wherein, After determining whether the i-th predicted category belongs to a preset category group, the method further includes: If the i-th predicted category belongs to the preset category group, the i-th predicted category is determined as the i-th actual category corresponding to the i-th detection box.

5. The method according to any one of claims 1 to 3, wherein, The step of matching the seed features with the image features corresponding to the i-th detection box to obtain a first matching result includes: The handcrafted features of the seed object are matched with the handcrafted features corresponding to the i-th detection box to obtain the first matching score between the i-th detection box and the seed object; The deep convolutional features of the seed object are matched with the deep convolutional features corresponding to the i-th detection box to obtain a second matching score between the i-th detection box and the seed object; The first matching result is determined based on the first matching score and the second matching score between the i-th detection box and the seed object.

6. The method according to claim 1, wherein, The step of determining the i-th actual category corresponding to the i-th detection box based on the seed object includes: The recognition and classification score corresponding to the i-th detection box is determined by the prediction model, where i is a positive integer not greater than N; Obtain the seed features corresponding to the seed object, and match the seed features with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box; Based on the recognition classification score corresponding to the i-th detection box and the matching classification score corresponding to the i-th detection box, the i-th actual category corresponding to the i-th detection box is determined.

7. The method according to claim 6, wherein, The seed objects comprise K objects. The seed features corresponding to the j-th seed object include: handcrafted features of the j-th seed object and deep convolutional features of the j-th seed object. K is a positive integer and j is a positive integer not greater than K. The seed features are matched with the image features corresponding to the i-th detection box to obtain the matching classification score corresponding to the i-th detection box, including: The handcrafted features of the j-th seed object are matched with the handcrafted features corresponding to the ith detection box to obtain the first matching score between the ith detection box and the j-th seed object; The deep convolutional features of the j-th seed object are matched with the deep convolutional features corresponding to the ith detection box to obtain a second matching score between the ith detection box and the j-th seed object; Based on the first matching score and the second matching score between the i-th detection box and the j-th seed object, determine the matching classification score between the i-th detection box and the j-th seed object; The matching classification score corresponding to the i-th detection box is determined based on the matching classification score between the i-th detection box and the K seed objects.

8. The method according to any one of claims 1 to 3 or any one of claims 6 to 7, wherein, After performing object detection on the image to obtain N detection boxes for the object, the method further includes: Cluster the N detection boxes in a preset direction to obtain the number of layers in which the object is distributed in the preset direction, as well as the number of vacancies in the target layer and / or the number of objects in the target layer. The number of layers in which the objects are distributed in the preset direction, and the number of vacancies and / or the number of objects contained in the target layer, are determined as the inventory data of the objects in the target area.

9. The method according to claim 8, wherein, After clustering the N detection boxes in a preset direction, the method further includes: Based on the clustering results, M detection boxes belonging to the target layer are determined from the N detection boxes; Determine the j-th actual category corresponding to the j-th detection box, where M is a positive integer and j is any integer between 1 and M, including 1 and M; Based on the actual categories corresponding to the M detection boxes, determine the inventory data of the objects in the target layer; The inventory data of the objects in the target layer includes: the number of object categories contained in the target layer, and / or the number of objects belonging to each category in the target layer.

10. An object inventory device, wherein, The device includes: The detection box determination module is used to perform target detection on the image about the target region and obtain N detection boxes about the object, where N is a positive integer. The category determination module is used to: determine the i-th actual category corresponding to the i-th detection box based on the seed object, where i takes the value of each integer between 1 and N, including 1 and N, and the category of the seed object is known; The object inventory module is used to: determine the inventory data of the objects in the target area according to the actual categories corresponding to the N detection boxes respectively; The inventory data of the objects within the target area includes: the number of object categories contained in the target area, and / or the number of objects belonging to each category in the target area; Specifically, the category determination module is used to: determine the i-th predicted category corresponding to the i-th detection box using a prediction model; determine whether the i-th predicted category belongs to a preset category group, wherein the preset category group contains all categories corresponding to the seed objects; if the i-th predicted category does not belong to the preset category group, obtain the seed features corresponding to the seed objects that are consistent with the (i-1)-th actual category, and match the seed features with the image features corresponding to the i-th detection box to obtain a first matching result, where i is an integer greater than 1; and determine the i-th actual category corresponding to the i-th detection box based on the first matching result. The seed features corresponding to the seed object include the handcrafted features of the seed object and the depth convolutional features of the seed object; the image features corresponding to the i-th detection box include the handcrafted features of the i-th detection box and the depth convolutional features of the i-th detection box. The handcrafted features include low-level features extracted from the image itself and mid-level features obtained by multi-feature fusion processing based on the low-level features; the deep convolutional features are obtained by mining based on a neural network model.

11. A computer-readable storage medium storing instructions, wherein, When the instructions are executed on a computer or processor, the computer or processor performs the object inventory method as described in any one of claims 1 to 9.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the object inventory method as described in any one of claims 1 to 9.

13. A computer program product containing instructions, wherein, When the computer program product is run on a computer or processor, it causes the computer or processor to perform the object inventory method as described in any one of claims 1 to 9.

Images