Shelf management method and system based on image recognition
Through the shelf management system based on image recognition, the problem of low shelf management accuracy caused by manual input is solved, automatic identification and supplementation is achieved, and the accuracy of product conditions and shelf management is improved.
Patent Information
- Application Number
- CN202410271183.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-11
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-03-11
AI Technical Summary
In the prior art, due to manual input of goods information and placement, only goods placement can be obtained by collecting shelf images, resulting in lower accuracy of goods on shelf conditions and lower accuracy of shelf management.
Using shelf management methods and systems based on image recognition, shelf image information is obtained by connecting to management cameras, multi-level image recognition, determine the identification information of shelves, goods and single items, perform product positioning and position marking, and identify abnormal goods and single items. , provide feedback to automatically remind or supplement.
It improves the accuracy of the product situation, enhances the accuracy of shelf management, avoids the low work efficiency and insufficient timeliness caused by manual viewing, and realizes automatic identification and supplementation.
Smart Images

Figure CN118052991B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of shelf management, and in particular to a shelf management method and system based on image recognition. Background Art
[0002] Shelf management is the effective organization and management of goods on the shelves to improve the efficiency of shelf space utilization.
[0003] At present, the existing shelf management is mostly performed manually, and the product information and placement of the shelf are manually input. At the same time, by collecting shelf images, most of the information can only obtain the placement of the goods on the shelf, resulting in low accuracy in obtaining out-of-stock goods on the shelf, resulting in low accuracy in shelf management.
[0004] In summary, the prior art has a technical problem that due to manual input of product information and placement status, only the placement status of the products on the shelf can be obtained by collecting shelf images, resulting in low accuracy in obtaining the status of the products on the shelf, which leads to low accuracy in shelf management. Summary of the invention
[0005] The purpose of this application is to provide a shelf management method and system based on image recognition, so as to solve the technical problem in the prior art that due to manual input of product information and placement status, only the placement status of the products on the shelf can be obtained by collecting shelf images, resulting in low accuracy in obtaining the status of the products on the shelf, resulting in low accuracy of shelf management.
[0006] In view of the above problems, the present application provides a shelf management method and system based on image recognition.
[0007] In a first aspect, the present application provides a shelf management method based on image recognition, which is implemented through a shelf management system based on image recognition, wherein the method includes: connecting a management camera to obtain collected shelf image information; performing multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information; locating products based on the shelf identification information to determine product shelf positioning information, and collecting product information on the shelves, and performing position identification marking on the product shelf positioning information; using the product identification information and the product shelf positioning information to determine abnormal product identification information, the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is provided based on the abnormal product identification information; using the single product placement identification information to perform single product feature identification with the shelf product information, determine abnormal single product identification information, and provide feedback.
[0008] In the second aspect, the present application also provides a shelf management system based on image recognition, which is used to execute the shelf management method based on image recognition as described in the first aspect, wherein the system includes: a shelf image information acquisition module, which is used to connect to a management camera to acquire collected shelf image information; a multi-level image recognition module, which is used to perform multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information; a product shelf positioning information determination module, which is used to locate products based on the shelf identification information and determine product placement information. The module collects the shelf positioning information, collects the product information on the shelves, and performs position identification marking on the product shelf positioning information; an abnormal product identification information determination module, which is used to use the product identification information and the product shelf positioning information to determine the abnormal product identification information, wherein the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is provided based on the abnormal product identification information; an abnormal single product identification information determination module, which is used to use the single product placement identification information and the shelf product information to perform single product feature identification, determine the abnormal single product identification information, and provide feedback.
[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:
[0010] By connecting to the management camera, the collected shelf image information is obtained; multi-level image recognition is performed on the shelf image information to determine the shelf identification information, the product identification information, and the single product placement identification information; the product is located based on the shelf identification information, the product shelf positioning information is determined, and the product information on the shelf is collected, and the position identification mark is performed on the product shelf positioning information; the abnormal product identification information is determined by using the product identification information and the product shelf positioning information, and the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is given based on the abnormal product identification information; the single product placement identification information is used to perform single product feature identification with the shelf product information, determine the abnormal single product identification information, and give feedback, that is, the shelf is automatically identified by collecting images, and then the existing placement problems or insufficient inventory are automatically reminded or supplemented, and the sales promotion personnel are reminded to manage, so as to avoid manual review resulting in low work efficiency and failure to ensure the timeliness and comprehensiveness of the work, and finally achieve the technical goal of improving the accuracy of obtaining product conditions and achieve the technical effect of improving the accuracy of shelf management.
[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0013] Figure 1 This is a flowchart of the shelf management method based on image recognition in this application;
[0014] Figure 2 This is a structural diagram of the shelf management system based on image recognition in this application.
[0015] Description of reference numerals:
[0016] Shelf image information acquisition module 11, multi-level image recognition module 12, product shelf positioning information determination module 13, abnormal product identification information determination module 14, abnormal single product identification information determination module 15. DETAILED DESCRIPTION
[0017] This application provides a shelf management method and system based on image recognition, which solves the technical problem in the prior art that due to manual input of product information and placement status, only the placement status of the products on the shelf can be obtained by collecting shelf images, resulting in low accuracy in obtaining the status of the products on the shelf, and thus low accuracy in shelf management. The technical goal of improving the accuracy of obtaining the status of the products is achieved, and the technical effect of improving the accuracy of shelf management is achieved.
[0018] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.
[0019] Embodiment 1
[0020] Please refer to the attached Figure 1 The present application provides a shelf management method based on image recognition, wherein the method is applied to a shelf management system based on image recognition, and the method specifically comprises the following steps:
[0021] Step 1: Connect to the management camera to obtain the collected shelf image information;
[0022] Specifically, the management camera is a camera that collects photos of the shelf to obtain shelf images. The management camera is connected to collect videos of the shelf, extract photo information in the video, and obtain shelf image information.
[0023] Step 2: Perform multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information;
[0024] Specifically, shelf recognition is performed on the shelf image information to obtain shelf recognition information, product recognition is performed on the shelf image information to obtain product recognition information, and single product placement recognition is performed on the shelf image information to obtain single product placement recognition information.
[0025] Step 3: Locate the goods based on the shelf identification information, determine the goods shelf positioning information, collect the goods on the shelf information, and mark the position of the goods shelf positioning information;
[0026] Specifically, the product is located based on the shelf identification information, the shelf location information with the product is obtained, and the product shelf location information is determined. The product information to be put on the shelf is collected, the location of the shelf to be put on the shelf is obtained, and the shelf location is marked to obtain the product shelf location information.
[0027] Step 4: using the product identification information and the product shelf location information, determining abnormal product identification information, wherein the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and providing feedback based on the abnormal product identification information;
[0028] Specifically, abnormal product information is obtained based on product identification information and product shelf location information, wherein the product information of which the product type does not match and the product quantity does not meet the preset requirements is obtained, and is determined as abnormal product identification information. Further, the product type does not match the type of the product to be put on the shelf and the product to be replenished on the shelf do not match. The product quantity does not meet the preset requirements for the number of products to be put on the shelf does not meet the number of products to be replenished on the shelf. Among them, the preset requirements are obtained by a technician in this field according to the actual situation. Further, the abnormal product identification information is extracted for feedback.
[0029] Step 5: Use the single product placement identification information and the shelf product information to identify single product features, determine abnormal single product identification information, and provide feedback.
[0030] Specifically, the characteristics of individual products are identified through the individual product placement identification information and the information of the goods on the shelves, wherein the difference between the placed goods and the goods to be put on the shelves is obtained, and it is determined as abnormal individual product identification information, and the abnormal individual product identification information with abnormalities is fed back. Furthermore, the placement of individual products is usually referred to as the SKU placement status. SKU is the smallest inventory unit, and its full name is Stock Keeping Unit, which is the basic unit for measuring inventory in and out, wherein it can be in units of pieces, boxes, pallets, etc. SKU is a necessary method for logistics management of large chain supermarket distribution centers (DC), and each product corresponds to a SKU number. For a product, when any of the attributes of the product such as brand, model, configuration, grade, color, packaging capacity, unit, production date, shelf life, use, price, origin, etc. is different from other products, it can be called a single product.
[0031] The image recognition-based shelf management method is applied to the image recognition-based shelf management system, and can automatically identify the shelf by collecting images, and then automatically identify the goods in the existing placement problems or insufficient inventory, and then automatically remind or replenish, avoiding manual checking and resulting in low work efficiency, and finally achieving the technical goal of improving the accuracy of obtaining the goods situation and achieving the technical effect of improving the accuracy of shelf management.
[0032] Furthermore, the present application also includes the following steps:
[0033] Connect to the settlement system to obtain commodity settlement data;
[0034] Generate single product retention verification data, where the single product retention verification data is determined by performing data quantification calculation based on the single product settlement data and the shelf product information in the product settlement data;
[0035] The placement is matched according to the single product placement identification information and the single product retention verification data, and when an abnormality exists, an abnormality reminder message is generated.
[0036] Specifically, connect to the settlement system, settle the quantity of listed goods, and obtain the goods settlement data.
[0037] Then, the single product settlement data in the commodity settlement data is obtained, that is, the identification result of a single commodity. The data quantification calculation is performed on the shelf commodity information, that is, the number of shelf commodities is calculated and obtained. The single product settlement data and the shelf commodity information are combined to determine the single product retention verification data, which is used to verify the commodities on the shelf.
[0038] Next, the placement is matched according to the single product placement identification information and the single product retention verification data. When the single product retention verification data does not match the single product placement identification information, there is an abnormality, indicating that the placement of goods on the shelf is abnormal, and then an abnormal reminder message is generated.
[0039] The settlement quantity and identification results can be used to determine whether goods have been placed elsewhere, thereby improving the accuracy of shelf management.
[0040] Furthermore, the present application also includes the following steps:
[0041] Based on the abnormal identification results, determine the abnormal single product image features;
[0042] Sending the abnormal single product image features to the management camera to activate an abnormal search mode, wherein the abnormal search mode is an automatic image recognition mode including the abnormal single product image features;
[0043] Based on the abnormal search mode, the image features of the abnormal single product are used to perform traversal identification and position positioning within the collection range, and the retained single product identification and positioning information is obtained for feedback.
[0044] Specifically, based on the abnormal recognition results of the mismatch between the item placement identification information and the item retention verification data, the mismatched items are extracted and determined as abnormal items, and the image features of the abnormal items are obtained and determined as abnormal item image features.
[0045] Then, the abnormal single product image features are sent to the management camera for abnormal search, and the abnormal search mode of the camera is activated for search through the camera. Among them, the abnormal search mode is an automatic image recognition mode containing the abnormal single product image features.
[0046] Then, the abnormal single product image features are identified and located in sequence within the acquisition range through the abnormal search mode to obtain the identification and location information of the retained single product, that is, the cargo position in the shelf. And feedback is given. Among them, the acquisition range is the range of the cargo position in a shelf.
[0047] The camera is used to identify the image features of abnormal items, thereby improving the accuracy of obtaining abnormalities.
[0048] Furthermore, the present application also includes the following steps:
[0049] Collect sample data sets of retained goods, analyze corresponding single product retention records based on single product retention verification data, determine retention area characteristics and retention probability, and set collaborative management cameras;
[0050] The collaborative management camera is determined based on the matching of the single product retention verification data, the image features of the abnormal single product are sent to the collaborative management camera, and the abnormal search mode of the collaborative management camera is activated.
[0051] Specifically, the retained items are obtained according to the abnormalities of the goods to be put on the shelves in the abnormal identification results and the abnormalities of the goods to be replenished on the shelves. According to the single product retention verification data, the retention records of the corresponding single product in the historical time are decomposed and extracted, and the historical retention area characteristics and historical retention probabilities are obtained, and the historical retention area characteristics and historical retention probabilities are determined as the retention area characteristics and retention probabilities of the retained single product. Further, a collaborative management camera is set for camera control. The camera is controlled according to the retention area characteristics. According to the retention probability corresponding to the retention area of the retention area characteristics, the camera is controlled to monitor the retention area, wherein the higher the retention probability, the higher the monitoring frequency of the camera, and vice versa. Then, a collaborative management camera is set. Then, the collaborative management camera is determined according to the matching of the single product retention verification data, and the abnormal single product image features are sent to the collaborative management camera, thereby activating the abnormal search mode of the collaborative management camera for abnormal search of abnormal single products.
[0052] The abnormality search mode is activated by controlling the camera, thereby improving the accuracy of abnormality search.
[0053] Furthermore, the present application also includes the following steps:
[0054] Constructing a multi-level image recognition model, the multi-level image recognition model includes a shelf recognition sub-model, a product recognition sub-model and a single product placement recognition sub-model, wherein the single product placement recognition sub-model includes a first channel and a second channel, the input image of the first channel is a grayscale image, and the input image of the second channel is a color image;
[0055] Extract key frames from the shelf image information, and perform image extraction according to a preset step length to obtain a shelf image extraction set;
[0056] The shelf image extraction set is input into the multi-level image recognition model for step-by-step processing and recognition, and the recognition results of each level are obtained and output to obtain the shelf recognition information, product recognition information, and single product placement recognition information.
[0057] Specifically, a network architecture of a multi-level image recognition model is constructed for image recognition training. The multi-level image recognition model includes a shelf recognition sub-model, a product recognition sub-model, and a single product placement recognition sub-model. Furthermore, the shelf recognition sub-model is used to recognize and extract shelves in shelf image information. The product recognition sub-model is used to recognize and extract products in shelf image information. The single product placement recognition sub-model is used to recognize and extract the placement of products in the shelf. Furthermore, the single product placement recognition sub-model includes a first channel and a second channel, the input image of the first channel is a grayscale image, and the input image of the second channel is a color image.
[0058] Then, the shelf image information is subjected to key frame extraction, wherein the photo information randomly obtained through the video is screened, key frames with shelf images are obtained, and then a shelf image extraction set is obtained. The preset step length is the time interval for obtaining the shelf image information, and the preset step length is obtained by a person skilled in the art by custom setting according to actual conditions. For example, the preset step length is 1 hour.
[0059] Next, the shelf image extraction set is input into the multi-level image recognition model for step-by-step processing and recognition. The shelf recognition information is obtained through the output of the shelf recognition sub-model, the product recognition information is obtained through the output of the product recognition sub-model, and the single product placement recognition information is obtained through the output of the single product placement recognition sub-model.
[0060] By performing multi-level image recognition, the shelf identification information, product identification information, and single product placement identification information are determined, thereby improving the accuracy of the image recognition results.
[0061] Furthermore, the present application also includes the following steps:
[0062] The shelf recognition sub-model includes a feature recognition layer and a shelf positioning layer, wherein the feature recognition layer is obtained by collecting shelf feature sets to construct training data for network model learning, and the shelf positioning layer is obtained by constructing a training data set using camera positioning, image acquisition parameters, image shooting range and corresponding positioning identifiers, and performing supervised learning. The shelf recognition sub-model is used to perform recognition results and shelf positioning on the shelf image extraction set to obtain the shelf recognition information;
[0063] Constructing a product core feature list, which includes a mapping relationship between products and core features, and the product identification sub-model traverses the shelf image extraction set based on the product core feature list to determine the product identification information;
[0064] The grayscale image converted from the shelf image extraction set is subjected to border recognition through the first channel to determine the border recognition information of the single product. The color image of the shelf image extraction set is subjected to color distribution recognition through the second channel to determine the color feature recognition information of the single product. The border recognition information of the single product and the color feature recognition information of the single product are combined and outputted through a fully connected layer.
[0065] Specifically, the shelf recognition sub-model includes a feature recognition layer and a shelf positioning layer. The feature recognition layer is used to identify and obtain shelf features, such as the distribution characteristics of the shelf. The shelf positioning layer is used to obtain the shelf position. Further, the structural information of the shelf is collected, the adjustability, load-bearing, and storage efficiency information of the shelf are obtained, and the shelf feature set is determined. According to the shelf feature set division, the shelf feature set training data and the shelf feature set verification data are obtained, and the feature recognition layer is trained by the shelf feature set training data. When the output data of the shelf feature set tends to be stable, the feature recognition layer is verified by the shelf feature set verification data. When the output accuracy of the feature recognition layer meets the output accuracy threshold of the feature recognition layer, the feature recognition layer is obtained. Among them, the output accuracy threshold of the feature recognition layer is obtained by a person skilled in the art according to the actual situation. For example, the output accuracy threshold of the feature recognition layer is 90%. Further, the shelf positioning information, that is, the preliminary fuzzy position of the shelf, is obtained through a camera. The image acquisition parameters, that is, the resolution information, are obtained. The image shooting range and the corresponding positioning mark are obtained, wherein the image shooting range is the proportion of the shelf in the image and the position of the shelf in the image, for example, the center position of the shelf in the image. Furthermore, the camera positioning, image acquisition parameters, image shooting range and corresponding positioning mark are used as input data of the shelf positioning layer. The input data of the shelf positioning layer is divided to obtain the training data of the shelf positioning layer and the verification data of the shelf positioning layer. The shelf positioning layer is trained by the training data of the shelf positioning layer. When the output data of the shelf positioning layer tends to be stable, the shelf positioning layer is verified by the verification data of the shelf positioning layer. When the output accuracy of the shelf positioning layer meets the output accuracy threshold of the shelf positioning layer, the shelf positioning layer is obtained. Among them, the output accuracy threshold of the shelf positioning layer is obtained by a person skilled in the art through custom settings according to actual conditions. For example, the output accuracy threshold of the shelf positioning layer is 90%. Furthermore, the shelf image extraction set is input into the shelf recognition sub-model, and the shelf feature information and shelf positioning information are obtained through recognition by the shelf recognition sub-model, and the shelf recognition information is obtained.
[0066] Next, the features of the product information set with the same attributes are obtained, and the maximum difference of the features of the feature products in the product information set with the same attributes is obtained. The features of the products within the maximum difference are obtained as the core features of the products. The core features of the products are mapped to each other to obtain the mapping relationship between the products and the core features, and then a list of core features of the products is constructed.
[0067] Next, in the product identification sub-model, the shelf image extraction set is matched with the products through the product core feature list to obtain the product matching results, and the product identification information is extracted from the product matching results. Among them, the product identification information is the product information with the number and basic information features of identifying the product, and the product identification information includes the number of product identifications and the basic information of the product.
[0068] Then, the grayscale image conversion is performed on the shelf image extraction set to obtain the grayscale image of the shelf image. The border of the goods is identified on the grayscale image through the first channel, that is, the border of the goods is obtained, and it is determined as the single product border identification information. The color distribution of the color image of the shelf image extraction set is identified through the second channel, that is, the color distribution of the goods is obtained, and it is determined as the single product color feature identification information. Further, the single product border identification information and the single product color feature identification information are merged and fused through the fully connected layer, and the identification result is output. Among them, the single product border identification information and the single product color feature identification information are weighted to obtain the fusion result as the single product placement identification information.
[0069] Through a multi-level image recognition model, shelf recognition information, product recognition information and single product placement recognition information are obtained, thereby improving the accuracy of recognition results at all levels.
[0070] Furthermore, the present application also includes the following steps:
[0071] Collecting information sets of goods with the same attributes, extracting placement strategies for the information sets of goods with the same attributes, and determining information sets of goods on the same shelf;
[0072] Performing abnormal identification on the information set of goods on the same shelf to determine the difference characteristics of the goods, wherein the difference characteristics of the goods include size difference, packaging shape difference, color difference, and standard image difference;
[0073] Setting a feature difference constraint degree based on the number of product information sets on the same shelf, and determining a core feature of the product difference feature based on the feature difference constraint degree, wherein the core feature is a feature with the largest feature difference of the product;
[0074] A mapping relationship between goods and core features is established, and a list of the core features of the goods is constructed.
[0075] Specifically, the product information set of the same attribute products on the shelf is collected, for example, the product information set includes clothes and shoes and bags. The historical placement records of the same attribute product information set are extracted, and the placement records are set as the placement strategy. The placement position information of each product on the shelf is obtained according to the placement strategy, and the product information set on the same shelf is determined.
[0076] Then, the placement strategies of the same attribute product information set and the same shelf product information set are compared to determine the product difference characteristics, wherein the product difference comparison includes size difference comparison, packaging shape difference comparison, color difference comparison, and standard image difference comparison. The product difference characteristics obtained include size difference, packaging shape difference, color difference, and standard image difference.
[0077] Next, the feature difference constraint is set based on the number of product information sets on the same shelf. The feature difference constraint is the difference threshold of each product. For example, the feature difference constraint includes the size difference threshold, the packaging form difference threshold, the color difference threshold, and the standard image difference threshold. The more the number of product information sets on the same shelf, the greater the feature difference constraint, and vice versa.
[0078] Next, the core features of the product difference features are determined based on the feature difference constraint degree, that is, the maximum difference of the product difference features is obtained, and then the feature difference constraint degree is verified to obtain the determined feature difference constraint degree. Among them, the core feature is the feature with the largest difference in product features.
[0079] Next, establish a mapping relationship between goods and core features. For example, the mapping relationship includes the size, packaging form, color, and standard image of the goods, and then build a list of core features of the goods.
[0080] By collecting information sets of goods with the same attributes, a list of core features of goods is constructed, thereby improving the accuracy of obtaining the list of core features of goods.
[0081] Furthermore, the present application also includes the following steps:
[0082] Clustering the goods in the concentrated images of the shelf image extraction, performing image segmentation according to the clustering results, and re-segmenting the image segmentation results to identify the objects to obtain a portrait segmentation result;
[0083] Traversing the image segmentation results based on the core features in the product core feature list to determine the product recognition results;
[0084] The same product identification results are aggregated to determine the product identification information, which includes the product identification quantity and the basic product information.
[0085] Specifically, the products in the shelf image extraction set are clustered, wherein the products are clustered according to their size, packaging form, color, and standard image as clustering features to obtain clustering results. The images are segmented according to the clustering results to obtain image segmentation results including the clustering results. The image segmentation results are re-segmented to identify the items, that is, the individual products are segmented to obtain image segmentation results.
[0086] Then, the image segmentation results are matched in sequence according to the core features in the product core feature list to obtain the product matching result of each core feature mapping, which is determined as the product recognition result.
[0087] Next, the same goods in the goods identification results are aggregated, and the goods identification information of the aggregated goods is extracted, wherein the goods identification information includes the goods identification quantity and the goods basic information. The goods identification quantity is the number of goods. The goods basic information is the size, packaging form and color information of the goods.
[0088] By obtaining the product identification results, product identification information is extracted, thereby improving the accuracy of the product identification information.
[0089] In summary, the shelf management method based on image recognition provided by this application has the following technical effects:
[0090] By connecting to the management camera, the collected shelf image information is obtained; multi-level image recognition is performed on the shelf image information to determine the shelf identification information, the product identification information, and the single product placement identification information; the product is located based on the shelf identification information to determine the product shelf positioning information, and the product information on the shelf is collected, and the position identification mark is performed on the product shelf positioning information; the abnormal product identification information is determined by using the product identification information and the product shelf positioning information, and the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is given based on the abnormal product identification information; the single product placement identification information is used to perform single product feature identification with the shelf product information, determine the abnormal single product identification information, and provide feedback, that is, the shelf is automatically identified by collecting images, and then the existing placement problems or insufficient inventory are automatically reminded or supplemented, and finally the technical goal of improving the accuracy of obtaining the product situation is achieved, and the technical effect of improving the accuracy of shelf management is achieved.
[0091] Embodiment 2
[0092] Based on the same inventive concept as the shelf management method based on image recognition in the aforementioned embodiment, the present application also provides a shelf management system based on image recognition, please refer to the attached Figure 2 , the system comprising:
[0093] The shelf image information acquisition module 11 is used to connect to the management camera to acquire the collected shelf image information;
[0094] A multi-level image recognition module 12, the multi-level image recognition module 12 is used to perform multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information;
[0095] A goods shelf positioning information determination module 13, the goods shelf positioning information determination module 13 is used to locate goods based on the shelf identification information, determine the goods shelf positioning information, collect goods on the shelf information, and perform position identification marking on the goods shelf positioning information;
[0096] An abnormal product identification information determination module 14, the abnormal product identification information determination module 14 is used to determine abnormal product identification information by using the product identification information and the product shelf location information, the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is provided based on the abnormal product identification information;
[0097] The abnormal single product identification information determination module 15 is used to use the single product placement identification information and the shelf commodity information to perform single product feature identification, determine the abnormal single product identification information, and provide feedback.
[0098] Furthermore, the abnormal single product identification information determination module 15 in the system is also used to:
[0099] Connect to the settlement system to obtain commodity settlement data;
[0100] Generate single product retention verification data, where the single product retention verification data is determined by performing data quantification calculation based on the single product settlement data and the shelf product information in the product settlement data;
[0101] The placement is matched according to the single product placement identification information and the single product retention verification data, and when an abnormality exists, an abnormality reminder message is generated.
[0102] Furthermore, the abnormal single product identification information determination module 15 in the system is also used to:
[0103] Based on the abnormal identification results, determine the abnormal single product image features;
[0104] Sending the abnormal single product image features to the management camera to activate an abnormal search mode, wherein the abnormal search mode is an automatic image recognition mode including the abnormal single product image features;
[0105] Based on the abnormal search mode, the image features of the abnormal single product are used to perform traversal identification and position positioning within the collection range, and the retained single product identification and positioning information is obtained for feedback.
[0106] Furthermore, the abnormal single product identification information determination module 15 in the system is also used to:
[0107] Collect sample data sets of retained goods, analyze corresponding single product retention records based on single product retention verification data, determine retention area characteristics and retention probability, and set collaborative management cameras;
[0108] The collaborative management camera is determined based on the matching of the single product retention verification data, the image features of the abnormal single product are sent to the collaborative management camera, and the abnormal search mode of the collaborative management camera is activated.
[0109] Furthermore, the multi-level image recognition module 12 in the system is also used for:
[0110] Constructing a multi-level image recognition model, the multi-level image recognition model includes a shelf recognition sub-model, a product recognition sub-model and a single product placement recognition sub-model, wherein the single product placement recognition sub-model includes a first channel and a second channel, the input image of the first channel is a grayscale image, and the input image of the second channel is a color image;
[0111] Extract key frames from the shelf image information, and perform image extraction according to a preset step length to obtain a shelf image extraction set;
[0112] The shelf image extraction set is input into the multi-level image recognition model for step-by-step processing and recognition, and the recognition results of each level are obtained and output to obtain the shelf recognition information, product recognition information, and single product placement recognition information.
[0113] Furthermore, the multi-level image recognition module 12 in the system is also used for:
[0114] The shelf recognition sub-model includes a feature recognition layer and a shelf positioning layer, wherein the feature recognition layer is obtained by collecting shelf feature sets to construct training data for network model learning, and the shelf positioning layer is obtained by constructing a training data set using camera positioning, image acquisition parameters, image shooting range and corresponding positioning identifiers, and performing supervised learning. The shelf recognition sub-model is used to perform recognition results and shelf positioning on the shelf image extraction set to obtain the shelf recognition information;
[0115] Constructing a product core feature list, which includes a mapping relationship between products and core features, and the product identification sub-model traverses the shelf image extraction set based on the product core feature list to determine the product identification information;
[0116] The grayscale image converted from the shelf image extraction set is subjected to border recognition through the first channel to determine the border recognition information of the single product. The color image of the shelf image extraction set is subjected to color distribution recognition through the second channel to determine the color feature recognition information of the single product. The border recognition information of the single product and the color feature recognition information of the single product are combined and outputted through a fully connected layer.
[0117] Furthermore, the multi-level image recognition module 12 in the system is also used for:
[0118] Collecting information sets of goods with the same attributes, extracting placement strategies for the information sets of goods with the same attributes, and determining information sets of goods on the same shelf;
[0119] Performing abnormal identification on the information set of goods on the same shelf to determine the difference characteristics of the goods, wherein the difference characteristics of the goods include size difference, packaging shape difference, color difference, and standard image difference;
[0120] Setting a feature difference constraint degree based on the number of product information sets on the same shelf, and determining a core feature of the product difference feature based on the feature difference constraint degree, wherein the core feature is a feature with the largest feature difference of the product;
[0121] A mapping relationship between goods and core features is established, and a list of the core features of the goods is constructed.
[0122] Furthermore, the multi-level image recognition module 12 in the system is also used for:
[0123] Clustering the goods in the concentrated images of the shelf image extraction, performing image segmentation according to the clustering results, and re-segmenting the image segmentation results to identify the objects to obtain a portrait segmentation result;
[0124] Traversing the image segmentation results based on the core features in the product core feature list to determine the product recognition results;
[0125] The same product identification results are aggregated to determine the product identification information, which includes the product identification quantity and the basic product information.
[0126] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The image recognition-based shelf management method and specific examples in the aforementioned embodiment 1 are also applicable to the image recognition-based shelf management system in this embodiment. Through the aforementioned detailed description of the image recognition-based shelf management method, those skilled in the art can clearly understand the image recognition-based shelf management system in this embodiment, so for the sake of brevity of the specification, it will not be described in detail here. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0127] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.
Claims
1. Shelf management method based on image recognition, It is characterized in that The method comprises: Connect to the management camera to obtain the collected shelf image information; Perform multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information; Locate the goods based on the shelf identification information, determine the goods shelf positioning information, collect the goods on the shelf information, and perform position identification marking on the goods shelf positioning information; Determine abnormal product identification information by using the product identification information and the product shelf location information, wherein the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and provide feedback based on the abnormal product identification information; Utilize the single product placement identification information and the shelf commodity information to identify single product features, determine abnormal single product identification information, and provide feedback; The method for determining abnormal item identification information and providing feedback includes: Connect to the settlement system to obtain commodity settlement data; Generate single product retention verification data, where the single product retention verification data is determined by performing data quantification calculation based on the single product settlement data and the shelf product information in the product settlement data; Place the items according to the item placement identification information and the item retention verification data, and generate abnormal reminder information when an abnormality exists; The method after generating the abnormal reminder information further includes: Based on the abnormal identification results, determine the abnormal single product image features; Sending the abnormal single product image features to the management camera to activate an abnormal search mode, wherein the abnormal search mode is an automatic image recognition mode including the abnormal single product image features; Based on the abnormal search mode, the image features of the abnormal single product are used to perform traversal identification and position positioning within the collection range, and the retained single product identification and positioning information is obtained for feedback.
2. The method according to claim 1, It is characterized in that The step of sending the abnormal single product image feature to the management camera and activating an abnormal search mode includes: Collect sample data sets of retained goods, analyze corresponding single product retention records based on single product retention verification data, determine retention area characteristics and retention probability, and set collaborative management cameras; The collaborative management camera is determined based on the matching of the single product retention verification data, the image features of the abnormal single product are sent to the collaborative management camera, and the abnormal search mode of the collaborative management camera is activated.
3. The method according to claim 1, It is characterized in that Performing multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information, including: Constructing a multi-level image recognition model, the multi-level image recognition model includes a shelf recognition sub-model, a product recognition sub-model and a single product placement recognition sub-model, wherein the single product placement recognition sub-model includes a first channel and a second channel, the input image of the first channel is a grayscale image, and the input image of the second channel is a color image; Extract key frames from the shelf image information, and perform image extraction according to a preset step length to obtain a shelf image extraction set; Input the shelf image extraction set into the multi-level image recognition model for step-by-step processing and recognition, obtain the recognition results of each level for output, and obtain the shelf recognition information, product recognition information, and single product placement recognition information; The step of inputting the shelf image extraction set into the multi-level image recognition model for step-by-step processing and recognition, and obtaining and outputting the recognition results at each level, includes: The shelf recognition sub-model includes a feature recognition layer and a shelf positioning layer, wherein the feature recognition layer is obtained by collecting shelf feature sets to construct training data for network model learning, and the shelf positioning layer is obtained by constructing a training data set using camera positioning, image acquisition parameters, image shooting range and corresponding positioning identifiers, and performing supervised learning. The shelf recognition sub-model is used to perform recognition results and shelf positioning on the shelf image extraction set to obtain the shelf recognition information; Constructing a product core feature list, which includes a mapping relationship between products and core features, and the product identification sub-model traverses the shelf image extraction set based on the product core feature list to determine the product identification information; The grayscale image converted from the shelf image extraction set is subjected to border recognition through the first channel to determine the border recognition information of the single product. The color image of the shelf image extraction set is subjected to color distribution recognition through the second channel to determine the color feature recognition information of the single product. The border recognition information of the single product and the color feature recognition information of the single product are combined and outputted through a fully connected layer.
4. The method according to claim 3, It is characterized in that The building of the product core feature list includes a mapping relationship between the product and the core feature, including: Collecting information sets of goods with the same attributes, extracting placement strategies for the information sets of goods with the same attributes, and determining information sets of goods on the same shelf; Performing abnormal identification on the information set of goods on the same shelf to determine the difference characteristics of the goods, wherein the difference characteristics of the goods include size difference, packaging shape difference, color difference, and standard image difference; Setting a feature difference constraint degree based on the number of product information sets on the same shelf, and determining a core feature of the product difference feature based on the feature difference constraint degree, wherein the core feature is a feature with the largest feature difference of the product; A mapping relationship between goods and core features is established, and a list of the core features of the goods is constructed.
5. The method according to claim 4, It is characterized in that The product identification sub-model traverses the shelf image extraction set based on the product core feature list to determine the product identification information, including: Clustering the goods in the concentrated images of the shelf image extraction, performing image segmentation according to the clustering results, and re-segmenting the image segmentation results to identify the objects to obtain a portrait segmentation result; Traversing the image segmentation results based on the core features in the product core feature list to determine the product recognition results; The same product identification results are aggregated to determine the product identification information, which includes the product identification quantity and the basic product information.
6. Shelf management system based on image recognition, It is characterized in that For implementing the steps of the method according to any one of claims 1 to 5, the system comprises: A shelf image information acquisition module, which is used to connect to a management camera to acquire collected shelf image information; A multi-level image recognition module, which is used to perform multi-level image recognition on the shelf image information to determine shelf identification information, product identification information, and single product placement identification information; Among them, the shelf image extraction set is input into the multi-level image recognition model for step-by-step processing and recognition, and the recognition results of each level are obtained and output, including: The shelf recognition sub-model includes a feature recognition layer and a shelf positioning layer, wherein the feature recognition layer is obtained by collecting shelf feature sets to construct training data for network model learning, and the shelf positioning layer is obtained by constructing a training data set using camera positioning, image acquisition parameters, image shooting range and corresponding positioning identifiers, and performing supervised learning. The shelf recognition sub-model is used to perform recognition results and shelf positioning on the shelf image extraction set to obtain the shelf recognition information; Constructing a product core feature list, which includes a mapping relationship between products and core features, and the product identification sub-model traverses the shelf image extraction set based on the product core feature list to determine the product identification information; Performing border recognition on the grayscale image converted from the shelf image extraction set through a first channel, the input image of the first channel is a grayscale image, determining the border recognition information of a single product, performing color distribution recognition on the color image of the shelf image extraction set through a second channel, the input image of the second channel is a color image, determining the color feature recognition information of a single product, and merging the border recognition information of the single product with the color feature recognition information of the single product through a fully connected layer for output; A goods shelf positioning information determination module, the goods shelf positioning information determination module is used to locate goods based on the shelf identification information, determine the goods shelf positioning information, collect goods on the shelf information, and perform position identification marking on the goods shelf positioning information; An abnormal product identification information determination module, the abnormal product identification information determination module is used to determine abnormal product identification information by using the product identification information and the product shelf location information, the abnormal product identification information includes that the product type does not match and the product quantity does not meet the preset requirements, and feedback is provided based on the abnormal product identification information; An abnormal single product identification information determination module, the abnormal single product identification information determination module is used to use the single product placement identification information and the shelf commodity information to perform single product feature identification, determine the abnormal single product identification information, and provide feedback; The method for determining abnormal item identification information and providing feedback includes: Connect to the settlement system to obtain commodity settlement data; Generate single product retention verification data, where the single product retention verification data is determined by performing data quantification calculation based on the single product settlement data and the shelf product information in the product settlement data; Place the items according to the item placement identification information and the item retention verification data, and generate abnormal reminder information when an abnormality exists; The method after generating the abnormal reminder information further includes: Based on the abnormal identification results, determine the abnormal single product image features; Sending the abnormal single product image features to the management camera to activate an abnormal search mode, wherein the abnormal search mode is an automatic image recognition mode including the abnormal single product image features; Based on the abnormal search mode, the image features of the abnormal single product are used to perform traversal identification and position positioning within the collection range, and the retained single product identification and positioning information is obtained for feedback.
